search

garbear / xbmc

XBMC Main Repository
http://xbmc.org
Other
132 stars 53 forks source link

Smart Home: Camera viewing and input streaming #128

Closed garbear closed 2 years ago

garbear commented 2 years ago

Description

This PR introduces a new paradigm of Kodi usage: smart home interaction and control. Camera viewing and input streaming are provided as two initial use cases.

The approach I've taken with smart home is a bit unique: I built on ROS 2. While this introduces some heavy overhead, building on an industrial decentralized communication framework allows for scalability to virtually unlimited low power smart home devices. My smart home's computation graph is currently at 8 Linux nodes and 3 Arduino nodes and growing.

Alongside this PR is a repo devoted to my smart home OS: https://github.com/eigendude/OASIS.

OASIS provides a complete ROS 2 stack for computer vision, input streaming and general automation. It also provides a complete implementation of the Firmata protocol for communicating with Arduinos, with additional support for temperature and humidity sensors, I2C, servos, sonar, SPI, stepper motors, and 4-wire CPU fans.

Camera viewing

A new cameraview control is introduced. It takes a parameter, the ROS image topic to subscribe to. For testing, I created the following SmartHome.xml window:

<?xml version="1.0" encoding="UTF-8"?>
<window>
    <controls>
        <control type="cameraview">
            <width>50%</width>
            <height>50%</height>
            <topic>/oasis/netbook/image_raw</topic>
        </control>
        <control type="cameraview">
            <left>50%</left>
            <height>50%</height>
            <topic>/oasis/lenovo/image_raw</topic>
        </control>
        <control type="cameraview">
            <width>50%</width>
            <top>50%</top>
            <topic>/oasis/netbook/foreground</topic>
        </control>
        <control type="cameraview">
            <left>50%</left>
            <top>50%</top>
            <topic>/oasis/lenovo/foreground</topic>
        </control>
    </controls>
</window>

This window creates a 2x2 matrix. The top two cameraview controls are the direct output of two laptop cameras. The bottom two cameraview controls are the output of a rudimentary person detector using a background subtraction library applied to the camera feeds.

Screenshot of the above window (not bad for low light netbook cameras):

Not very good person tracker

I forked the Kinect 2 driver and ported it to ROS 2. You can see depth registration working, but I'll probably do skeletal tracking on the 2D image via MediaPipe as a person detection solution.

Smart Home Camera View

Input streaming

The second use case is input streaming for smart home control. Kodi advertises two ROS topics:

Kodi also provides a ROS service:

In my home, I'm using a PS controller to drive my childhood Lego train. The input is captured by Kodi running on my NAS with a Bluetooth dongle, and sent to a Raspberry Pi with an Ardiuno and a robotics motor controller that drives the train's 9V motors.

Lego Train via PS4 Controller

Motivation and context

Smart Home!!!!

How has this been tested?

I've been using Kodi as part of my smart home setup since May of 2021. ROS 2 is so robust that my system barely goes down.

What is the effect on users?

Related PRs

Depends on:

Types of change

garbear commented 2 years ago

I'm very much building a stable system and then not touching it. Suggestions welcome but I don't currently have plans for this code.

garbear commented 2 years ago

PR sent upstream, and it drops the some of the [temp] commits.