Sensor module abstraction

[kizniche]

Separating the sensor conversation started in issue #125, let's outline a general status of where the system is and where we'd like to go.

Current Setup

The sensor modules (Mycodo/mycodo/sensors/) provide a class to measure specific environmental conditions of each sensor and what those conditions are (but only at the time of measuring, returning a dictionary of conditions as keys and measurements as values).

The daemon (mycodo_daemon.py) starts a sensor controller thread (controller_sensor.py) for each sensor that's been added to the database (mycodo.db). Each thread then uses the sensor module class to retrieve measurements at predefined intervals, writes those measurements to influxdb, and enacts any added sensor conditionals.

The flask web UI (Mycodo/mycodo/mycodo_flask) has information about each sensor hard-coded into templates that allow it to know information about what to display for the sensors that have been added to the database (such as what measurements each sensor can provide). This is done using sensor types, that provide general information about measurements of each sensor. For instance, a pressure sensor of the "presssensor" type, will indicate that the sensor can return temperature, pressure, and altitude measurements (such as the BMP 180/185 sensor). However, this can be troublesome, as another pressure sensor may be able to return temperature, humidity, dew point, pressure, and altitude (such as the BME280).

Future Direction

If we can contain all information about each sensor into each module, a more robust system could develop, enabling only one area to modify to add a new sensor module (such as just dropping a new sensor module in the sensors directory and refreshing the frontend/backend). Each sensor module could be queried when needed to get information (for instance, about measurements in the multiselect form field on the graph page that displays all available measurements to choose from when creating a new graph).

@etiology has suggested adding middleware.

Things that all sensor modules should be able to do

• Read sensor and return measurements (individual or all?)
• Return all measurements that the sensor can sense/calculate
• Return the unit and its abbreviation for each measurement (e.g. Celsius, C; Relative Humidity, %; Pescals, Pa; Relay Duration, seconds; etc.)
• ?

  Questions

How is this best implemented? Should calculated measurements (e.g. dew point, altitude) be calculated in each sensor module or handled by the threaded sensor controller? How could unit conversion be handled (for instance if the user wants all units as standard instead of metric- could be tricky if all measurements are stored in the database in metric)?

[kizniche]

In addition to each sensor module returning a list of all possible measurements, it may be beneficial to store the desired measurements for each sensor in the database. This could be useful for saving time and space, as this will enable choosing which measurements you actually want logged for each sensor, instead of logging them all. For instance, the BME280 sensor only measures temperature, humidity, and pressure, but in addition to those, dew point and altitude is also calculated and stored along side the measurements. If you only wanted to measure pressure, then you could save ~80% of storage space and some time by not calculating and logging the other measurements.

This would necessitate either indicating to the sensor module what measurements you would like returned or retrieving them all and then selecting only the desired ones [for storage in the database]. The first option would require a refactoring of the modules but would be more efficient, and the second option would be able to work with the current modules but would be less efficient.

[kizniche]

Example sensor class function to return all possible measurements and info for a temperature/humidity sensor:

def get_measurements(self):
    return {
        'temperature': {
            'name': 'Temperature',
            'measure': 'Temperature',
            'unit': 'Celsius',
            'unit_short': 'C'
        },
        'humidity': {
            'name': 'Relative Humidity',
            'measure': 'Relative Humidity',
            'unit': 'Percent',
            'unit_short': '%'
        },
        'dewpoint': {
            'name': 'Dew Point',
            'measure': 'Temperature',
            'unit': 'Celsius',
            'unit_short': 'C'
        }
    }

Output would be:

sensor = sensor_module()
sensor.get_measurements()['humidity']['measure']

Relative Humidity

sensor.get_measurements()['humidity']['unit_short']

%

[ghost]

How is this best implemented? I'm in no position to tell how to do, only what I had already started...

Before discovering Mycodo, I was going to implement this project using json and rest/mqtt between the devices/sensors and acquisition gateway, and use a publish/subscribe model. My use cases are about exterior agricultural monitoring, like fields, silos, drying crops, etc.

🎱 http://fortune.com/2016/07/25/agriculture-farming-tech-startup-bubble/ ;-)

For example, a farmer wishes to monitor soil moisture/temperature to determine when to plant seeds: he places 20 measuring sensors over 300 acres of land that broadcast data that will be picked up by mqtt gateways installed nearby. The sensors don't know anything about the gateways and/or business logic, all they do is shout their json payload to the world, using whatever way they are meant to (rfm95,xbee,etc.), and the mqtt gateways subscribe to the sensors messages and stores the data in local databases, similar to part of what Mycodo does with Daemon (if I understand well), except that no action is taken by either devices/sensors or gateways.

Then, you have business logic (UI, PIDs, methods, etc., not depicted in my picture above) querying the middleware (in my initial plan -> https://github.com/adafruit/Adafruit_IO_Arduino ).

Should calculated measurements (e.g. dew point, altitude) be calculated in each sensor module or handled by the threaded sensor controller? Handled and stored outside of the sensor modules, part of business logic (when needed to be queried, reported, acted upon, etc.). In my picture above, I should have made the arrow between the mqtt middleware and the devices go both ways.

How could unit conversion be handled (for instance if the user wants all units as standard instead of metric- could be tricky if all measurements are stored in the database in metric)? In addition to each sensor module returning a list of all possible measurements, it may be beneficial to store the desired measurements for each sensor in the database.

Easy, store everything in metric (business logic-middleware responsability). Then the reporting can be adjusted on the fly for presentation purposes. The following example asks DarkSky with a private key (security has to be part of the design) for the weather info at my place and returning it in a json packet:

https://api.darksky.net/forecast/bf87dd8bfa2d7abe6df93541a3d984bc/45.7537,-73.2745?units=ca&lang=fr&exclude=minutely,hourly

If you are curious, you can copy the results of that query and paste it in the input field of the following form to see the message formatted for mere humans: http://jsonparseronline.com

[etiology]

You've put down some great stuff here. I really like the use of the message broker. I did something similar with a sensor & relay device using Redis. What you've laid out has a bunch of advantages. You could subscribe multiple RRDBs to some or all sensors areas. Likewise you could subscribe email notification services too and none of these need to touch the sensor code or RRDBs. Super neat.

Also good thoughts on measurement units. I'd say stick to SI units and let the consumer convert it to our backward imperial system for display purposes.

Anyway, really good thoughts in your post. I'll have to look into MQTT brokers.

[ghost]

I am tempted to use https://www.rethinkdb.com as the db to store the broadcasted sensor info capted by the msg broker.