Questions & Suggestions

[stantond]

Hi

Are you planning to add functionality to:

1. Allow a python app to easily create and read multiple sensor instances?
2. Allow readings on demand, returning readings in an appropriate structure, so that the timing and storing of data can be managed by the python app using PyPMS?

Would be very handy having a single package that supports so many sensors!

Also, do you make use of the SET and RESET pins on the PMSA003?

Thanks

[avaldebe]

Hi @stantond

Thanks for considering using my application.

1. Allow a python app to easily create and read multiple sensor instances?

I consider this, but found out it was easier to loop over in a bash script than headlining as sub-process/tasks in Python. Mainly, because I do not know how to async in Python and seemed to be an overkill.

A loop in bash would look something like:

# write devise output in csv format every 5 min
# append output from /dev/ttyUSB0 into pms0.csv, ...
for PORT in /dev/ttyUSB*; do
  pms --serial $PORT --sensor-model PMSA003 --interval 300 csv --filename pms${PORT#*USB}.csv &
done
wait

2. Allow readings on demand, returning readings in an appropriate structure, so that the timing and storing of data can be managed by the python app using PyPMS?

At some point I considered adding an one-shot mode, but I discarded the idea because the first 2 or 3 observations after the sensor wakes up are not reliable. Nevertheless, the underlying modules can be easily reused to provide one-shot readings. Mind to keep the sensor awake between readings in your application, to avoid the problem I just described.

# load the SensorReader reader class from the sensor module
from pms.sensor import SensorReader

# instantiate a reader object for the PMSA003 on /dev/ttyUSB0
# the last parameter is the time interval, you won't used for one-shot obs
reader = SensorReader("PMSA003", "/dev/ttyUSB0", 0)

# initiate the serial communication and wake up the sensor
# the sensor will be keep awake between reading inside the `with` clause
with reader:
    # one-shot reading
    obs = next(reader())

    # obs is a dataclass object, containing the observation values and timestamp
    obs.time # seconds since 1970 UTC (linux time)
    obs.date # datetime object, derived from obs.time
    obs.pm01 # PM1.0 [ug/m3]
    obs.pm25 # PM2.5 [ug/m3]
    obs.pm10 # PM10  [ug/m3]
    # the PMSA003 also give you the number counts
    obs.n0_3  # number of particles under 0.3 um [#/cm3]
    obs.n0_5  # number of particles under 0.5 um [#/cm3]
    obs.n1_0  # number of particles under 1.0 um [#/cm3]
    obs.n2_5  # number of particles under 2.5 um [#/cm3]
    obs.n5_0  # number of particles under 5.0 um [#/cm3]
    obs.n10_0 # number of particles under 10  um [#/cm3]
    # different sensors have different "extras"
    # help(obs) will give you more information

# the sensor will be put to sleep and the serial port will be closed
# when you leave the `with reader` clause

Also, do you make use of the SET and RESET pins on the PMSA003?

I'm not using the SET/RESET pins. The application sends the appropriate commands to wake up the sensor and request a reading.

I hope have answered your questions. Can you tell me what do you plan to do?

I developed PyPMS as an application, not as a library, so did not put real effort into documenting the inner workings.

Knowing more about what you want do will help me to better document the underlying modules.

[stantond]

Sure - I'm planning to build a plugin for OctoPrint to help people 3D printing at home monitor for potential health risks. A lot more information here.

I'll likely have two PMSA003 sensors, one inside and one outside the 3D Printer enclosure, and some others I'll add later (VOC sensors will be next).

It looks like https://github.com/FEEprojects/plantower/ may be built for this kind of use, though the number of supported sensor models is much lower.

[avaldebe]

Sure - I'm planning to build a plugin for OctoPrint to help people 3D printing at home monitor for potential health risks. A lot more information here.

Very interesting. I see a few ways I could help you out.

On the hardware side, I designed a USB2TTL board with the 8 pin connector the PMS3003 and PMS5003 uses and a adaptor to the connector used PMS7003 and PMSA003 https://easyeda.com/avaldebe/aqmonv2-usb

If you are not conformable with SMT soldering, I can send you 2 of my prototype units. I bodged the adapted board on this batch but they can still be used, with a simple hack. We can discuss the details by email if you are interested.

On the software side, I see you are considering the BME680. There is a cheap Chinese board which makes the sensor available via UART. I have one of this boards, and I was considering to add support for it. This is not really an PM sensor, but provides an air quality index. I could add support on a separate branch if you are interested.

Third, would it make it easier for you if this application would be available to pip install? Would you find useful instructions to install into a virtual environment?

I'll likely have two PMSA003 sensors, one inside and one outside the 3D Printer enclosure, and some others I'll add later (VOC sensors will be next).

It looks like https://github.com/FEEprojects/plantower/ may be built for this kind of use, though the number of supported sensor models is much lower.

If you plan to use this library, you still need to take discard the first 2-3 observations after a sleep/wake up cycle. On my tests I found out that it is not enough to wait some time before reading from a sensor who just woke up. The sensor needs to be "exercised" a few times before the measurements become meaningful.

After a sleep/wake up cycle, the sensor the first few readings have non zero PM values but all the number concentrations are zero. IMO, this observations should be discarded so I'm considering to add an --strict option for this. I have not observed the problem when the modules are just powered up.

[stantond]

Ok, lots of good ideas here. Just wanted to check you've seen https://github.com/FEEprojects/plantower - I'm not sure how much overlap there is between these projects, but it might be worth joining forces

[stantond]

I think what's needed for using this in other projects is something like this (for the gist, very much pseudocode):

sensor1 = SensorReader(port="/dev/ttyUSB0", sensor="PMSA003", read_mode="On Demand", output_mode="")

In passive mode, the sensor doesn't send data until requested, which is what you'd need for any type of "one shot" mode, though it isn't really one shot, it's more "on demand" vs "continuous". One shot would suggest discarding the sensor1 object.

I think you'd also want to change the mode. `sensor1.set_mode("Active")

And some checks to make sure you can't create multiple SensorReaders that would conflict/use the same port, if possible.

In passive mode, there'd need to be an easy way to digest the results. The objective wouldn't be to print them, but store or present them, so sensor1.read() could return a dict?

Could probably call it Sensor instead of SensorReader, as it's our digital representation of the sensor, and it also controls the sensor, e.g. sensor1.turn_fan_on().

The "plantower" project outputs 12 values per read instead of the 9 in your example above - I'm not sure which 3 are missing through, still figuring all of this out.

Here's a little more on what I intend to do with this over time:

1. Allow the user to specify multiple sensors in settings
2. Instantiate a sensor object for each
3. When reading (always in passive/on demand), iterate through each sensor, store the value in a simple database, plot the values on a graph, and update the overall "Air Quality" advice (e.g. "Bad, do not enter")
4. Eventually, based on the 3D printing progress, stage and total time, the readings will be spaced further apart. As the 3D printer heats up, the plastic melts, and emissions increase. At this time, it's interesting to read measurements a lot. Once the temperature stabilizes, my assumption is the readings become stable, so I can turn the sensors off (at least the fans where applicable) to extend life span, and every 30 minutes or 1 hour wake them up, read new measurements, and put to sleep again. Near the end of the print, monitoring increases once more so you can see as soon as it's safe to enter the room or open the enclosure.

I think that's the greatest value of a project like yours that communicates with so many different devices in one package. Being able to do the above for different sensors with mostly the same code will be very useful.

[avaldebe]

The "plantower" project outputs 12 values per read instead of the 9 in your example above - I'm not sure which 3 are missing through, still figuring all of this out.

The message contains 2 versions of PM1/2.5/10 values. The first set of values is for calibration purposes (CF=1 on the datasheet), the second set of values is calibrated values for "atmospheric conditions". This library reports the second set of values as pm01, pm25 and pm10. The 1st set of values is available as raw01, raw25 and raw10, which I omitted on the example.

[avaldebe]

In passive mode, the sensor doesn't send data until requested, which is what you'd need for any type of "one shot" mode, though it isn't really one shot, it's more "on demand" vs "continuous".

Almost all the sensors supported on this library work on continuous mode by default, where the spam the serial port with new results on regular-ish interval. This library puts the sensor on passive mode, and request new data on a user defined interval. The difference between reading the sensor once and the one-shot mode proposed on #4 is the handling of spurious measurements right after the sensor wakes up.

[avaldebe]

In passive mode, there'd need to be an easy way to digest the results. The objective wouldn't be to print them, but store or present them, so sensor1.read() could return a dict?

obs = sensor1.read() is a dataclass instance, which can be converted into a dictionary with dataclasses.asdict(obs)

[avaldebe]

And some checks to make sure you can't create multiple SensorReaders that would conflict/use the same port, if possible.

As far as I know, the serial.Serial object won't allow to open a port twice.

[stantond]

Quick update, I'm at the point in this project where the user can manage their serial sensors properly in the settings UI. I've used the Plantower library for now, but plan to switch to this one when "one-shot" is ready and it's in PIP.

Is there a mapping of the data attribute provided against each sensor? My next step is to store each read in a db.

Have you thought about renaming this project to a more general Air Quality library if you're planning to add the BME680 and others, if that's the direction you want to go in?

[avaldebe]

I've used the Plantower library for now, but plan to switch to this one when "one-shot" is ready and it's in PIP.

Good to know about your progress. I'll put some time to close #3 and #4, merge #5 so the next version goes into pypi/pip (#10)

Is there a mapping of the data attribute provided against each sensor? My next step is to store each read in a db.

Each observation (obs) is a DataClass instance, so you can obtain convert into a dictionary and extract the keys with dataclasses.asdict(obs).keys(). The headers for csv files are created using this mechanism. Maybe you can use the header and csv formats (f"{obs:header}" and f"{obs:csv}") in your db call.

What kind of db have you in mind? Maybe I can create a custom format for that.

Have you thought about renaming this project to a more general Air Quality library if you're planning to add the BME680 and others, if that's the direction you want to go in?

Good question. I'm interested in Air Quality sensors in general, but most AQ sensors have analogue or I2C/SPI interfaces. The MH-Z19 uses UART so it might include it in the future, or I might develop my own hardware bridges for interesting sensors.

If I where to expand the library to other AQ (serial/UART) sensors, it would be a good idea to rename the pypi/pip package (PyPMS). As I have not uploaded any version to pypi, this would be the time to find a better package name (PyPMS was not a good name to begin with). Do you have any sugestions?

[stantond]

I'm not sure if a verbose name helps with discoverability or not, though I suspect if the description contains the keywords it doesn't matter. Any of these any good?

• PySerialSensors
• PySerialAQSensors
• PySAQS
• PyAirQualitySensors
• PyAQS
• PyAirQualitySerial
• PyAirQualitySerialSensors
• PyAQSS

It's probably going to be a sqlite database with two tables, sensors and readings. Sensors will contain info like display name, port, location. Each reading references a sensor. I might also try to integrate with a filament management plugin eventually so that each reading also references the filament that was printing at the time.

[avaldebe]

I'm not sure if a verbose name helps with discoverability or not, though I suspect if the description contains the keywords it doesn't matter.

Thanks for the suggestions, I think you are right and that a verbose name is not help with discoberability. I'll most likely use something short, like saqs for the package, module and command line (unless I find out it is a curse word is some language).

It's probably going to be a sqlite database with two tables, sensors and readings. Sensors will contain info like display name, port, location. Each reading references a sensor. I might also try to integrate with a filament management plugin eventually so that each reading also references the filament that was printing at the time.

Have a look at obs.subset("pm") will convert an observation (dataclass instace) into a dictionary and return only the keywords that start with "pm" ("pm01", "pm25" and "pm10")

[avaldebe]

@stantond

I think your questions were answers, so I'll close this issue.

Cheers, Á.

[stantond]

I'm still not really sure how to do this:

Here's a little more on what I intend to do with this over time:

1. Allow the user to specify multiple sensors in settings
2. Instantiate a sensor object for each
3. When reading (always in passive/on demand), iterate through each sensor, store the value in a simple database, plot the values on a graph, and update the overall "Air Quality" advice (e.g. "Bad, do not enter")

There's a single thread that loops through all sensors, taking a reading and storing it in the database, Something like:

self.sensors = []
devices = self.database_manager.get_devices()
for device in devices:
    if device["port"] in self.serial_port_details.keys():
        device["reader"] = SensorReader(device["model"], device["port"], 0)
        self.sensors.append(device)`

while self.readThreadStop is False:
    for sensor in sensors:
        try:
            reading = sensor["reader"].read()
            self.database_manager.insert_reading(sensor["id"], reading)
        except serial.SerialException:
                self._connected = False
                self._logger.error("Error reading from sensor")
                self.stopReadThread()
    time.sleep(30)

In the with example from earlier, it looks like I'd need to wake and sleep the sensors for every reading as I couldn't leave the with running, but that is problematic, mostly because the readings can't be trusted immediately on wake.

Have I misunderstood?

[avaldebe]

Since 0.3.0 you do not longer need to worry abut inconsistent readings after waking up a PMSx003 sensor. Therefore, you can use the with statement as in the previous example. Then, your single thread sensor loop looks like like:

while self.readThreadStop is False:
    try:
        for sensor in self.sensors:
            with sensor["reader"] as reader:
                self.database_manager.insert_reading(sensor["id"], next(reader()))
    except serial.SerialException:
            self._connected = False
            self._logger.error("Error reading from sensor")
            self.stopReadThread()
    time.sleep(30)

[stantond]

That works, thanks!