search

jonathankoren / weewx-aqi

Air Quality Calculations for WeeWx
GNU General Public License v3.0
9 stars 4 forks source link

custom PM sensor #13

Open misko903 opened 3 years ago

misko903 commented 3 years ago

Hello Jonathan,

I have installed your plugin on WeeWX 4.2.0 with extended database - pm10_0 and pm2_5 values are stored in main weewx.sdb database. I dont have weewx-purpleair sensor, but custom sensor, which fills out the database through FilePile. So I modified the weewx.conf to:

# Options for extension 'aqi'
[AqiService]
    [[standard]]
        data_binding = aqi_binding
        standard = user.aqi.us.NowCast
    [[air_sensor]]
        data_binding = wx_binding
        usUnits = usUnits
        dateTime = dateTime
#        temp = extraTemp1
#        pressure = pressure
        pm2_5 = pm2_5
        pm10_0 = pm10_0

but I got these errors:

Dec 15 23:07:06 WeeWX /weewxd: AqiService: ValueError AQI calculation for pm2_5 on 1608069960 failed: NowCast AQI could not be calculated for the observations. Too many missing. Missing 2, which meets or exceeding the limit of 1
Dec 15 23:07:06 WeeWX /weewxd: AqiService: ValueError AQI calculation for pm10_0 on 1608069960 failed: NowCast AQI could not be calculated for the observations. Too many missing. Missing 2, which meets or exceeding the limit of 1
Dec 15 23:07:06 WeeWX /weewxd: AqiService: not storing record for dateTime 1608069960

what I have done wrong?

Thanks, Michal, Slovakia

Distortionist commented 3 years ago

Did you ever figure this out? I just found this trying to get my pm2_5 readings to be calculated to air quality index of canada or US. I am in over my head, I am trying to do what you are doing, just use the existing values stored in weewx.sdb like the pm2_5, temp, pressure and humidity. I have not installed this extension, I am still trying to figure out if it will work for me and how easy it is to get going. The instructions assume I know more than I do haha.

misko903 commented 3 years ago

This issue looks like it is missing two parameters for AQI calculation. So I have uncommented

#        temp = extraTemp1
#        pressure = pressure

But then weewx crashes:

Dec 28 03:50:11 WeeWX weewx[32674] INFO user.interceptor: shutting down server thread
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__: Caught unrecoverable exception:
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  unsupported operand type(s) for -: 'NoneType' and 'float'
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  Traceback (most recent call last):
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 210, in run
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      self.dispatchEvent(weewx.Event(weewx.CHECK_LOOP, packet=packet))
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 241, in dispatchEvent
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      callback(event)
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 630, in check_loop
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      raise BreakLoop
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  weewx.engine.BreakLoop
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  During handling of the above exception, another exception occurred:
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  Traceback (most recent call last):
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 646, in post_loop
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      self._catchup(self.engine.console.genArchiveRecords)
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 690, in _catchup
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      for record in generator(lastgood_ts):
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/drivers/__init__.py", line 30, in genArchiveRecords
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      raise NotImplementedError("Method 'genArchiveRecords' not implemented")
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  NotImplementedError: Method 'genArchiveRecords' not implemented
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  During handling of the above exception, another exception occurred:
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  Traceback (most recent call last):
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewxd", line 154, in main
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      engine.run()
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 217, in run
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      self.dispatchEvent(weewx.Event(weewx.POST_LOOP))
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 241, in dispatchEvent
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      callback(event)
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 648, in post_loop
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      self._software_catchup()
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 710, in _software_catchup
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      origin='software'))
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/engine.py", line 241, in dispatchEvent
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      callback(event)
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/user/aqi/service.py", line 311, in new_archive_record
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      temp_kelvin = weewx.units.CtoK(weewx.units.FtoC(row['outTemp']))
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****    File "/usr/share/weewx/weewx/units.py", line 43, in FtoC
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****      return (x - 32.0) * 5.0 / 9.0
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'
Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL __main__:     ****  Exiting.

Q: what is doing this in my weewx.conf AQI Stanza: usUnits = usUnits in my weewx i am using METRIC, or METRICWX for example [FilePile], which is collecting data from my SDS sensor: unit_system = METRICWX but in [StdConvert] i am using target_unit = US is the usUnits ok in my case? what are the possibilities?

Distortionist commented 3 years ago

I got mine working for US standards with these settings for my aqi sensor :

[AqiService] [[standard]] data_binding = aqi_binding standard = user.aqi.us.NowCast [[air_sensor]] data_binding = wx_binding usUnits = usUnits dateTime = dateTime temp = outTemp pressure = pressure pm2_5 = pm2_5 My issue is I live in Canada. I can't figure out how to tell it to use the Canadian AQHI standard, and it mentions that it is the only one that needs more than one reading but doesn't mention what readings I need for it.

I think this project needs better documentation for newbs like me.

misko903 commented 3 years ago

I got mine working for US standards with these settings for my aqi sensor :

[AqiService] [[standard]] data_binding = aqi_binding standard = user.aqi.us.NowCast [[air_sensor]] data_binding = wx_binding usUnits = usUnits dateTime = dateTime temp = outTemp pressure = pressure pm2_5 = pm2_5 My issue is I live in Canada. I can't figure out how to tell it to use the Canadian AQHI standard, and it mentions that it is the only one that needs more than one reading but doesn't mention what readings I need for it.

I think this project needs better documentation for newbs like me.

Are you using Purple Air Sensor? Me not, I just want to read everything from weewx database. Try to add humidity to your settings.

M.

Distortionist commented 3 years ago

No I am using Ambient Weather sensor for ws2000 and gw1000.

All my readings are from weewx database. It seems to work fine.

I don't see their humidity option in the settings of this aqi extension. They need to document it better I don't know how to change to Canada aqhi index or what readings it needs. The default example of settings doesn't mention humidity unless i missed it. I'm not home now.

On Thu., Dec. 31, 2020, 9:44 a.m. misko903, notifications@github.com wrote:

I got mine working for US standards with these settings for my aqi sensor :

[AqiService] [[standard]] data_binding = aqi_binding standard = user.aqi.us.NowCast [[air_sensor]] data_binding = wx_binding usUnits = usUnits dateTime = dateTime temp = outTemp pressure = pressure pm2_5 = pm2_5 My issue is I live in Canada. I can't figure out how to tell it to use the Canadian AQHI standard, and it mentions that it is the only one that needs more than one reading but doesn't mention what readings I need for it.

I think this project needs better documentation for newbs like me.

Are you using Purple Air Sensor? Me not, I just want to read everything from weewx database. Try to add humidity to your settings.

M.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jonathankoren/weewx-aqi/issues/13#issuecomment-752962035, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARJFPEI3ID6XPHXYKK5JRKLSXR54LANCNFSM4U5CBDZQ .

misko903 commented 3 years ago

ok, what did you changed that it work? Mine is still crashing.

Distortionist commented 3 years ago

absolutely nothing! just the config file I posted works for me. Are you sure your temperature and pressure is stored correctly? I see you are using temp = extraTemp1 are you sure that is correct? My outdoor array has the temperature in it stored as outTemp in my database weewx database. I don't really know what FilePile is and why you have it set to something different than weewx, if you are storing to the same weewx database. I think it is storing metric numbers and weewx thinks they are US numbers. I suggest trying to convert them to match or start over. I am just a newb I wish I can help more, but by the looks of it, one of those values is not what it is expecting. Something like your pressure is 1013 in metric, and it is like WOW that is crazy if it was expecting 29.91.

Distortionist commented 3 years ago

"Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL main: temp_kelvin = weewx.units.CtoK(weewx.units.FtoC(row['outTemp'])) Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL main: File "/usr/share/weewx/weewx/units.py", line 43, in FtoC Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL main: * return (x - 32.0) 5.0 / 9.0 Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL main: ** TypeError: unsupported operand type(s) for -: 'NoneType' and 'float' Dec 28 03:50:12 WeeWX weewx[32674] CRITICAL main: **** Exiting."

It is definitely complaining about your out temp. I really suck at this but something is wrong with the way your out temp is stored. It is trying to convert F to C, then C to K, but it can't because of something not making sense is the best I can make of what is happening here.

Distortionist commented 3 years ago

https://stackoverflow.com/questions/23037408/typeerror-unsupported-operand-types-for-float-and-nonetype-python

Reading that, I think outTemp one is empty or something. But from what you say too, the units may be stored in C to begin with and even if the conversion works, the value would be wrong. I think you need to tell your FilePile to store in us since that is what weewx thinks the database is, and make sure it is actually getting stored correctly. The error references outTemp not extraTemp1. I suggest Telling FilePile to store it as outTemp, in F, and match the setting temp = outTemp. See if that works! The code may be wrong and is not looking up the temp = setting somewhere, or your database has it missing or empty, I have no idea whos fault it is but I am pretty sure doing what I suggested will fix, at least this part of it. It works for me!

misko903 commented 3 years ago

So i changed the temp = outTemp. now it looks like this:

Jan  3 20:35:03 WeeWX /weewxd: AqiService: ValueError AQI calculation for pm2_5 on 1609702440 failed: NowCast AQI could not be calculated for the observations. Too many missing. Missing 2, which meets or exceeding the limit of 1
Jan  3 20:35:03 WeeWX /weewxd: AqiService: ValueError AQI calculation for pm10_0 on 1609702440 failed: NowCast AQI could not be calculated for the observations. Too many missing. Missing 1, which meets or exceeding the limit of 1
Jan  3 20:35:03 WeeWX /weewxd: AqiService: not storing record for dateTime 1609702440

Solution to your Canadian problem maybe here: https://github.com/jonathankoren/weewx-aqi/blob/master/bin/user/aqi/standards.py

Distortionist commented 3 years ago

Ok but do you know for sure that there is a value stored under outlets? I honestly don't know. Those are the values I gave it and its enough. I'm going to assume your outTemp field is there but empty now.

On Sun., Jan. 3, 2021, 3:36 p.m. misko903, notifications@github.com wrote:

So i changed the temp = outTemp. now it looks like this:

Jan 3 20:35:03 WeeWX /weewxd: AqiService: ValueError AQI calculation for pm2_5 on 1609702440 failed: NowCast AQI could not be calculated for the observations. Too many missing. Missing 2, which meets or exceeding the limit of 1 Jan 3 20:35:03 WeeWX /weewxd: AqiService: ValueError AQI calculation for pm10_0 on 1609702440 failed: NowCast AQI could not be calculated for the observations. Too many missing. Missing 1, which meets or exceeding the limit of 1 Jan 3 20:35:03 WeeWX /weewxd: AqiService: not storing record for dateTime 1609702440

Solution to your Canadian problem maybe here:

https://github.com/jonathankoren/weewx-aqi/blob/master/bin/user/aqi/standards.py

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jonathankoren/weewx-aqi/issues/13#issuecomment-753666046, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARJFPENXPAZ4SW6YMVZFBMDSYDBLDANCNFSM4U5CBDZQ .

misko903 commented 3 years ago

I have checked all the values, they are filled in - as US units in weewx.sdb

        usUnits = usUnits
        dateTime = dateTime
        temp = outTemp
        pressure = pressure
        pm2_5 = pm2_5
        pm10_0 = pm10_0

The PM values are measured/written every 3rd minute. There are no missing fields, the value is written 3 times unchanged. Can this be an issue?

Distortionist commented 3 years ago

I have no idea!! My values are written every 20 seconds I believe. I also don't have a pm 10 sensor but I can't see that being a problem. I installed it and followed your configuration adapted to me and it worked perfectly lol. Its complaining about not enough vales to calculate but if its all right you are giving it more than I am.

On Sun., Jan. 3, 2021, 3:56 p.m. misko903, notifications@github.com wrote:

I have checked all the values, they are filled in - as US units in weewx.sdb

    usUnits = usUnits
    dateTime = dateTime
    temp = outTemp
    pressure = pressure
    pm2_5 = pm2_5
    pm10_0 = pm10_0

The PM values are measured/written every 3rd minute. Can this be an issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jonathankoren/weewx-aqi/issues/13#issuecomment-753668154, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARJFPEKDFCBTRQ6PXTCIMCTSYDDXBANCNFSM4U5CBDZQ .

Distortionist commented 3 years ago

Since mine comes from the same station, all the values of everything have the same time stamp and frequency of update. I am guessing that has something to do with it. You are injecting data, maybe without or mismatching time stamps. I honestly don't know and this is way beyond me.

On Sun., Jan. 3, 2021, 4:27 p.m. distortionist, distortionist@gmail.com wrote:

I have no idea!! My values are written every 20 seconds I believe. I also don't have a pm 10 sensor but I can't see that being a problem. I installed it and followed your configuration adapted to me and it worked perfectly lol. Its complaining about not enough vales to calculate but if its all right you are giving it more than I am.

On Sun., Jan. 3, 2021, 3:56 p.m. misko903, notifications@github.com wrote:

I have checked all the values, they are filled in - as US units in weewx.sdb

    usUnits = usUnits
    dateTime = dateTime
    temp = outTemp
    pressure = pressure
    pm2_5 = pm2_5
    pm10_0 = pm10_0

The PM values are measured/written every 3rd minute. Can this be an issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jonathankoren/weewx-aqi/issues/13#issuecomment-753668154, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARJFPEKDFCBTRQ6PXTCIMCTSYDDXBANCNFSM4U5CBDZQ .

misko903 commented 3 years ago

Added to write values every 60 seconds. Ending up with the same error... maybe @jonathankoren can help here?

I'm glad that my conf helped you at least!

Distortionist commented 3 years ago

The time stamp is probably not the same. I'm a newb doing this with my 10 year old daughter for fun lol. I can't even believe I made it this far haha

On Sun., Jan. 3, 2021, 4:33 p.m. misko903, notifications@github.com wrote:

Added to write values every 60 seconds. Ending up with the same error... maybe @jonathankoren https://github.com/jonathankoren can help here?

I'm glad that my conf helped you at least!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jonathankoren/weewx-aqi/issues/13#issuecomment-753672197, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARJFPEKOP6SSGOMHZKPT54LSYDIATANCNFSM4U5CBDZQ .

misko903 commented 3 years ago

It works somehow :) Dont know why... I am setting up plotting now and coloured values. I dont know which AQI is used in my country - Slovakia

misko903 commented 3 years ago

OK, it doesn't work again. My sensor does not have cut of the extreme high values, so this is done by weewx.conf MinMax values - weewx will simply delete them. Then there are missing fields of pm2_5 and pm10_0 in the database (NULL) and weewx-aqi can't calculate the values. @jonathankoren, is it possible to ignore some predefined number of NULL fields?

jonathankoren commented 3 years ago

@misko903 I think it's better if you just delete the records from your DB

jonathankoren commented 3 years ago

@misko903 I believe Slovakia will use the EU standards that are in the new release. I found two online. I believe EuropeanAirQualityIndex is the new one.