search

bakerkj / weewx-purpleair

WeeWx Extension to pull data from the local interface of a Purple Air air quality sensor
GNU General Public License v2.0
17 stars 11 forks source link

purpleair extension won't write to purpleair.sdb #31

Open splobsterman opened 4 months ago

splobsterman commented 4 months ago

I have installed and uninstalled, and I still can't get the extension to write to the sqlite db

My weewx.conf DB details all look right I have the correct IP address for the purpleair unit. What am I missing?

WeeWx 4.10 NUC 5 Raspian Desktop Debian 10

bakerkj commented 4 months ago

1) Are there any errors in the logs?

2) What user is the process running as? What ownership is on the directory that you are trying to write to?

splobsterman commented 4 months ago

Process: weewx -rw-r--r-- 1 root root 102400 Jun 2 16:28 purpleair.sdb -rwxr-xr-x 1 root root 384401408 Jun 2 21:00 weewx.sdb

splobsterman commented 4 months ago

Ooops, no log errors

splobsterman commented 4 months ago

pi@NUC62:/home/weewx $ ls -l archive
total 454364 -rwxr-xr-x 1 root root 3006464 Jun 2 21:03 forecast.sdb -rw-r--r-- 1 root root 22269952 Jun 2 19:00 nws.sdb -rw-r--r-- 1 root root 102400 Jun 2 16:28 purpleair.sdb -rwxr-xr-x 1 root root 55476224 Oct 15 2021 weewx-orig.sdb -rwxr-xr-x 1 root root 384401408 Jun 2 21:04 weewx.sdb

splobsterman commented 4 months ago

Changed purpleair.sdb permissions to 755 to match weewx.sdb

splobsterman commented 4 months ago

And re-started weewx

splobsterman commented 4 months ago

Still not writing to purpleair.sdb

splobsterman commented 4 months ago

Still tearing my hair out on this. Ironically, it works perfectly w/weewx 4.10.2 in another location, same hardware.

pi@NUC62:/home/weewx $ sudo tail -f /var/log/syslog | grep weewx | grep purple Jun 3 21:52:51 NUC62 weewx[6510] INFO user.purpleair: Exception: 'response_date' Jun 3 21:53:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty Jun 3 21:54:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty Jun 3 21:55:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty Jun 3 21:56:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty Jun 3 21:57:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty Jun 3 21:57:52 NUC62 weewx[6510] INFO user.purpleair: Exception: 'response_date' Jun 3 21:58:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty Jun 3 21:59:19 NUC62 weewx[6510] DEBUG user.purpleair: Skipping record: empty ` Response from PA unit: at 192.168.26/json

{"SensorId":"c8:c9:a3:f:5e:6d","DateTime":"2024/06/04T02:01:06z","Geo":"PurpleAir-5e6d","Mem":7416,"memfrag":42,"memfb":4064,"memcs":432,"Id":615,"lat":43.732399,"lon":-69.844299,"Adc":0.02,"loggingrate":15,"place":"outside","version":"7.04","uptime":34151,"rssi":-49,"period":120,"httpsuccess":294,"httpsends":294,"hardwareversion":"3.0","hardwarediscovered":"3.0+OPENLOG+NO-DISK+BME68X+PMSX003-A+PMSX003-B","current_temp_f":67,"current_humidity":55,"current_dewpoint_f":50,"pressure":1016.15,"current_temp_f_680":67,"current_humidity_680":55,"current_dewpoint_f_680":50,"pressure_680":1016.15,"gas_680":747.61,"p25aqic_b":"rgb(0,228,0)","pm2.5_aqi_b":2,"pm1_0_cf_1_b":0.00,"p_0_3_um_b":175.39,"pm2_5_cf_1_b":0.50,"p_0_5_um_b":51.11,"pm10_0_cf_1_b":0.80,"p_1_0_um_b":5.86,"pm1_0_atm_b":0.00,"p_2_5_um_b":1.25,"pm2_5_atm_b":0.50,"p_5_0_um_b":0.29,"pm10_0_atm_b":0.80,"p_10_0_um_b":0.00,"p25aqic":"rgb(0,228,0)","pm2.5_aqi":3,"pm1_0_cf_1":0.17,"p_0_3_um":209.19,"pm2_5_cf_1":0.71,"p_0_5_um":58.97,"pm10_0_cf_1":1.14,"p_1_0_um":7.53,"pm1_0_atm":0.17,"p_2_5_um":1.86,"pm2_5_atm":0.71,"p_5_0_um":0.80,"pm10_0_atm":1.14,"p_10_0_um":0.00,"pa_latency":293,"wlstate":"Connected","status_0":2,"status_1":0,"status_2":2,"status_3":2,"status_4":2,"ssid":"Stowaway"}

bakerkj commented 4 months ago

@splobsterman this is exactly the information I was looking for. Thanks!

This is the key to the failure:

Jun 3 21:52:51 NUC62 weewx[6510] INFO user.purpleair: Exception: 'response_date'

When connected directly to a PurpleAir the code attempts to extract the parameter response_date:

        j = r.json()
        last_seen = datetime.datetime.utcfromtimestamp(j['response_date'])

My unit (as you can see below) returns that parameter. From your json it looks like your PurpleAir does not. I wonder why? Is that because your software is newer (7.02 vs 7.04) or the hardware is (2.0 vs 3.0).

I am thinking we could extract the DateTime parameter as a fallback when the resonse_date isn't there...

This is the data from my unit:

{
  "SensorId": "xx:xx:xx:xx:xx:xx",
  "DateTime": "2024/06/05T14:19:21z",
  "Geo": "PurpleAir-xxxx",
  "Mem": 20088,
  "memfrag": 12,
  "memfb": 17704,
  "memcs": 736,
  "Id": 332969,
  "lat": xx.xxx,
  "lon": -xx.xxx,
  "Adc": 0,
  "loggingrate": 15,
  "place": "outside",
  "version": "7.02",
  "uptime": 1945557,
  "rssi": -55,
  "period": 120,
  "httpsuccess": 166619,
  "httpsends": 178497,
  "hardwareversion": "2.0",
  "hardwarediscovered": "2.0+OPENLOG+16021 MB+DS3231+BME280+PMSX003-B+PMSX003-A",
  "current_temp_f": 76,
  "current_humidity": 65,
  "current_dewpoint_f": 63,
  "pressure": 1013.6,
  "p25aqic_b": "rgb(255,255,0)",
  "pm2.5_aqi_b": 53,
  "pm1_0_cf_1_b": 9.45,
  "p_0_3_um_b": 1693.91,
  "pm2_5_cf_1_b": 13.07,
  "p_0_5_um_b": 478.53,
  "pm10_0_cf_1_b": 13.5,
  "p_1_0_um_b": 85.52,
  "pm1_0_atm_b": 9.45,
  "p_2_5_um_b": 5,
  "pm2_5_atm_b": 13.07,
  "p_5_0_um_b": 0.52,
  "pm10_0_atm_b": 13.5,
  "p_10_0_um_b": 0.52,
  "p25aqic": "rgb(255,255,0)",
  "pm2.5_aqi": 52,
  "pm1_0_cf_1": 8.41,
  "p_0_3_um": 1625.12,
  "pm2_5_cf_1": 12.33,
  "p_0_5_um": 452.36,
  "pm10_0_cf_1": 12.5,
  "p_1_0_um": 81.05,
  "pm1_0_atm": 8.41,
  "p_2_5_um": 3.66,
  "pm2_5_atm": 12.33,
  "p_5_0_um": 0.34,
  "pm10_0_atm": 12.5,
  "p_10_0_um": 0,
  "pa_latency": 285,
  "response": 201,
  "response_date": 1717597075,
  "latency": 1171,
  "wlstate": "Connected",
  "status_0": 2,
  "status_1": 2,
  "status_2": 2,
  "status_3": 2,
  "status_4": 0,
  "status_5": 0,
  "status_6": 2,
  "status_7": 0,
  "status_8": 0,
  "status_9": 0,
  "ssid": "Xxxxx"
}
bakerkj commented 4 months ago

@splobsterman could you try a quick experiment? Specifically, could you find where purpleair.py is installed. And make the following change. I believe this will get things working for you. If it does I'll go ahead and make a more complete change.

+++ b/bin/user/purpleair.py
@@ -188,7 +188,7 @@ def collect_data(session, hostname, port, timeout, api_key):
         last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
     else:
         j = r.json()
-        last_seen = datetime.datetime.utcfromtimestamp(j['response_date'])
+        last_seen = datetime.datetime.strptime(j['DateTime'], "%y/%m/%dT%H:%M:%Sz")

     record = dict()
     record['dateTime'] = int(time.time())
splobsterman commented 4 months ago

Ken,

Happy to, but to be sure I am doing it right; is this what the replacement should look like?

Line 155:

def collect_data(session, hostname, port, timeout, api_key):

replace line 155 with this:?

@@ -188,7 +188,7 @@ def collect_data(session, hostname, port, timeout, api_key): last_seen = datetime.datetime.utcfromtimestamp(j['last_seen']) else: j = r.json()

On 5 Jun, 2024, at 11:16, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman could you try a quick experiment? Specifically, could you find where purpleair.py is installed. And make the following change. I believe this will get things working for you. If it does I'll go ahead and make a more complete change.

+++ b/bin/user/purpleair.py @@ -188,7 +188,7 @@ def collect_data(session, hostname, port, timeout, api_key): last_seen = datetime.datetime.utcfromtimestamp(j['last_seen']) else: j = r.json()

splobsterman commented 4 months ago

I pulled down my sensor and checked its firmware, which says it is 1.06, with no updates available.

On 5 Jun, 2024, at 10:56, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman this is exactly the information I was looking for. Thanks!

This is the key to the failure:

Jun 3 21:52:51 NUC62 weewx[6510] INFO user.purpleair: Exception: 'response_date' When connected directly to a PurpleAir the code attempts to extract the parameter response_date:

    j = r.json()
    last_seen = datetime.datetime.utcfromtimestamp(j['response_date'])

My unit (as you can see below) returns that parameter. From your json it looks like your PurspleAir does not. I wonder why? Is that because your software is newer (7.02 vs 7.04) or the hardware is (2.0 vs 3.0).

I am thinking we could extract the DateTime parameter as a fallback when the resonse_date isn't there...

This is the data from my unit:

{ "SensorId": "xx:xx:xx:xx:xx:xx", "DateTime": "2024/06/05T14:19:21z", "Geo": "PurpleAir-xxxx", "Mem": 20088, "memfrag": 12, "memfb": 17704, "memcs": 736, "Id": 332969, "lat": xx.xxx, "lon": -xx.xxx, "Adc": 0, "loggingrate": 15, "place": "outside", "version": "7.02", "uptime": 1945557, "rssi": -55, "period": 120, "httpsuccess": 166619, "httpsends": 178497, "hardwareversion": "2.0", "hardwarediscovered": "2.0+OPENLOG+16021 MB+DS3231+BME280+PMSX003-B+PMSX003-A", "current_temp_f": 76, "current_humidity": 65, "current_dewpoint_f": 63, "pressure": 1013.6, "p25aqic_b": "rgb(255,255,0)", "pm2.5_aqi_b": 53, "pm1_0_cf_1_b": 9.45, "p_0_3_um_b": 1693.91, "pm2_5_cf_1_b": 13.07, "p_0_5_um_b": 478.53, "pm10_0_cf_1_b": 13.5, "p_1_0_um_b": 85.52, "pm1_0_atm_b": 9.45, "p_2_5_um_b": 5, "pm2_5_atm_b": 13.07, "p_5_0_um_b": 0.52, "pm10_0_atm_b": 13.5, "p_10_0_um_b": 0.52, "p25aqic": "rgb(255,255,0)", "pm2.5_aqi": 52, "pm1_0_cf_1": 8.41, "p_0_3_um": 1625.12, "pm2_5_cf_1": 12.33, "p_0_5_um": 452.36, "pm10_0_cf_1": 12.5, "p_1_0_um": 81.05, "pm1_0_atm": 8.41, "p_2_5_um": 3.66, "pm2_5_atm": 12.33, "p_5_0_um": 0.34, "pm10_0_atm": 12.5, "p_10_0_um": 0, "pa_latency": 285, "response": 201, "response_date": 1717597075, "latency": 1171, "wlstate": "Connected", "status_0": 2, "status_1": 2, "status_2": 2, "status_3": 2, "status_4": 0, "status_5": 0, "status_6": 2, "status_7": 0, "status_8": 0, "status_9": 0, "ssid": "Xxxxx" } — Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2150285633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVFN4WDQJC7TICBKG5TZF4RKRAVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJQGI4DKNRTGM. You are receiving this because you were mentioned.

bakerkj commented 4 months ago

@splobsterman

This is the old code:

    r.raise_for_status()
    # convert to json                                                                                                                                                                                                                                                                                                 
    if is_data_from_purpleair_website:
        rj = r.json()
        j = rj['sensor']
        last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
    else:
        j = r.json()
        last_seen = datetime.datetime.utcfromtimestamp(j['response_date'])

    record = dict()
    record['dateTime'] = int(time.time())
    record['usUnits'] = weewx.US

This is the new

    r.raise_for_status()
    # convert to json                                                                                                                                                                                                                                                                                                 
    if is_data_from_purpleair_website:
        rj = r.json()
        j = rj['sensor']
        last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
    else:
       j = r.json()
       last_seen = datetime.datetime.strptime(j['DateTime'], "%y/%m/%dT%H:%M:%Sz")

    record = dict()
    record['dateTime'] = int(time.time())
    record['usUnits'] = weewx.US
splobsterman commented 4 months ago

Ken,

I have purpleair.py running with your mod and no errors are thrown, but neither does it write to the .sdb file.

# raise error if status is invalid
r.raise_for_status()

convert to json

if is_data_from_purpleair_website:
    rj = r.json()
    j = rj['sensor']
    last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
else:
    j = r.json()
    last_seen = datetime.datetime.strptime(j['DateTime'], "%y/%m/%dT%H:%M:%Sz")
record = dict()
record['dateTime'] = int(time.time())
record['usUnits'] = weewx.US

On 6 Jun, 2024, at 00:34, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman This is the old code:

r.raise_for_status()
# convert to json                                                                                                                                                                                                                                                                                                 
if is_data_from_purpleair_website:
    rj = r.json()
    j = rj['sensor']
    last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
else:
    j = r.json()
    last_seen = datetime.datetime.utcfromtimestamp(j['response_date'])

record = dict()
record['dateTime'] = int(time.time())
record['usUnits'] = weewx.US

This is the new

r.raise_for_status()
# convert to json                                                                                                                                                                                                                                                                                                 
if is_data_from_purpleair_website:
    rj = r.json()
    j = rj['sensor']
    last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
else:
   j = r.json()
   last_seen = datetime.datetime.strptime(j['DateTime'], "%y/%m/%dT%H:%M:%Sz")

record = dict()
record['dateTime'] = int(time.time())
record['usUnits'] = weewx.US

— Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2151394377, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVAS6OHCCVHE22AMH5DZF7RFBAVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJRGM4TIMZXG4. You are receiving this because you were mentioned.

bakerkj commented 4 months ago

Ken, I have purpleair.py running with your mod and no errors are thrown, but neither does it write to the .sdb file.

That is good that there are no more errors. I'll have a look in the next few days to try and figure out what else might be wrong.

-- Ken

splobsterman commented 4 months ago

What has puzzled me right along is that I have the exact same setup running on another site, which works:

Intel NUC 5 Raspian Desktop (Debian 10) WeeWx 4.10.2 Your PA extension

So it would seem the PA sensor is perhaps the issue. PS is looking at my json output and think there may be something not right in it; more when I hear.

On 6 Jun, 2024, at 16:04, Ken Baker @.***> wrote:

Ken, I have purpleair.py running with your mod and no errors are thrown, but neither does it write to the .sdb file.

That is good that there are no more errors. I'll have a look in the next few days to try and figure out what else might be wrong.

-- Ken

— Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2153316227, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVHUGULWHVDEX6ODHMDZGC6FNAVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGMYTMMRSG4. You are receiving this because you were mentioned.

splobsterman commented 4 months ago

Ken,

How is the 'response_date' formatted on your unit?

I managed to ssh into my other site and retrieve the json output, which is here. It also uses DateTime, but that setup works, using your extension.

{"SensorId":"xx:xx:xx:xx:xx:xx" "DateTime":"2024/06/07T15:43:11z" "Geo":"PurpleAir-795d" "Mem":10176 "memfrag":22 "memfb":7944 "memcs":352 "Id":26197 "lat":26.480400 "lon":-82.174599 "Adc":0.00 "loggingrate":15 "place":"outside" "version":"7.04" "uptime":1303297 "rssi":-68 "period":120 "httpsuccess":22078 "httpsends":22078 "hardwareversion":"3.0" "hardwarediscovered":"3.0+OPENLOG+NO-DISK+RV3028+BME68X+PMSX003-A+PMSX003-B" "current_temp_f":100 "current_humidity":43 "current_dewpoint_f":73 "pressure":1014.11 "current_temp_f_680":100 "current_humidity_680":43 "current_dewpoint_f_680":73 "pressure_680":1014.11 "gas_680":93.54 "p25aqic_b":"rgb(8 229 0)" "pm2.5_aqi_b":16 "pm1_0_cf_1_b":3.23 "p_0_3_um_b":709.18 "pm2_5_cf_1_b":3.86 "p_0_5_um_b":206.62 "pm10_0_cf_1_b":4.50 "p_1_0_um_b":15.12 "pm1_0_atm_b":3.23 "p_2_5_um_b":1.61 "pm2_5_atm_b":3.86 "p_5_0_um_b":1.20 "pm10_0_atm_b":4.50 "p_10_0_um_b":0.82 "p25aqic":"rgb(4 228 0)" "pm2.5_aqi":13 "pm1_0_cf_1":2.15 "p_0_3_um":731.24 "pm2_5_cf_1":3.07 "p_0_5_um":206.03 "pm10_0_cf_1":3.61 "p_1_0_um":19.97 "pm1_0_atm":2.15 "p_2_5_um":1.19 "pm2_5_atm":3.07 "p_5_0_um":0.80 "pm10_0_atm":3.61 "p_10_0_um":0.19 "pa_latency":272 "response":201 "response_date":1717774900 "latency":576 "wlstate":"Connected" "status_0":2 "status_1":0 "status_2":2 "status_3":2 "status_4":2 "status_6":2 "ssid":"XXXXXXXXX"}

On 5 Jun, 2024, at 10:56, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman this is exactly the information I was looking for. Thanks! This is the key to the failure:

Jun 3 21:52:51 NUC62 weewx[6510] INFO user.purpleair: Exception: 'response_date' When connected directly to a PurpleAir the code attempts to extract the parameter response_date:

    j = r.json()
    last_seen = datetime.datetime.utcfromtimestamp(j['response_date'])

My unit (as you can see below) returns that parameter. From your json it looks like your PurspleAir does not. I wonder why? Is that because your software is newer (7.02 vs 7.04) or the hardware is (2.0 vs 3.0).

I am thinking we could extract the DateTime parameter as a fallback when the resonse_date isn't there...

This is the data from my unit:

{ "SensorId": "xx:xx:xx:xx:xx:xx", "DateTime": "2024/06/05T14:19:21z", "Geo": "PurpleAir-xxxx", "Mem": 20088, "memfrag": 12, "memfb": 17704, "memcs": 736, "Id": 332969, "lat": xx.xxx, "lon": -xx.xxx, "Adc": 0, "loggingrate": 15, "place": "outside", "version": "7.02", "uptime": 1945557, "rssi": -55, "period": 120, "httpsuccess": 166619, "httpsends": 178497, "hardwareversion": "2.0", "hardwarediscovered": "2.0+OPENLOG+16021 MB+DS3231+BME280+PMSX003-B+PMSX003-A", "current_temp_f": 76, "current_humidity": 65, "current_dewpoint_f": 63, "pressure": 1013.6, "p25aqic_b": "rgb(255,255,0)", "pm2.5_aqi_b": 53, "pm1_0_cf_1_b": 9.45, "p_0_3_um_b": 1693.91, "pm2_5_cf_1_b": 13.07, "p_0_5_um_b": 478.53, "pm10_0_cf_1_b": 13.5, "p_1_0_um_b": 85.52, "pm1_0_atm_b": 9.45, "p_2_5_um_b": 5, "pm2_5_atm_b": 13.07, "p_5_0_um_b": 0.52, "pm10_0_atm_b": 13.5, "p_10_0_um_b": 0.52, "p25aqic": "rgb(255,255,0)", "pm2.5_aqi": 52, "pm1_0_cf_1": 8.41, "p_0_3_um": 1625.12, "pm2_5_cf_1": 12.33, "p_0_5_um": 452.36, "pm10_0_cf_1": 12.5, "p_1_0_um": 81.05, "pm1_0_atm": 8.41, "p_2_5_um": 3.66, "pm2_5_atm": 12.33, "p_5_0_um": 0.34, "pm10_0_atm": 12.5, "p_10_0_um": 0, "pa_latency": 285, "response": 201, "response_date": 1717597075, "latency": 1171, "wlstate": "Connected", "status_0": 2, "status_1": 2, "status_2": 2, "status_3": 2, "status_4": 0, "status_5": 0, "status_6": 2, "status_7": 0, "status_8": 0, "status_9": 0, "ssid": "Xxxxx" } — Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2150285633, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVFN4WDQJC7TICBKG5TZF4RKRAVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJQGI4DKNRTGM. You are receiving this because you were mentioned.

bakerkj commented 4 months ago

@splobsterman

The response_date for my sensor and yours looks the same: yours: "response_date":1717774900 mine: "response_date": 1717597075

All three of our systems do return DateTime. The code prefers response_date because it uses that date to choice to publish the data to the database, and I wanted to ensure the date of the data to be used, not the date of the device.

bakerkj commented 4 months ago

Could you share what your system shows when you access the purpleair.sdb database?

sqlite3 /home/weewx/archive/purpleair.sdb
.table
.headers on
select * from archive order by dateTime DESC limit 10;

This is the output form my system:

bakerkj@kaylee:~$ sqlite3 /var/lib/weewx/purpleair.sdb 
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> .table                                                                                                                                       
archive                         archive_day_pm2_5_atm         
archive_day__metadata           archive_day_pm2_5_cf_1        
archive_day_pm10_0_atm          archive_day_purple_dewpoint   
archive_day_pm10_0_cf_1         archive_day_purple_humidity   
archive_day_pm1_0_atm           archive_day_purple_pressure   
archive_day_pm1_0_cf_1          archive_day_purple_temperature
sqlite> .headers on
sqlite> select * from archive order by dateTime DESC limit 10;                                                                                       
dateTime|usUnits|interval|purple_temperature|purple_humidity|purple_dewpoint|purple_pressure|pm1_0_cf_1|pm1_0_atm|pm2_5_cf_1|pm2_5_atm|pm10_0_cf_1|pm10_0_atm
1717779244|1|5|82.0|56.0|65.0|29.58255087775|3.515|3.515|5.135|5.135|5.31|5.31
1717778944|1|5|82.0|58.0|65.0|29.58373207725|3.31|3.31|5.065|5.065|5.065|5.065
1717778643|1|5|82.0|60.0|67.0|29.583436777375|4.075|4.075|6.23|6.23|6.44|6.44
1717778343|1|5|82.0|62.0|67.0|29.585799176375|4.415|4.415|6.575|6.575|6.715|6.715
1717778043|1|5|81.0|62.0|67.0|29.586685076|4.24|4.24|6.425|6.425|6.61|6.61
1717777742|1|5|81.0|63.0|67.0|29.586685076|4.285|4.285|6.575|6.575|6.83|6.83
1717777442|1|5|80.0|64.0|66.0|29.551249091|4.2|4.2|6.485|6.485|6.585|6.585
1717777142|1|5|81.0|65.0|68.0|29.5878662755|4.49|4.49|6.65|6.65|6.775|6.775
1717776841|1|5|80.0|65.0|67.0|29.584617976875|4.755|4.755|7.21|7.21|7.335|7.335
1717776539|1|5|80.0|67.0|68.0|29.5878662755|4.895|4.895|7.47|7.47|7.79|7.79
splobsterman commented 4 months ago

Sure, but as no data has ever been written to it, here is what you get:

$ sqlite3 /home/weewx/archive/purpleair.sdb SQLite version 3.27.2 2019-02-25 16:06:06 Enter ".help" for usage hints. sqlite> .table archive archive_day_pm2_5_atm
archive_day__metadata archive_day_pm2_5_cf_1
archive_day_pm10_0_atm archive_day_purple_dewpoint
archive_day_pm10_0_cf_1 archive_day_purple_humidity
archive_day_pm1_0_atm archive_day_purple_pressure
archive_day_pm1_0_cf_1 archive_day_purple_temperature sqlite> .headers on sqlite> select * from archive order by dateTime DESC limit 10;

On 7 Jun, 2024, at 12:59, Ken Baker @.***> wrote:

Could you share what your system shows when you access the purpleair.sdb database?

sqlite3 /home/weewx/archive/purpleair.sdb .table .headers on select * from archive order by dateTime DESC limit 10; This is the output form my system:

@.**:~$ sqlite3 /var/lib/weewx/purpleair.sdb SQLite version 3.11.0 2016-02-15 17:29:24 Enter ".help" for usage hints. sqlite> .table
archive archive_day_pm2_5_atm
archive_day__metadata archive_day_pm2_5_cf_1
archive_day_pm10_0_atm archive_day_purple_dewpoint
archive_day_pm10_0_cf_1 archive_day_purple_humidity
archive_day_pm1_0_atm archive_day_purple_pressure
archive_day_pm1_0_cf_1 archive_day_purple_temperature sqlite> .headers on sqlite> select from archive order by dateTime DESC limit 10;
dateTime|usUnits|interval|purple_temperature|purple_humidity|purple_dewpoint|purple_pressure|pm1_0_cf_1|pm1_0_atm|pm2_5_cf_1|pm2_5_atm|pm10_0_cf_1|pm10_0_atm 1717779244|1|5|82.0|56.0|65.0|29.58255087775|3.515|3.515|5.135|5.135|5.31|5.31 1717778944|1|5|82.0|58.0|65.0|29.58373207725|3.31|3.31|5.065|5.065|5.065|5.065 1717778643|1|5|82.0|60.0|67.0|29.583436777375|4.075|4.075|6.23|6.23|6.44|6.44 1717778343|1|5|82.0|62.0|67.0|29.585799176375|4.415|4.415|6.575|6.575|6.715|6.715 1717778043|1|5|81.0|62.0|67.0|29.586685076|4.24|4.24|6.425|6.425|6.61|6.61 1717777742|1|5|81.0|63.0|67.0|29.586685076|4.285|4.285|6.575|6.575|6.83|6.83 1717777442|1|5|80.0|64.0|66.0|29.551249091|4.2|4.2|6.485|6.485|6.585|6.585 1717777142|1|5|81.0|65.0|68.0|29.5878662755|4.49|4.49|6.65|6.65|6.775|6.775 1717776841|1|5|80.0|65.0|67.0|29.584617976875|4.755|4.755|7.21|7.21|7.335|7.335 1717776539|1|5|80.0|67.0|68.0|29.5878662755|4.895|4.895|7.47|7.47|7.79|7.79 — Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2155205176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVAPQHRNTZ54ICDPCRTZGHRH3AVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGIYDKMJXGY. You are receiving this because you were mentioned.

splobsterman commented 4 months ago

Here's the output from the working setup in Florida:

sqlite3 /home/weewx/archive/purpleair.sdb SQLite version 3.27.2 2019-02-25 16:06:06 Enter ".help" for usage hints. sqlite> .table archive archive_day_pm2_5_atm
archive_day__metadata archive_day_pm2_5_cf_1
archive_day_pm10_0_atm archive_day_purple_dewpoint
archive_day_pm10_0_cf_1 archive_day_purple_humidity
archive_day_pm1_0_atm archive_day_purple_pressure
archive_day_pm1_0_cf_1 archive_day_purple_temperature sqlite> .headers on sqlite> select * from archive order by dateTime DESC limit 10; dateTime|usUnits|interval|purple_temperature|purple_humidity|purple_dewpoint|purple_pressure|pm1_0_cf_1|pm1_0_atm|pm2_5_cf_1|pm2_5_atm|pm10_0_cf_1|pm10_0_atm 1717795578|1|5|105.0|36.0|72.0|29.877555452875|2.335|2.335|2.715|2.715|3.055|3.055 1717795277|1|5|105.0|37.0|73.0|29.88021315175|2.075|2.075|2.66|2.66|3.26|3.26 1717794977|1|5|104.0|37.0|73.0|29.881984951|1.905|1.905|2.18|2.18|2.575|2.575 1717794676|1|5|105.0|37.0|73.0|29.883461450375|1.96|1.96|2.44|2.44|2.755|2.755 1717794375|1|5|105.0|36.0|73.0|29.881984951|2.01|2.01|2.63|2.63|3.25|3.25 1717794073|1|5|105.0|37.0|73.0|29.8855285495|1.92|1.92|2.26|2.26|3.06|3.06 1717793772|1|5|106.0|35.0|73.0|29.889072148|2.08|2.08|2.515|2.515|2.905|2.905 1717793472|1|5|106.0|35.0|73.0|29.891729846875|2.295|2.295|2.715|2.715|3.32|3.32 1717792894|1|5|107.0|35.0|73.0|29.89674994475|1.865|1.865|2.29|2.29|3.005|3.005 1717792593|1|5|106.0|36.0|73.0|29.898521744|1.835|1.835|2.42|2.42|3.1|3.1

On 7 Jun, 2024, at 12:59, Ken Baker @.***> wrote:

Could you share what your system shows when you access the purpleair.sdb database?

sqlite3 /home/weewx/archive/purpleair.sdb .table .headers on select * from archive order by dateTime DESC limit 10; This is the output form my system:

@.**:~$ sqlite3 /var/lib/weewx/purpleair.sdb SQLite version 3.11.0 2016-02-15 17:29:24 Enter ".help" for usage hints. sqlite> .table
archive archive_day_pm2_5_atm
archive_day__metadata archive_day_pm2_5_cf_1
archive_day_pm10_0_atm archive_day_purple_dewpoint
archive_day_pm10_0_cf_1 archive_day_purple_humidity
archive_day_pm1_0_atm archive_day_purple_pressure
archive_day_pm1_0_cf_1 archive_day_purple_temperature sqlite> .headers on sqlite> select from archive order by dateTime DESC limit 10;
dateTime|usUnits|interval|purple_temperature|purple_humidity|purple_dewpoint|purple_pressure|pm1_0_cf_1|pm1_0_atm|pm2_5_cf_1|pm2_5_atm|pm10_0_cf_1|pm10_0_atm 1717779244|1|5|82.0|56.0|65.0|29.58255087775|3.515|3.515|5.135|5.135|5.31|5.31 1717778944|1|5|82.0|58.0|65.0|29.58373207725|3.31|3.31|5.065|5.065|5.065|5.065 1717778643|1|5|82.0|60.0|67.0|29.583436777375|4.075|4.075|6.23|6.23|6.44|6.44 1717778343|1|5|82.0|62.0|67.0|29.585799176375|4.415|4.415|6.575|6.575|6.715|6.715 1717778043|1|5|81.0|62.0|67.0|29.586685076|4.24|4.24|6.425|6.425|6.61|6.61 1717777742|1|5|81.0|63.0|67.0|29.586685076|4.285|4.285|6.575|6.575|6.83|6.83 1717777442|1|5|80.0|64.0|66.0|29.551249091|4.2|4.2|6.485|6.485|6.585|6.585 1717777142|1|5|81.0|65.0|68.0|29.5878662755|4.49|4.49|6.65|6.65|6.775|6.775 1717776841|1|5|80.0|65.0|67.0|29.584617976875|4.755|4.755|7.21|7.21|7.335|7.335 1717776539|1|5|80.0|67.0|68.0|29.5878662755|4.895|4.895|7.47|7.47|7.79|7.79 — Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2155205176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVAPQHRNTZ54ICDPCRTZGHRH3AVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGIYDKMJXGY. You are receiving this because you were mentioned.

bakerkj commented 4 months ago

Sure, but as no data has ever been written to it, here is what you get:

@splobsterman - good just as expected. I'll put together some better debugging in the code and send that along shortly.

splobsterman commented 3 months ago

Hi Ken, happy to try out whatever you might have anytime.

On 7 Jun, 2024, at 17:38, Ken Baker @.***> wrote:

Sure, but as no data has ever been written to it, here is what you get:

@splobsterman https://github.com/splobsterman - good just as expected. I'll put together some better debugging in the code and send that along shortly.

— Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2155594889, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVC2JI2QM6DH5X2DAYLZGIR63AVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJVGU4TIOBYHE. You are receiving this because you were mentioned.

bakerkj commented 3 months ago

@splobsterman a quick question, are you running weewx with debug=1?

bakerkj commented 3 months ago

@splobsterman at the end of the collect_data() function could you change:

                 record[key] = valA
             else:
                 record[key] = (valA + valB) / 2.0
     return record

To the following:

                 record[key] = valA
             else:
                 record[key] = (valA + valB) / 2.0

    logdbg("record: %s" % record)

     return record

This should show us what record is being generated.

Mine generates the following:

Jun 16 19:07:58 xxx weewx[29532] DEBUG user.purpleair: record: {'pm10_0_atm': 4.695, 'pm10_0_cf_1': 4.695, 'pm1_0_cf_1': 2.95, 'pm2_5_atm': 4.390000000000001, 'purple_pressure': 30.223646906375002, 'dateTime': 1718579278, 'purple_dewpoint': 48.0, 'purple_humidity': 38.0, 'pm1_0_atm': 2.95, 'purple_temperat
ure': 75.0, 'pm2_5_cf_1': 4.390000000000001, 'usUnits': 1}
Jun 16 19:12:58 xxx weewx[29532] DEBUG user.purpleair: record: {'pm10_0_atm': 5.300000000000001, 'pm10_0_cf_1': 5.300000000000001, 'pm1_0_cf_1': 3.505, 'pm2_5_atm': 5.065, 'purple_pressure': 30.221875107124998, 'dateTime': 1718579578, 'purple_dewpoint': 48.0, 'purple_humidity': 39.0, 'pm1_0_atm': 3.505, 'p
urple_temperature': 75.0, 'pm2_5_cf_1': 5.065, 'usUnits': 1}
splobsterman commented 3 months ago

Yes, would you like a copy of the log from a re-start?

On 16 Jun, 2024, at 18:56, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman a quick question, are you running weewx with debug=1?

— Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2171922919, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVGAWHQX52JGTPHGD6DZHYJ25AVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZRHEZDEOJRHE. You are receiving this because you were mentioned.

bakerkj commented 3 months ago

Yes, would you like a copy of the log from a re-start?

yes, please

splobsterman commented 3 months ago

Ken,

Sorry to be difficult but would you point me to the line number in purpleair.py, as collect_data() occurs more than once.

And: do you want me to add it to the original script, or to the one we modified with:

This is the new

r.raise_for_status()
# convert to json                                                                                                                                                                                                                                                                                                 
if is_data_from_purpleair_website:
    rj = r.json()
    j = rj['sensor']
    last_seen = datetime.datetime.utcfromtimestamp(j['last_seen'])
else:
   j = r.json()
   last_seen = datetime.datetime.strptime(j['DateTime'], "%y/%m/%dT%H:%M:%Sz")

record = dict()
record['dateTime'] = int(time.time())
record['usUnits'] = weewx.US

Regards,

On 16 Jun, 2024, at 19:12, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman at the end of the collect_data() function could you change:

             record[key] = valA
         else:
             record[key] = (valA + valB) / 2.0
 return record

To the following:

             record[key] = valA
         else:
             record[key] = (valA + valB) / 2.0

logdbg("record: ", record)

 return record

This should show us what record is being generated.

— Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2171927639, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVGUKMJBDHZ7KHI6GI3ZHYLV5AVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZRHEZDONRTHE. You are receiving this because you were mentioned.

splobsterman commented 3 months ago

@splobsterman at the end of the collect_data() function could you change:

                 record[key] = valA
             else:
                 record[key] = (valA + valB) / 2.0
     return record

To the following:

                 record[key] = valA
             else:
                 record[key] = (valA + valB) / 2.0

    logdbg("record: %s" % record)

     return record

This should show us what record is being generated.

Mine generates the following:

Jun 16 19:07:58 xxx weewx[29532] DEBUG user.purpleair: record: {'pm10_0_atm': 4.695, 'pm10_0_cf_1': 4.695, 'pm1_0_cf_1': 2.95, 'pm2_5_atm': 4.390000000000001, 'purple_pressure': 30.223646906375002, 'dateTime': 1718579278, 'purple_dewpoint': 48.0, 'purple_humidity': 38.0, 'pm1_0_atm': 2.95, 'purple_temperat
ure': 75.0, 'pm2_5_cf_1': 4.390000000000001, 'usUnits': 1}
Jun 16 19:12:58 xxx weewx[29532] DEBUG user.purpleair: record: {'pm10_0_atm': 5.300000000000001, 'pm10_0_cf_1': 5.300000000000001, 'pm1_0_cf_1': 3.505, 'pm2_5_atm': 5.065, 'purple_pressure': 30.221875107124998, 'dateTime': 1718579578, 'purple_dewpoint': 48.0, 'purple_humidity': 39.0, 'pm1_0_atm': 3.505, 'p
urple_temperature': 75.0, 'pm2_5_cf_1': 5.065, 'usUnits': 1}

Here is the output from:

pi@NUC62:/home/weewx $ sudo tail -f /var/log/syslog | grep purple

pi@NUC62:/home/weewx $ sudo tail -f /var/log/syslog | grep purple Jun 21 16:42:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:43:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:44:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:45:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:46:17 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:46:19 NUC62 weewx[13016] INFO user.purpleair: Exception: time data '2024/06/21T20:46:18z' does not match format '%y/%m/%dT%H:%M:%Sz' Jun 21 16:47:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:48:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:49:19 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:50:17 NUC62 weewx[13016] DEBUG user.purpleair: Skipping record: empty Jun 21 16:51:19 NUC62 weewx[13016] INFO user.purpleair: Exception: time data '2024/06/21T20:51:18z' does not match format '%y/%m/%dT%H:%M:%Sz'

splobsterman commented 3 months ago

Just posted the log output to github

On 16 Jun, 2024, at 19:12, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman at the end of the collect_data() function could you change:

             record[key] = valA
         else:
             record[key] = (valA + valB) / 2.0
 return record

To the following:

             record[key] = valA
         else:
             record[key] = (valA + valB) / 2.0

logdbg("record: ", record)

 return record

This should show us what record is being generated.

— Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2171927639, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVGUKMJBDHZ7KHI6GI3ZHYLV5AVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZRHEZDONRTHE. You are receiving this because you were mentioned.

bakerkj commented 3 months ago

@splobsterman I am sorry, the code had a typo in the date conversion. Can you try this change instead - https://raw.githubusercontent.com/bakerkj/weewx-purpleair/bca1cc2f3e9a0a0be1409ee8b8fa8ee447799841/bin/user/purpleair.py

splobsterman commented 3 months ago

I tried that file as a replacement for purpleair.py, and it threw some other error. For the time being I am abandoing this, despite the fact I have it working perfectly on my other NUC in Florida.

I may just reformat my NUC Raspberry Desktop install and start fresh

Regards

On 22 Jun, 2024, at 13:49, Ken Baker @.***> wrote:

@splobsterman https://github.com/splobsterman I am sorry, the code had a typo in the date conversion. Can you try this change instead - https://raw.githubusercontent.com/bakerkj/weewx-purpleair/bca1cc2f3e9a0a0be1409ee8b8fa8ee447799841/bin/user/purpleair.py https://raw.githubusercontent.com/bakerkj/weewx-purpleair/bca1cc2f3e9a0a0be1409ee8b8fa8ee447799841/bin/user/purpleair.py — Reply to this email directly, view it on GitHub https://github.com/bakerkj/weewx-purpleair/issues/31#issuecomment-2184127524, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHGMYVAG2L3KL5LRZGGAIU3ZIW2JJAVCNFSM6AAAAABIVQKYISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBUGEZDONJSGQ. You are receiving this because you were mentioned.

bakerkj commented 2 months ago

I tried that file as a replacement for purpleair.py, and it threw some other error.

@splobsterman did you happen to note what new error it generated?

splobsterman commented 1 week ago

Back at this...

replaced original purpleair.py with the one at your link above.

Error:

INFO user.purpleair: Exception: logdbg() takes 1 positional argument but 2 were given