JSON value pair mapping

[pmendiuk]

I have been adding sensors to WeeWX via WeeWX-MQTTSubscribe and previously I have been republishing the topics to "conform" to existing WeeWX database fields. I am dealing with topics like this:

{
  "Time": "2019-07-11T10:00:38",
  "BME280": {
    "Temperature": 70.6,
    "Humidity": 61.4,
    "Pressure": 1013.1
  },
  "ADS1115": {
    "A0": 135.78
    "Unit": "PSI"
  },
  "SI1145": {
    "Illuminance": 262,
    "Infrared": 281,
    "UvIndex": 0.3
  },
  "PressureUnit": "hPa",
  "TempUnit": "F"
}

I would like to be able to use this time stamp value pair format directly. The second issue is mapping these value pairs to the WeeWX database fields. What I'm thinking is adding something like this to weewx.conf:

[living_room_temperature] = [BME280.Temperature]
[living_room_humidity] = [BME280.Humidity]
[living_room_barometer] = ([BME280.Pressure] * 0.02953) + 0.2 
[pump_water_pressure] = [ADS1115.A0]
[living_room_illuminance] = [SI1145.Illuminance]
[living_room_infrared] = [SI1145.Infrared]
[living_room_UV] = [SI1145.UvIndex]

Thanks, Patrick

[bellrichm]

Patrick, First, do you really have multiple topics? From your example it looks like one JSON file is being published. If it is one JSON file with the data nested as shown, I’m pretty sure the current JSON parser won’t handle it. I’ll have to do some experiments to be sure. And unfortunately the concept of converting data, like your barometer example, is not supported. Currently the datetime has to be unix epoch. But, since I believe you are running as a service, you might be able to ignore that in the MQTT payload. It is only used to try to ensure the MQTT data should be added to the packet created by the driver. Depending on your answer to the first question, it might make sense to write a custom on_message handler to handle all of this. In the meantime, I’ll run some experiments with a file formatted like your example. Rich

[pmendiuk]

Rich,

Yes I have multiple nested topics.  That nested JSON example came from an ESP8266 sensor running Tasmota.  I modified the Tasmota code to use Epoch time for the time value pair but the Tasmota author doesn't seem inclined to   allow it in the main branch.

Thanks,

Patrick

Get Outlook for Android

On Thu, Jul 11, 2019 at 1:29 PM -0700, "Rich Bell" notifications@github.com wrote:

Patrick,

First, do you really have multiple topics? From your example it looks like one JSON file is being published.

If it is one JSON file with the data nested as shown, I’m pretty sure the current JSON parser won’t handle it. I’ll have to do some experiments to be sure.

And unfortunately the concept of converting data, like your barometer example, is not supported.

Currently the datetime has to be unix epoch. But, since I believe you are running as a service, you might be able to ignore that in the MQTT payload. It is only used to try to ensure the MQTT data should be added to the packet created by the driver.

Depending on your answer to the first question, it might make sense to write a custom on_message handler to handle all of this.

In the meantime, I’ll run some experiments with a file formatted like your example.

Rich

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[bellrichm]

Patrick, As I suspected, nested JSON is currently not supported. I am working on support for it. Once it is complete, you would add the following to the [[[label_map]]] section of the [MQTTSubscribe] section In weewx.conf.

BME280_Humidity = living_room_humidity 
BME280_Pressure = living_room_pressure
ADS1115_A0 = pump_water_pressure
SI1145_Illuminance = living_room_illuminance
SI1145_Infrared = living_room_infrared
SI1145_UvIndex = living_room_UV

And if I understand the StdCalibrate service you would add something like this to its [[Corrections]] section. living_room_barometer = living_room_pressure * 0.02953 + 0.2 I’ve never delved into this part of WeeWx, so we might need to get help from the experts.

I’ll also look into accepting different datetime formats and converting to unix epoch time.

Thoughts? Rich

[pmendiuk]

Rich,

I think parsing the Tasmota MQTT time stamp should be straight forward. He has added an Epoch value pair:

11:05:10 MQT: tele/wemos4/STATE = {"Time":"2019-07-11T11:05:10","Epoch":1562838655,"Uptime":"0T00:00:16","Heap":30,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"POWER1":"OFF","POWER2":"OFF","POWER3":"OFF","Wifi":{"AP":2,"SSId":"indebuurt2","BSSId":"A0:AB:1B:7D:42:AC","Channel":5,"RSSI":100,"LinkCount":1,"Downtime":"0T00:00:06"}}

I think having the ability to use either time format would be really useful. There is also the UTC offset or Zulu character that I have seen on some sensors.

2019-07-11T08:15:30-05:00

2019-07-11T15:15:30-05:00Z

I will look at the StdCalibrate service.

Thanks,

Patrick

From: Rich Bell notifications@github.com Sent: Friday, July 12, 2019 12:08 To: bellrichm/WeeWX-MQTTSubscribe WeeWX-MQTTSubscribe@noreply.github.com Cc: pmendiuk pmendiuk@comcast.net; Author author@noreply.github.com Subject: Re: [bellrichm/WeeWX-MQTTSubscribe] JSON value pair mapping (#28)

Patrick, As I suspected, nested JSON is currently not supported. I am working on support for it. Once it is complete, you would add the following to the [[[label_map]]] section of the [MQTTSubscribe] section In weewx.conf.

BME280_Humidity = living_room_humidity

BME280_Pressure = living_room_pressure

ADS1115_A0 = pump_water_pressure

SI1145_Illuminance = living_room_illuminance

SI1145_Infrared = living_room_infrared

SI1145_UvIndex = living_room_UV

And if I understand the StdCalibrate service http://www.weewx.com/docs/usersguide.htm#StdCalibrate you would add something like this to its [[Corrections]] section. living_room_barometer = living_room_pressure * 0.02953 + 0.2 I’ve never delved into this part of WeeWx, so we might need to get help from the experts.

I’ll also look into accepting different datetime formats and converting to unix epoch time.

Thoughts? Rich

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bellrichm/WeeWX-MQTTSubscribe/issues/28?email_source=notifications&email_token=ABYUWUBHC54FTSK7D7FHSSDP7DJCPA5CNFSM4IBPHGF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2T3ZQ#issuecomment-511000038 , or mute the thread https://github.com/notifications/unsubscribe-auth/ABYUWUFYZ4U6NVATIMCAJHTP7DJCPANCNFSM4IBPHGFQ . https://github.com/notifications/beacon/ABYUWUAUFVPXOUHNF7UJLYLP7DJCPA5CNFSM4IBPHGF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2T3ZQ.gif

[bellrichm]

If I understand, you could now map Epoch to dateTime via the [[[label_map]]] configuration setting. Epoch = dateTime But, I do see the usefulness of supporting additional formats so I have opened issue #29 to deal with that. Again, if you running as a service, the incoming dateTime is only used as a check that the MQTT ‘fits’ in the packet’s timespan. So, you might not even need it. Rich

[bellrichm]

Release 1.3.0 flattens nested/hiererchical json.