Int Values Stored As Floats in DB

[tomdotorg]

Using a wifi logger2 to send messages to be picked up by Weewx-MQTTSubscribe. Soil temps, which are subfields (if it matters), are sent by the Logger as ints, but they are being stored as floats with seemingly random decimals since they are all emitted as ints.

I did try setting conversion_type = int at the root of the tree to force ints, but that didn't change it.

Obviously, I could be off-base on this.

Thanks, Rich.

Here is my config:

[MQTTSubscribeDriver]

This section is for the MQTTSubscribe driver.

# The driver to use:
driver = user.MQTTSubscribe
host = mqtt.tom.org
port = 1883
keepalive = 60
log = true
[[message_callback]]
    type = json
[[topics]]
    unit_system = US
[[[weather/vt-dev/wflwflexpj.json]]]
    ignore = true # opt out of processing fields for this topic by default - override per field
    [[[[tempout]]]]
        ignore = false # opt back in for this field
        name = outTemp
    [[[[tempin]]]]
        ignore = false # opt back in for this field
        name = inTemp
    [[[[humin]]]]
        ignore = false # opt back in for this field
        name = inHumidity
    [[[[humout]]]]
        ignore = false # opt back in for this field
        name = outHumidity
    [[[[windspd]]]]
        ignore = false # opt back in for this field
        name = windSpeed
    [[[[winddir]]]]
        ignore = false # opt back in for this field
        name = windDir

conversion_error_to_none = true

    [[[[gust]]]]
        ignore = false # opt back in for this field
        name = windGust
    [[[[gustdir]]]]
        ignore = false # opt back in for this field
        name = windGustDir

conversion_error_to_none = true

    [[[[bar]]]]
        ignore = false # opt back in for this field
        name = barometer
    [[[[rainyear]]]]
        ignore = false # opt back in for this field
        contains_total = true
        name = rain
    [[[[rainr]]]]
        ignore = false # opt back in for this field
        name = rainRate
    [[[[etyear]]]]
        ignore = false # opt back in for this field
        name = ET
        contains_total = true
    [[[[dew]]]]
        ignore = false # opt back in for this field
        name = dewpoint
    [[[[heat]]]]
        ignore = false # opt back in for this field
        name = heatindex
    [[[[solar]]]]
        ignore = false # opt back in for this field
        name = radiation
    [[[[uv]]]]
        ignore = true # opt back in for this field
        name = UV
        conversion_error_to_none = true
    [[[[bat]]]]
        ignore = false # opt back in for this field
        name = consBatteryVoltage
    [[[[xst]]]]
        ignore = false # opt back in for this field
        conversion_type = int
        [[[[[subfields]]]]]
             [[[[[[soilTemp1]]]]]]
             [[[[[[soilTemp2]]]]]]
             [[[[[[soilTemp3]]]]]]
             [[[[[[soilTemp4]]]]]]
    [[[[xsm]]]]
        [[[[[subfields]]]]]
             [[[[[[soilMoist1]]]]]]
             [[[[[[soilMoist2]]]]]]
             [[[[[[soilMoist3]]]]]]
                ignore = false # opt back in for this field
             [[[[[[soilMoist4]]]]]]

[bellrichm]

It looks like subfields are not 'inheriting' the configuration. Would it be possible to capture a few MWTT messages with mosquitto_sub, or the MQTT client of your choice?

[tomdotorg]

Thanks, Rich. You mean the source messages? The fields in question are the xst.

{"stnname":"vtdev","stnmod":16,"ver":3.88,"mac":"8C:CE:4E:82:98:4C","apmac":"8C:CE:4E:82:98:4D","ip":"192.168.0.20","ssid":"FortressOfSolitude","rssi":-42,"wifimod":0,"lastboot":1664216968,"uptime":74283,"wflver":"2.40","loctime":1664276661,"utctime":1664291061,"tzone":10,"units":4,"conlati":445,"conlongi":-717,"wfllati":-300.0000000,"wfllongi":-300.0000000,"tempout":55.2,"humout":85,"tempin":71.5,"humin":42,"windspd":4.0,"winddir":219,"windavg2":5.8,"windavg10":3.6,"gust":8.0,"gustdir":225,"bar":29.801,"bartr":20,"dew":51,"cdew":50.8,"chill":55,"heat":55,"thsw":66,"uv":null,"solar":645,"rainr":0.00000,"storm":0.64000,"rain15":0.00000,"rain1h":0.00000,"raind":0.02000,"rain24":0.29000,"rainmon":5.16000,"rainyear":23.49000,"etday":0.012,"etmon":1.62,"etyear":22.76,"xt":[null,null,null,null,null,null,null],"xlt":[null,null,null,null],"xst":[59,59,55,60],"xh":[null,null,null,null,null,null,null],"xsm":[null,null,3,null],"xlw":[null,null,null,0],"bat":4.78,"trbat":0,"foreico":3,"forrule":192,"sunrt":"6:40","sunst":"18:35","hlbar":[29.746,29.801,"0:15","10:46",30.372,29.507,30.836,29.030],"hlwind":[null,9.0,null,"10:51",23.0,null,45.0,null],"hltempin":[71.1,74.6,"8:30","0:00",91.0,66.0,96.0,50.0],"hlhumin":[42,43,"0:02","0:01",49,37,59,16],"hltempout":[40.9,55.1,"6:51","11:03",79.0,34.0,88.0,-25.0],"hlhumout":[85,100,"10:54","7:37",100,44,100,14],"hldew":[41,51,"5:59","9:48",70,33,78,-30],"hlchil":[41,null,"6:17",null,null,33,null,-32],"hlheat":[null,55,null,"10:57",81,null,99,null],"hlthsw":[null,66,null,"10:57",92,null,109,null],"hlsolar":[null,661,null,"10:49",1208,null,1296,null],"hluv":[null,0.0,null,null,0.0,null,0.0,null],"hlrainr":[0.000,0.080,null,"1:03",3.920,null,4.270,null],"hlxt0":[null,null,null,null,null,null,null,null],"hlxt1":[null,null,null,null,null,null,null,null],"hlxt2":[null,null,null,null,null,null,null,null],"hlxt3":[null,null,null,null,null,null,null,null],"hlxt4":[null,null,null,null,null,null,null,null],"hlxt5":[null,null,null,null,null,null,null,null],"hlxt6":[null,null,null,null,null,null,null,null],"hlxh0":[null,null,null,null,null,null,null,null],"hlxh1":[null,null,null,null,null,null,null,null],"hlxh2":[null,null,null,null,null,null,null,null],"hlxh3":[null,null,null,null,null,null,null,null],"hlxh4":[null,null,null,null,null,null,null,null],"hlxh5":[null,null,null,null,null,null,null,null],"hlxh6":[null,null,null,null,null,null,null,null],"hlxst0":[59,60,"8:46","0:00",63,59,63,59],"hlxst1":[58,59,"0:00","6:11",62,58,62,58],"hlxst2":[53,59,"6:56","0:00",65,47,65,47],"hlxst3":[59,60,"2:40","0:00",164,-27,164,-27],"hlxlt0":[null,null,null,null,null,null,null,null],"hlxlt1":[null,null,null,null,null,null,null,null],"hlxlt2":[null,null,null,null,null,null,null,null],"hlxlt3":[null,null,null,null,null,null,null,null],"hlxsm0":[null,null,"0:00",null,null,null,null,null],"hlxsm1":[null,null,"0:00",null,null,null,null,null],"hlxsm2":[3,3,"0:00","0:00",16,2,16,2],"hlxsm3":[null,null,"0:00",null,null,null,null,null],"hlxlw0":[null,null,"0:00",null,null,null,null,null],"hlxlw1":[null,null,"0:00",null,null,null,null,null],"hlxlw2":[null,null,"0:00",null,null,null,null,null],"hlxlw3":[null,null,"0:00",null,null,null,null,null]}

{"stnname":"vtdev","stnmod":16,"ver":3.88,"mac":"8C:CE:4E:82:98:4C","apmac":"8C:CE:4E:82:98:4D","ip":"192.168.0.20","ssid":"FortressOfSolitude","rssi":-42,"wifimod":0,"lastboot":1664216968,"uptime":74379,"wflver":"2.40","loctime":1664276757,"utctime":1664291157,"tzone":10,"units":4,"conlati":445,"conlongi":-717,"wfllati":-300.0000000,"wfllongi":-300.0000000,"tempout":55.3,"humout":84,"tempin":71.5,"humin":42,"windspd":10.0,"winddir":219,"windavg2":6.9,"windavg10":4.0,"gust":12.0,"gustdir":224,"bar":29.801,"bartr":20,"dew":51,"cdew":50.5,"chill":55,"heat":55,"thsw":66,"uv":null,"solar":650,"rainr":0.00000,"storm":0.64000,"rain15":0.00000,"rain1h":0.00000,"raind":0.02000,"rain24":0.29000,"rainmon":5.16000,"rainyear":23.49000,"etday":0.012,"etmon":1.62,"etyear":22.76,"xt":[null,null,null,null,null,null,null],"xlt":[null,null,null,null],"xst":[59,59,55,60],"xh":[null,null,null,null,null,null,null],"xsm":[null,null,3,null],"xlw":[null,null,null,0],"bat":4.78,"trbat":0,"foreico":3,"forrule":192,"sunrt":"6:40","sunst":"18:35","hlbar":[29.746,29.801,"0:15","10:46",30.372,29.507,30.836,29.030],"hlwind":[null,9.0,null,"10:51",23.0,null,45.0,null],"hltempin":[71.1,74.6,"8:30","0:00",91.0,66.0,96.0,50.0],"hlhumin":[42,43,"0:02","0:01",49,37,59,16],"hltempout":[40.9,55.1,"6:51","11:03",79.0,34.0,88.0,-25.0],"hlhumout":[85,100,"10:54","7:37",100,44,100,14],"hldew":[41,51,"5:59","9:48",70,33,78,-30],"hlchil":[41,null,"6:17",null,null,33,null,-32],"hlheat":[null,55,null,"10:57",81,null,99,null],"hlthsw":[null,66,null,"10:57",92,null,109,null],"hlsolar":[null,661,null,"10:49",1208,null,1296,null],"hluv":[null,0.0,null,null,0.0,null,0.0,null],"hlrainr":[0.000,0.080,null,"1:03",3.920,null,4.270,null],"hlxt0":[null,null,null,null,null,null,null,null],"hlxt1":[null,null,null,null,null,null,null,null],"hlxt2":[null,null,null,null,null,null,null,null],"hlxt3":[null,null,null,null,null,null,null,null],"hlxt4":[null,null,null,null,null,null,null,null],"hlxt5":[null,null,null,null,null,null,null,null],"hlxt6":[null,null,null,null,null,null,null,null],"hlxh0":[null,null,null,null,null,null,null,null],"hlxh1":[null,null,null,null,null,null,null,null],"hlxh2":[null,null,null,null,null,null,null,null],"hlxh3":[null,null,null,null,null,null,null,null],"hlxh4":[null,null,null,null,null,null,null,null],"hlxh5":[null,null,null,null,null,null,null,null],"hlxh6":[null,null,null,null,null,null,null,null],"hlxst0":[59,60,"8:46","0:00",63,59,63,59],"hlxst1":[58,59,"0:00","6:11",62,58,62,58],"hlxst2":[53,59,"6:56","0:00",65,47,65,47],"hlxst3":[59,60,"2:40","0:00",164,-27,164,-27],"hlxlt0":[null,null,null,null,null,null,null,null],"hlxlt1":[null,null,null,null,null,null,null,null],"hlxlt2":[null,null,null,null,null,null,null,null],"hlxlt3":[null,null,null,null,null,null,null,null],"hlxsm0":[null,null,"0:00",null,null,null,null,null],"hlxsm1":[null,null,"0:00",null,null,null,null,null],"hlxsm2":[3,3,"0:00","0:00",16,2,16,2],"hlxsm3":[null,null,"0:00",null,null,null,null,null],"hlxlw0":[null,null,"0:00",null,null,null,null,null],"hlxlw1":[null,null,"0:00",null,null,null,null,null],"hlxlw2":[null,null,"0:00",null,null,null,null,null],"hlxlw3":[null,null,"0:00",null,null,null,null,null]}

[bellrichm]

Thanks, that is exactly what I was looking for. Running the code, I’m surprised that you are getting anything… I’ll have to dig a bit deeper.

[tomdotorg]

Thank you. To be clear, all of the whole number part of the values are correct. FWIW, I have one weewx running via this plugin and another that polls the Vantage Pro2 console (it's actually a WifiLogger2 but acts like a Vantage Data Logger) directly. From that second one, the data are all correct.

The MQTT topic is world-readable if you want to play with it.

[bellrichm]

Yes, that would be helpful Also, for completeness, what versions of MQTTSubscribe, WeeWX, and python? Thanks. Something strange in my environment

[tomdotorg]

If I recall correctly, I was using WeeWX-MQTTSubscribe-development after you (thank you!) created the nested structure I am using. Weewx 4.8 from the Docker image mitct02/weewx:4.8.0, which uses Python 3.6.9.

[bellrichm]

Well this is embarrassing… I wonder if I broke it after that. Could you attach/send me what you are running? Thanks

[bellrichm]

Meaning MQTTSubscribe.py

[tomdotorg]

Been there, done that. Here is what I am running.

MQTTSubscribe.py.zip

[bellrichm]

Still trying to figure this out. It does seem the released version is broken. When I use the version you have, it seems to pass the value to WeeWX as an int. I doubt it is this, but wonder if something is happening when WeeWX creates the archive record. I think the fastest way forward would be for me to directly subscribe to your data

[tomdotorg]

It will be interesting if our data differed after subscribing to the same data. Mine are available at vtwx-dev.tom.org and the production version (same site, just no MQTT in the mix) is vtwx.tom.org if that helps.

[bellrichm]

Another option/data point, if you run from the command line, what do the loop packets and archive records look like?

[tomdotorg]

If I run what from the command line?

[bellrichm]

Weewx… but if I can subscribe to your broker that is probably easiest Small world, Im on the other side of VT -essex

[bellrichm]

Fyi, http://weewx.com/docs/usersguide.htm#Running_directly

[tomdotorg]

Thanks. I didn't know if WeeWX-MQTTSubscribe had some diagnostic tool or something I should try. Sorry if I wasn't making sense. BTW, both of the sites I mentioned use the Belchertown plugins, so they essentially emit their loop packets to the same MQTT server under the loop topic for that site (vt (generic) vs vt-dev (MQTT)).

[bellrichm]

And head-slap her. I just realized you had posted your broker and topic. I’ve got all I need. I’ll start looking into it

[tomdotorg]

I really appreciate it!

[bellrichm]

Tom, Well I think I have figured out what is going on.

1. The released version of MQTTSubscribe does not work with array data. I am in the process of trying to figure out what I was trying to accomplish and fix it.
2. I believe the reason that you are seeing the decimal places is due to the different way the logger is interacting with WeeWX. I think that the logger that interacts directly with the console is running in 'hardware generation mode'. It pulls the archive directly from the console and puts it in the DB. The one that is interacting via MQTT is running in 'software generation mode'. MQTTSubscribe creates loop packets from every MQTT message. WeeWX then creates archive records from these. For temperature it is the average of the temperature in all the loop packets (MQTT messages)

Using the version of MQTTSubscribe you have, I ran weewx from the command line and watched the data. The temperature in the loop packets was integers but in the archive record it was float. Temp4 loop packets fluctuated between 59 and 60 and the archive record had temp4 with some set of decimal points. The other temps did not fluctuate and the archive record had a decimal point of .0.

If my theory is correct (I am not positive about the running in 'hardware generation), I don't think there is much MQTTSubscribe can do. rich

[tomdotorg]

That all makes sense to me. Since there is no archive record published by the WifiLogger2, we are left to synthesize them.

I'll look at options in weewx to strip the decimals. Thanks, Rich.

[bellrichm]

Off the top of my head, a service bound to the archive record might be the easiest. Let me know if you need anything else.

[tomdotorg]

Now that I understand it, I am going to leave it. If it is calculating in intra-archive period mean, that's fine. I thought it was just some random garbage or something. SoilTemp4 is the water (pond) temperature and changes much more frequently than SoilTemp1-3 which are ground temps at 5', 3', and 1'.

I am interested in knowing when I should move to a production version of MQTT-Subscribe that supports nested and arrays like I am using.

Thanks as always.

-- Tom Mitchell @.*** Real Time Beverly, MA Weather http://weather.tom.org

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts."

• Bertrand Russell

On Thu, Sep 29, 2022 at 7:57 AM Rich Bell @.***> wrote:

Off the top of my head, a service bound to the archive record might be the easiest. Let me know if you need anything else.

— Reply to this email directly, view it on GitHub https://github.com/bellrichm/WeeWX-MQTTSubscribe/issues/166#issuecomment-1262173400, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAM2ERFHQA46TTM55PEKDTWAV7ZNANCNFSM6AAAAAAQWEPYHM . You are receiving this because you authored the thread.Message ID: @.***>

[bellrichm]

I’ll create a release and let you know.

[tomdotorg]

Sounds great. FWIW, I have another matrix field, SM3, which is being transmitted from the WifiLogger2 but is not showing up in loop packets:

    [[[[xsm]]]]
        [[[[[subfields]]]]]
             [[[[[[soilMoist1]]]]]]
             [[[[[[soilMoist2]]]]]]
             [[[[[[soilMoist3]]]]]]
                ignore = false # opt back in for this field
             [[[[[[soilMoist4]]]]]]

[bellrichm]

Yeah, I can see the problem with soilMoist3. I’ve been a bit distracted getting the build/test pipeline to support Python 3.10 and some AppVeyor idiosyncrasies. I’ve got the build/test work done and a good start on fixing this. I’ll let you know when I have something.

[tomdotorg]

Great. No rush from my side. Appreciate your work.

[bellrichm]

Tom, Unless I messed something up again, 2.2.1 should be good to go.

[tomdotorg]

Thanks. This works, but soilMoist3 isn't showing up. Do I have it configured correctly or are we still waiting on handling that?

    [[[[xsm]]]]
        [[[[[subfields]]]]]
             [[[[[[soilMoist1]]]]]]
             [[[[[[soilMoist2]]]]]]
             [[[[[[soilMoist3]]]]]]
                ignore = false # opt back in for this field
             [[[[[[soilMoist4]]]]]]

[bellrichm]

My bad, I see the bug.