Merging data from several nodes in Wunderground case

[njbuch]

I have been running with the new feature of having Wunderground gateway collect data and feed it into my influxdb. It works, but I am realizing that its slightly sub-optimal.

As you can see on the funny image, this gateway has exploded into many different nodes with seperate sensor-nodes, making it impossible to make rules and draw graphs without cleaning the data.

I suspect its a feature from wunderground trying to find the most stable sensor in the closest location from the coordinates used in the api. Actually if I insist on a specific weather-station it claims it to be down. So, in this way there is always weatherdata to look at, despite coming from different stations.

[jkandasa]

@njbuch you want to merge all the stations into one node?

[njbuch]

Yes, that would make sense to have the catch-all nodename as an option on the config window...

[jkandasa]

@njbuch added fix for this request. Kindly have a look at SNAPSHOT version.

[njbuch]

Took the SNAPSHOT from 3 days ago, and tested with the new improvements, I see this:

2017-11-13 21:35:47,624 ERROR [Thread-428] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:114] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"unable to parse 'wu_node_Wind,host=mycontroller,location=skur value=From the WSW at 1.2 MPH Gusting to 3.1 MPH 1510605025000000000': invalid boolean"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-13 21:35:47,669 ERROR [Thread-430] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:114] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"unable to parse 'wu_node_Barometer,host=mycontroller,location=skur value=partlycloudy 1510605025000000000': invalid boolean"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-13 21:35:47,671 ERROR [Thread-431] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:114] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"unable to parse 'wu_node_Barometer,host=mycontroller,location=skur value=Partly Cloudy 1510605025000000000': invalid boolean"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-13 21:50:49,778 ERROR [Thread-883] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:114] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"unable to parse 'wu_node_Wind,host=mycontroller,location=skur value=From the SW at 1.9 MPH Gusting to 3.1 MPH 1510606111000000000': invalid boolean"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-13 21:50:49,834 ERROR [Thread-885] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:114] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"unable to parse 'wu_node_Barometer,host=mycontroller,location=skur value=partlycloudy 1510606111000000000': invalid boolean"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-13 21:50:49,836 ERROR [Thread-886] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:114] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"unable to parse 'wu_node_Barometer,host=mycontroller,location=skur value=Partly Cloudy 1510606111000000000': invalid boolean"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086

[jkandasa]

@njbuch looks like in influxdb, value has been defined as boolean. Did you touch influxdb manually?

[njbuch]

No, not at all.

[njbuch]

I am still having problems with the wind, in sending to the external server there was no luck. Its only the direction when I refer to the Wind in Influxdb. I have made there seperate sensors vars transfer to external server:

[jkandasa]

@njbuch issue found on influxdb write query section for string data. fixed it. Now I can see all the data.

> show series
key
---
wu_node_barometer_forecast,mc=local
wu_node_barometer_pressure,mc=local
wu_node_barometer_variable_1,mc=local
wu_node_humidity_humidity,mc=local
wu_node_temperature_temperature,mc=local
wu_node_wind_direction,mc=local
wu_node_wind_text,mc=local
wu_node_wind_wind,mc=local
> select * from wu_node_barometer_forecast
name: wu_node_barometer_forecast
time                mc    value
----                --    -----
1510723505000000000 local mostlycloudy
> select * from wu_node_barometer_pressure
name: wu_node_barometer_pressure
time                mc    value
----                --    -----
1510723505000000000 local 1014
> select * from wu_node_barometer_variable_1
name: wu_node_barometer_variable_1
time                mc    value
----                --    -----
1510723505000000000 local Mostly Cloudy
> select * from wu_node_humidity_humidity
name: wu_node_humidity_humidity
time                mc    value
----                --    -----
1510723505000000000 local 66
> select * from wu_node_temperature_temperature
name: wu_node_temperature_temperature
time                mc    value
----                --    -----
1510723505000000000 local 26.2
1510723806000000000 local 26.4
> select * from wu_node_wind_direction
name: wu_node_wind_direction
time                mc    value
----                --    -----
1510723505000000000 local 293
1510723806000000000 local 0
> select * from wu_node_wind_text
name: wu_node_wind_text
time                mc    value
----                --    -----
1510723505000000000 local From the WNW at 7.0 MPH Gusting to 9.0 MPH
1510723806000000000 local From the North at 4.0 MPH Gusting to 7.0 MPH
> select * from wu_node_wind_wind
name: wu_node_wind_wind
time                mc    value
----                --    -----
1510723505000000000 local 11.3
1510723806000000000 local 6.4
> 

To send all the data, you can use a gateway. No need to use separate entry.

fix is available on SNAPSHOT version.

Where have you hosted influxdb? In RasbperryPI? how is the performance?

[njbuch]

Hi again, its not working for me.

The serverlog:

2017-11-16 20:43:05,420 ERROR [Thread-5345] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:115] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"partial write: field type conflict: input field \"value\" on measurement \"wu_node_Barometer\" is type string, already exists as type float dropped=1"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://<myip>:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://<myip>:8086
2017-11-16 20:43:05,587 ERROR [Thread-5342] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:115] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"partial write: field type conflict: input field \"value\" on measurement \"wu_node_Wind\" is type string, already exists as type float dropped=1"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://192.168.1.197:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://<myip>:8086

Looking at influxdb:

> show series

key

wu_node_Barometer,host=mycontroller,location=skur

wu_node_Humidity,host=mycontroller,location=skur

wu_node_Temperature,host=mycontroller,location=skur

wu_node_Wind,host=mycontroller,location=skur

It does seem to be a type problem... not sure if it is a configuration thing in influx??

[njbuch]

Regarding performance. I am running mycontroller and influxdb on a decent linux-server, so performance is not an issue for me.

Sidenote: It could be nice to have an update script to get the SNAPSHOT version installed easily in a docker container.

[jkandasa]

@njbuch Can you try with a fresh database in influxdb? or drop existing tables?

[njbuch]

I tried to drop the series and the measurements.... did not work.

Found this: https://github.com/influxdata/influxdb/issues/8649

Which indicates a problem with influxdb that seems to keep some typing and data despite series being dropped. I have now dropped the whole database, lets see what happens.

[njbuch]

Nope, it did not work after dropping the whole database. Errors are still:

2017-11-19 07:21:01,114 ERROR [Thread-110740] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:115] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"partial write: field type conflict: input field \"value\" on measurement \"wu_node_Wind\" is type string, already exists as type float dropped=1"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-19 07:21:01,225 ERROR [Thread-110742] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:115] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"partial write: field type conflict: input field \"value\" on measurement \"wu_node_Barometer\" is type string, already exists as type float dropped=1"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-19 07:21:01,281 ERROR [Thread-110743] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:115] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"partial write: field type conflict: input field \"value\" on measurement \"wu_node_Barometer\" is type string, already exists as type float dropped=1"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086
2017-11-19 07:21:01,459 ERROR [Thread-110740] [org.mycontroller.standalone.exernalserver.model.ExternalServerInfluxdb:115] Failed to send data to remote server! Status Code:400, Is Success:false, Error Message:-, Raw Entity:{"error":"partial write: field type conflict: input field \"value\" on measurement \"wu_node_Wind\" is type string, already exists as type float dropped=1"}\n, Entity:[null], Remote server:ExternalServerInfluxdb(url=http://MYIP:8086, trustHostType=ANY, database=mydb, username=admin, tags=host=mycontroller,location=skur), http://MYIP:8086

The four lines repeat for every poll to Wunder.

I am guessing it happens in the first registration of data, when the measurement is placed in an autogenerated schema.

[jkandasa]

@njbuch what is the version of InfluxDB are you using?

[njbuch]

The most recent. Will check exactly when I am back.

[njbuch]

I was using 1.3.5 and now I tried with 1.3.7 which has the same problem....

I have found this page in the documentation: https://docs.influxdata.com/influxdb/v1.3/write_protocols/line_protocol_reference/#example-2-write-the-field-value-1-as-a-float-to-influxdb which might give you some hints as to how the java formats the line.

[jkandasa]

@njbuch let me have a look thank you!

[njbuch]

Any news on this?

[jkandasa]

@njbuch I am unable to reproduce this issue locally. I am using docker influxdb version 1.4.1

[root@jkandasa tmp]# cd /tmp/
[root@jkandasa tmp]# docker run -p 8086:8086 -v $PWD:/var/lib/influxdb influxdb

[njbuch]

Ok, I have upgraded to 1.4.1 and started on a fresh install. Still exactly the same issue. Can you point me to more detailed debugging options?

[jkandasa]

@njbuch can you post your external server page screenshot?

[njbuch]

[jkandasa]

@njbuch can you send this page?

Also about page, to know the version details. I would like to replicate locally. to reproduce this issue. Give me Wunderground gateway page detail too. api ley not required. I can create one.

[jkandasa]

@njbuch Thank you for the screenshots. I identified the problem. On the influxDB configuration, Measurement field should be unique to the field which has a different type of value field.

When we use Measurement field as sensorName, leads problem for other string fields. Also, I request to force Alphabetical key case to Lowercase. Which will make easier when we do a query on influxdb.

Kindly update Influxdb settings as shown here and let me know how it goes.

> show measurements
name: measurements
name
----
wu_node_barometer_forecast
wu_node_barometer_pressure
wu_node_barometer_variable_1
wu_node_humidity_humidity
wu_node_temperature_temperature
wu_node_wind_direction
wu_node_wind_text
wu_node_wind_wind

[njbuch]

It works! Thanks for the effort buddy! Not sure what to do without the sensorname though...

[jkandasa]

@njbuch You can use sensor name, still variableType is required. You may go something like this if you want sensor name. $noceEui_sensorName_$variableType. In this case, I do not see any difference between sensorName and sensorId