Closed wetterfrosch closed 3 years ago
Dear @wetterfrosch,
thanks for reporting this. We will have a look into the issue you are observing.
With kind regards, Andreas.
Just some notes while working on it and following your method of installing directly from the Git repository.
The full command to install the wetterdienst
package including the optional influxdb
"extra" is:
pip install git+https://github.com/earthobservations/wetterdienst#egg=wetterdienst[influxdb]
Otherwise, one would have to install this optional "extra package" manually.
Some packages have optional extras. You can tell pip to install these by specifying the extra in brackets.
-- https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-extras
The full command to install the
wetterdienst
package including the optionalinfluxdb
"extra" is:pip install git+https://github.com/earthobservations/wetterdienst#egg=wetterdienst[influxdb]
Thank, I pip uninstall
ed wetterdienst an reinstalled it this way; error keeps on an is the same.
I have been able to trace the error back to the optional "tidying" of the data introduced by @gutzbenj with https://github.com/earthobservations/wetterdienst/pull/118. While I introduced the command line option to toggle this data reshaping on/off with https://github.com/earthobservations/wetterdienst/pull/210 and also supported it with appropriate tests for both cases,
I didn't test both options together with the InfluxDB export feature and unfortunately, we don't ride the unicorn of having full integration tests for this matter yet. So, this might well be an opportunity to introduce that into the mix in order to improve on the QA side.
When explicitly using the --tidy
option, things start working again.
wetterdienst dwd readings --parameter=air_temperature --resolution=10_minutes --period=recent --station=1048,4411 --date=2020-10/2020-11 --target="influxdb://localhost:8086/?database=dwd&table=weather" --tidy
When explicitly using the
--tidy
option, things start working again.
Now, you might actually ask about the difference, right?
You can investigate what you will get from that by invoking the command with and without the --tidy
option and look at the output to stdout.
wetterdienst dwd readings \
--parameter=air_temperature --resolution=10_minutes --period=recent \
--station=1048,4411 --date=2020-10/2020-11
vs.
wetterdienst dwd readings \
--parameter=air_temperature --resolution=10_minutes --period=recent \
--station=1048,4411 --date=2020-10/2020-11 \
--tidy
In fact, this is a topic I already wanted to discuss with you. While the former invocation without --tidy
will output the data in the accustomed tabular format, @gutzbenj had very good intentions on reshaping the data into the other format:
I added a "tidy_data" option that reshapes the returned DataFrame properly so that one line has now a datetime, value and corresponding quality flag. (https://github.com/earthobservations/wetterdienst/pull/118)
You might even appreciate this format, because it will yield the respective quality field attached to each single datapoint. You might know already that some datasets of DWD have multiple fields binding to different quality designations within the very same dataset. Providing data in this format clearly resolves that issue, so that each data point has the very same shape.
{
"station_id":1048,
"date":"2020-11-11T14:10:00.000Z",
"parameter":"temperature_air",
"element":"temperature_air_005",
"value":6.1,
"quality":3
}
When explicitly using the
--tidy
option, things start working again.
Wow, that just works this way!
You might even appreciate this format [because] each data point has the very same shape.
Now, we will be excited to hear if you also appreciate this format and if you will be able to make the fine Grafana dashboards work with that as good as before, or maybe even better.
If you still prefer the tabular format, please let me know. I will see whether I can provide a fix to be able to ingest data into InfluxDB using the non-tidy data shape as it worked before introducing that.
Let's keep this issue open until this is resolved and the data export feature will be able to handle both shapes again. Thanks!
Describe the bug Can't push CDC data to InfluxDB. Tested with
hourly
and10_minutes
resolutions ofair_temperature
. JSON export looks fine.To Reproduce Using InfluxDB v1.7.9.
installiert mittels
Expected behavior A happy push of the data to InfluxDB.
Desktop (please complete the following information):