search

jim-easterbrook / pywws

Python software for USB Wireless WeatherStations
https://pywws.readthedocs.io/
GNU General Public License v2.0
204 stars 63 forks source link

extending pywws for "external" data #12

Closed benehuber closed 11 years ago

benehuber commented 11 years ago

Hi Jim,

what do you think about extending pywws for "external" data, which is provided from other measuring systems where you can retrieve data eg. with a python script or something like that? I'd like to integrate a selfmade Snowhightsensor (sonar) on my pywws graphs and therefore it would by great to store other measurements in pywws datastore for further analysis.

regards Bene

PS: pywws is realy a great piece of work! Thank you!

jim-easterbrook commented 11 years ago

I can see several problems in doing this in a flexible, extensible fashion. 1/ The current data stores don't store the names of the fields in the files - it's implicitly assumed that the expected set of fields are there, in the right order. This would need to be changed. 2/ The 'live log' loop doesn't really have any means of pulling in external data. Given the external data source isn't synchronised to the weather station, some sort of asynchronous queue would be needed to bring it in. Not an insurmountable problem. How would we configure the use of an external data source (in a general way that would be useful for other data sources). 3/ The processing step would need to know what to do with the extra data. You may not need hourly or daily snow summaries, but someone else will want to summarise their data. Another configuration nightmare?

If you only need to pull in external data for graphs then this might be a possible extension to the plotting module. Currently pywws data is copied to ASCII files for plotting by gnuplot. If your snow data logger can write a suitable file, then it could automatically get plotted along with the pywws data. The only synchronisation would be to ensure the file is complete before gnuplot is run (and not deleted until it's finished). All that's needed would be an "external" element in the plot XML file that gives the file name of the data to be plotted.

benehuber commented 11 years ago

Hi,

First of all I have to sy that my snowhight sensor is still in my mind. I'm thinking about a raspberry with I²c Sonar SRF02. Maybe with a small python script it schould be possible to receive a value which represents the snowhight in cm or m.

ad 1) what about a second data store?

ad 2) with issue #9 you will introduce async calls :) after that we/you need a new config section in weather.ini where the external calls are listed and the fieldnames of result with units. eg. ext_call='python snowhight.py' result_name='snowhight' unit='m'

I think the easiest way is to assume that you only get one result back from the external script.

ad 3) why not another config entry in addition to the mentioned above? summarize='hourly,daily,...'

For me plotting alone wouldn't be so interessting! I think it is more important to store the data.

jim-easterbrook commented 11 years ago

Why not just write a separate program to store data from the separate device? Piling this extra complexity into pywws seems to me to be a bad idea. I've tried to keep it simple, not always with much success. I would also have no easy way to test and maintain the additional functionality. Adding extra data to plots, and perhaps to text templates, is something I think is more practical. Feel free to fork pywws and add what you need. If it turns out to be much simpler than I expect, I'll look at merging it back into the main repos.