opendata-stuttgart / madavi-api

api for rrdtool graphics per sensors + graphics from archive.luftdaten.info
18 stars 11 forks source link

CSV data files have more fields in data rows than in header row #8

Open dr-1 opened 6 years ago

dr-1 commented 6 years ago

Example file

Time;durP1;ratioP1;P1;durP2;ratioP2;P2;SDS_P1;SDS_P2;Temp;Humidity;BMP_temperature;BMP_pressure;BME280_temperature;BME280_humidity;BME280_pressure;Samples;Min_cycle;Max_cycle;Signal 2017/10/15 00:02:04;;;;;;;19.23;10.80;;;16.50;79.40;;;;;;592692;242;27774;-61

Header row indicates 20 fields; data rows have 22. The two additional fields are between the SDS_P2 and Temp values, which are adjacent in the header. As a result, a correction is needed to parse the files.

I am not sure where this discrepancy comes from. data_simple.php, which writes the files, looks fine:

Line 35, writing 20 header fields fwrite($outfile,"Time;durP1;ratioP1;P1;durP2;ratioP2;P2;SDS_P1;SDS_P2;Temp;Humidity;BMP_temperature;BMP_pressure;BME280_temperature;BME280_humidity;BME280_pressure;Samples;Min_cycle;Max_cycle;Signal\n");

Line 60, writing 20 fields per data row fwrite($outfile,$now.";".$values["durP1"].";".$values["ratioP1"].";".$values["P1"].";".$values["durP2"].";".$values["ratioP2"].";".$values["P2"].";".$values["SDS_P1"].";".$values["SDS_P2"].";".$values["temperature"].";".$values["humidity"].";".$values["BMP_temperature"].";".$values["BMP_pressure"].";".$values["BME280_temperature"].";".$values["BME280_humidity"].";".$values["BME280_pressure"].";".$values["samples"].";".$values["min_micro"].";".$values["max_micro"].";".$values["signal"]."\n");

watty62 commented 6 years ago

I am starting to work on the data for four Aberdeen, UK sensors. See https://github.com/watty62/abdn_air_quality

I have an issue with the extra columns (or too few headers).

Can anyone explain where the problem lies? Pulling the data into data frames for analysis needs manual intervention at the moment. Reading the issue above, it appears that the headers are correct, but that two extra (blank?) data columns are created.

Can the write error be fixed, as noted above, please?

What effect will this have on historic data - will it always have to be corrected once downloaded? Thanks Ian