search

opendata-stuttgart / feinstaub-api

Django project to store sensor pushed via REST api
62 stars 28 forks source link

lots of missing data in the CSV-files on http://archive.luftdaten.info #26

Closed djim1 closed 3 years ago

djim1 commented 4 years ago

For the 'Luchtdata project' in Arnhem (the Netherlands) we use the data from Madavi.de and archive.luftdaten.info for some analyzes. Now we see a lot of gaps between measurements in the daily CSV dumps on http://archive.luftdaten.info/ and also on Madavi.de.

we see big gaps, for example on 8-2-2020 between aprox. 9 and 11:15 and on 12-10-2020 between 01:15 and 02:30 for all sensors. And then there are a lot of sensors with smaller (but more frequently) interruptions from 5 to 15 minuuts (or more). We thought the time between each measurement was/is approximately 145/150 seconds.

Is this a known issue and why so many interruptions between meassurements? This problem seems to be getting bigger the last days/weeks. Is this a database/server issue or maybe a sending problem of the sensors?

Alsow the monthly data from last January is missing, is this on his way? afbeelding

ricki-z commented 4 years ago

The data for January at Madavi.de should be available now. It was a missing directory for 2020. So the zipped files couldn't be transfered to the place where the script was looking for these files. The gaps at Madavi.de should also be smaller now. This server wasn't meant to receive the data of more than 10.000 sensors. The gap on 02-10-2020 at the archive was caused by a server OS update.

djim1 commented 4 years ago

thanks again for your responce @ricki-z The problem is on the server/database site and not on de sensro site....the gap on 12-02-2020 was caused by a server update, what about the other gap on 8-2-2020, also server issues? hopefully the server is getting more stable.

Then we still have the same problem with finding a link between the sensorID and mapID. If the Madavi database/server is not so reliable (not made for receiving data of more than 10000 sensors) and we can't make a match from sensorID (used in URL by Madavi database) to mapID (used in url on archive.luftdaten) we have a problem to track our participants their data (we only have the sensroID's of the participants).

When you have a suggestion to solve this we like to here from you. For now it is clear to use the luftdaten data preferably.

ricki-z commented 4 years ago

All users should have access to their sensorID and the ID used my Luftdaten.info/Sensor.Community. Either they got those ID sent as email at registration or they can look it up at my.luftdaten.info if they registered their sensor there.

djim1 commented 4 years ago

@ricki-z we know, but don't alway's have good contact/response with all of our participants after the workshop. That's why we prefere to have a link from sensorID to the ID used by Luftdaten. But we untherstand that is what is missing/not available.

Now we know this, we have to make sure that we also ask for that ID as well.

thanks for your response, information and help.

succes with this fantastic project!!!

ricki-z commented 4 years ago

To the IDs: The chipID is part of the MAC address of the WiFi interface. Together with the manufacturer prefix you could scan for a device with a wifi scanner. So if you look for a special sensor on the map you would have an approximate position. If you could map this to the chipID you could easily find this sensor from there. Depending on the country this wouldn't be a good thing. Also some people may have a problem with that kind of project and could try to "remove" sensors this way.