opendata-stuttgart / sensors-software

sourcecode for reading sensor data
575 stars 313 forks source link

Luftdaten.info registration #257

Open LuchtwachtersDelft opened 6 years ago

LuchtwachtersDelft commented 6 years ago

Is this the place to file issues related to the Luftdaten.info website?

A lot of workshop participants told me their registration attempts for the Luftdaten.info map aren't going through. They sent e-mails as specified on the Dutch site and also tried the form on the German site, but didn't get added.

They prefer a dashboard registration system like openSenseMap, which gives immediate feedback and adds their sensors instantaneously to the map.

ricki-z commented 6 years ago

They are added, but at the moment this may need some time (up to three days). Some of the registration attempts are marked as spam (even with the form) and we don't look in the spam folder every day. Please tell the people at your workshops that the registration is not just in time at the moment. We are working on a self service portal. We hope to get it working in the next weeks. Then we only need the chipID of the nodeMCU and an email address. All other informations are added by the user via this portal. If this is working as expected we will add the new sensors automatically. But we need to check even this informations. What is the problem? An example: We ask for the height above ground/street, but in 1 out of 10 cases the user enters the the height above sealevel. But its a difference whether the sensor is located at 20m above the steet in a house 10m above sealevel or at 5m above ground in a house 25m above sealevel. I don't know if the opensensemap solution is better at this point.

LuchtwachtersDelft commented 6 years ago

OpenSenseMap doesn't do any validation. You can leave the height field blank or fill in a random number. It doesn't check whether it's actually a GPS Height number.

image

It could help people enter more valid data by adding an explanation with illustrations in the form. I could also imagine a system that converts height above ground level to above mean sea level with the supplied longitude and latitude.

The indoor/outdoor setting could be more difficult to check beforehand. I found one case where the sensor was labeled outdoors. With PM2.5 readings often in the hundreds that sensor was a huge outlier. When I asked it turned out the user intended to hang the sensor outside, but in the meantime he put it indoors while having a frequent source of significant PM.

Cirromulus commented 5 years ago

I just wanted to push this topic, my friends and I are also not coming through the registration process.

blassepl commented 5 years ago

Hi, I see this is still the case, as I'm waiting over two weeks for registration now. A suggestion: would be good to get confirmation that submission was received successfully. It could be some simple autoresponder. This way we would at least knew, that submission is there (and not in the spam folder or some limbo).

romeok01 commented 5 years ago

I am also waiting to add a sensor when it will automatically add a sensor?

Cirromulus commented 5 years ago

It wont automatically add a sensor. It is a manual process of the maintainers

Adorfer commented 5 years ago

are registration of PMS5003/7003 still banned on Luftdaten info? Or are they allowed meanwhile?

bertrik commented 5 years ago

I have a PMS7003 based sensor and it was accepted.

stealth-ultd commented 5 years ago

My attempt at registering a sensor seems stuck in no-mans land for about two weeks now. Sent also an e-mail that remains unanswered. I can surely live with the fact that the team has other urgencies than adding piles of new sensors, so simply a confirmation message that the form has gone through would be very welcome. Could avoid a flurry of unnecessary correspondence in following up on the status of registration of individual sensors. Besides that, the sensor is working happily!

ropeters68 commented 5 years ago

Maybe change the text on the site 3 days --> 3 weeks.

Cirromulus commented 5 years ago

Maybe change the text on the site 3 days --> 3 weeks.

lol, waiting for over a month now :+1:

bertrik commented 5 years ago

I know this isn't really anyone's fault, but the long registration time makes it hard to keep the enthusiasm going when you organise a workshop but the registration of the participants takes several weeks.

So I told the people it might take up to a week, maybe two weeks, but for some people it's been over a month. Can we make this easier perhaps? I would welcome a kind of portal where people could enter the data themselves, so they only need approval. I noticed the German build instructions already features a form (the Dutch-language does not yet), so I guess that would save having to copy-paste the data or at least get the relevant data in a standardized format.

So what I appreciate very much:

poinck commented 5 years ago

I have tried to register via the mentioned form 2 weeks ago, but my sensor is still not on the map. I am not sure whether my data got through. So I eventually registered a second and a third time. But still nothing. Gladly openSenseMap works for me and many other Luftdaten-Feinstaubsensors are there, too, so that I can compare and validate my sensor values. But, as mentioned above, it would be very great to have the sensor on the Luftdaten.info map, too. (:

Edit 21.3.2019: [/impatience-switched-off]

ricki-z commented 5 years ago

And yjust another one who haven't read the page after sending the form ... The registration was received. This is a unsalaried project. We do all the work in our leisure time. And we haven't received any money for developing this like i.e. OpenSenseMap. So there may be a longer delay if we have less available time.

poinck commented 5 years ago

@ricki-z I am sorry, I did not intend to rush things here. I can wait. Running the sensor is leisure for me as well. Take your time.

ghost commented 5 years ago

have recently used the self registration process and my sensor appeared on the site within a few minutes. I was very impressed.

ricki-z commented 5 years ago

Since Easter our new self registration portal is online. So you can now register your sensors at https://meine.luftdaten.info/ .

RikDrabs commented 5 years ago

This is now the third time, that a registration attempt at the registration portal sends the message " This sensor ID is already registered. " Since i did not register it myself, and the user for whom i'm registering has not registered himself, and his sensor is nowhere near to his location on the map where he lives and has his sensor mounted, the only other explanation is that somebody registered a - for himself - wrong sensor number, or that the way sensor number are "calculated" by the sensor, does not produce unique numbers all the time. How to solve this? Till now i replaced the MCU with a fresh one, and threw the "double" in the bin. this is the number now: 3873513 We have no way to read the street address of this sensor - for privacy reasons - but this is contrary to the open source - open data - open hardware concept !! So we have no way to contact owners of sensors registered in error, owners of defective sensors (showing erroneous data on the map, etc. What do you propose? Rik Drabs - volunteer luchtpijp.be (contact info@luchtpijp.be)

ricki-z commented 5 years ago

We use the ESP8266 chipID which is part of the MAC address of the wifi interface. This number should be unique. Until now we had only around 10 (out of 10.000 active) devices where users couldn'd register their device because of a duplicate ID and told us this. And in some of these cases the users only copied the chipID not completely. So is it possible that some of those NodeMCU modules were use before? As you haven't send me the chipID of a NodeMCU that couldn't be registered I can't look for this.

Defective sensors: we are working on a notification system for those sensors. Sorry that a citizen science project (done unsalaried by all participants) can't do all at once!!!!!!!

Addesses: There is a misunderstanding of the term "open data". We publish a defined amount of data. These are the measurements, the location with a defined accuracy and the used sensor type. So why should not publishing the address be contrary to the principle of open data? Please think about people in other countries that may get problems it it would be known that they are participating in our project and that they are sending environmental data to servers outside their country.

RikDrabs commented 5 years ago

In this case the MCU was used before, and the user tried to re-register his already registered sensor. I know this because i specifically asked this to the user, and he confirmed it. However the first registration was a manual one, and so the user nor me know how to access the address data in the record, since we have no account nor password to access that record in the automatic registration system ... Trying to "re-register" an existing manually registered sensor must be facilitated, or at least explained. But this is only one of the possible reasons. Another one is that your automatic system permits to register any sensor number, even one you don't own, even one that is not active. This permits any user to block any other user from legally registering his own sensor, and thus causing the described duplicate message. This is a loophole in the automatic registration process that needs to be closed. Further on i understand to need to protect the user record from prying eyes, as specifically asked in GDPR law, but the same law accepts exceptions, for example if the user has given permission to do so. Your organisation should appoint several other organisations per country involved, each with their own territory, for maintenance of the installed park (mote than 16500 sensors at this point), follow-up of problems, repairing & replacing what is defective, and for that reason being able to trace back from API-id to Chip-id, and from there, trace back to user's contact e-mail address. Approving the specific organisation responsible for the territory of the user should then be part of the registration. I hope i've given you some usefull ideas. By the way, i've designed here in Brussels a nice display, for use to show all measured values, together with qualifying messages, on an array of 16 display blocks 8x8 matrix type LED. Interested to incorporate this display in your project, for free? I can put it on Github, and make it public-domain, but i feel that the connection with your project will make my display more usefull than as a stand-alone project, for everyone searching to display "feinstaub" data !

Op wo 16 okt. 2019 13:36 schreef Rajko Zschiegner <notifications@github.com

:

We use the ESP8266 chipID which is part of the MAC address of the wifi interface. This number should be unique. Until now we had only around 10 (out of 10.000 active) devices where users couldn'd register their device because of a duplicate ID and told us this. And in some of these cases the users only copied the chipID not completely. So is it possible that some of those NodeMCU modules were use before? As you haven't send me the chipID of a NodeMCU that couldn't be registered I can't look for this.

Defective sensors: we are working on a notification system for those sensors. Sorry that a citizen science project (done unsalaried by all participants) can't do all at once!!!!!!!

Addesses: There is a misunderstanding of the term "open data". We publish a defined amount of data. These are the measurements, the location with a defined accuracy and the used sensor type. So why should not publishing the address be contrary to the principle of open data? Please think about people in other countries that may get problems it it would be known that they are participating in our project and that they are sending environmental data to servers outside their country.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opendata-stuttgart/sensors-software/issues/257?email_source=notifications&email_token=AJLIM7ELYPROWK63P2RMKMLQO34DLA5CNFSM4FV4A3QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBME5LA#issuecomment-542658220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLIM7D4WOWY5F33RRVZU63QO34DLANCNFSM4FV4A3QA .

RikDrabs commented 5 years ago

1: One of the reasons for receiving "this sensor is already registered" message is that your registration portal permits to register any chip-id, even one you don't own, and even one that is not (yet) active.

This was the only explanation left in two previous cases. Some user did

register a chip-id owned by a second user, with as result that the second user couldn't register his sensor anymore.

What aggravates this problem further is that the chip-id entered in

error cannot be modified afterwards, nor can the registration in error be deleted.

This is a backdoor in the registration portal that needs to be closed.

And it can: permitting registration only when the sensor is on-line and

active, and not yet registered, would prevent that sensors still "in the box", or even not yet assembled, can be registered by anybody at anytime.

This is a simple and effective modification, that doen't take years to

implement.

2: I understand the need to protect the user record from prying eyes, as specifically obliged by GDPR law, but the same law also accepts exceptions, in case the user has given permission.

Your organisation should appoint other organisations (for example per

country involved), each with their own territory, for maintenance of the installed park (more than 16000 sensors up to this point). They should do the follow-up of problems, repairing & replacing what is defective, and so on. For that reason they should be able to trace back from API-id to user's contact e-mail address, but only if the user has given permission to that organisation to do this, while registering.

Approving the specific organisation responsible for the territory of

the user should then be part of the registration form.

  1. You say that the chip-id is calculated from part of the MAC-address. Should that not be the whole MAC address, since only the whole MAC address is unique ! ! ! Example: If the first character in a number of 3 characters is not accounted for, then 100, 200, 300, 400, etc, are all IDENTICAL ! Only when the whole sequence of 3 is accounted for, are those numbers unique.

    Isn't there a problem when some users who want to use other MCU's like the ESP32, the ESP8266 12F, or the ESP8266-CP2102 Amica, instead of the original ESP8266 Lua CH340G V3 ? Aren't there any overlaps in the part of the MAC that is used, so that DOUBLES CAN EXIST? They are all similar, and can all execute the feinstaub sensor program, some with recompilation, others directly.

I hope i've given you some usefull & logic ideas.

Op wo 16 okt. 2019 om 13:36 schreef Rajko Zschiegner < notifications@github.com>:

We use the ESP8266 chipID which is part of the MAC address of the wifi interface. This number should be unique. Until now we had only around 10 (out of 10.000 active) devices where users couldn'd register their device because of a duplicate ID and told us this. And in some of these cases the users only copied the chipID not completely. So is it possible that some of those NodeMCU modules were use before? As you haven't send me the chipID of a NodeMCU that couldn't be registered I can't look for this.

Defective sensors: we are working on a notification system for those sensors. Sorry that a citizen science project (done unsalaried by all participants) can't do all at once!!!!!!!

Addesses: There is a misunderstanding of the term "open data". We publish a defined amount of data. These are the measurements, the location with a defined accuracy and the used sensor type. So why should not publishing the address be contrary to the principle of open data? Please think about people in other countries that may get problems it it would be known that they are participating in our project and that they are sending environmental data to servers outside their country.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opendata-stuttgart/sensors-software/issues/257?email_source=notifications&email_token=AJLIM7ELYPROWK63P2RMKMLQO34DLA5CNFSM4FV4A3QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBME5LA#issuecomment-542658220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLIM7D4WOWY5F33RRVZU63QO34DLANCNFSM4FV4A3QA .

-- Rik Drabs Vrijwilliger Luchtpijp project van CM / beweging.net

ricki-z commented 5 years ago

At first: We started this project as an attempt to measure air quality in Stuttgart, with only 300 devices. That the project would become that large in such short time no one of us could imagine at that time.

1: Deleting a sensor (i.e. with a wrongly entered ID and other situations) is work in progress. Deleting a sensor is not trivial, as we need some of the informations even the case of a deletion (archiving data some hours or days later). But people should be able to contact us. There are not so many requests regarding this. There will be checks in the near future if we have received any data short after registration. So the registration of "non-existant" chipIDs should be limited. And we check the amount of registration attempts per user account (which needs to be confirmed before a sensor can be registered). This will also limit the chance of a misuse.

Comment to: "This is a simple and effective modification, that doesn't take years to implement." The complete code (excl. the 2 mentioned checks that are done with other tools) of the portal and the API server is open source and published here on Github. So please feel free to add functionality that is missing. We do this project unsalaried in our leisure time. So depending on our available time some things need a little bit longer.

2: Also this is work in progress: We will add an additional field to the device data where users can define an organization that is "supporting" them. This is some kind of giving permission to forward their contact details to that organization.

3: The MAC address contains 3 byte of a vendor specific prefix and 3 byte as a unique number. (This system is used for also for ethernet interfaces). So the MAC addresses of devices of the same vendor should be unique. In our database we use an additional prefix, normally the used MCU/CPU type. You can see some of them in the registration form. So there are different prefixes for Raspi, ESP32, ESP8266, TheThingsNetwork devices and also some alternative firmware versions like "Smogomierz". Raspi has an own cpu serial (which should be unique). The mentioned ESP8266 versions are only different boards with mostly the same ESP8266 module. So the vendor prefix should be identical in all 3 cases and the chipID should be unique.

RikDrabs commented 5 years ago

At first: I'm NOT salaried for this work too. My (passed) career is one between hardware & software: this volunteer work is taking me back to my younger years, designing hardware & the accompanying software, then in assembler for 8085, 8051, Z8, Z80, 6809, the PC with INT21, before going on with C; now on ESP8266 on Arduino IDE in C, for the display I designed for your project.

1) What i meant by: "This is a simple and effective modification, that doen't take years to implement" is the following:

       Permitting registration only when the sensor is on-line and

active, and not yet registered, would prevent that sensors still "in the box", or even not yet assembled, can be registered by anybody at anytime.

The check "not yet registered" is already there, causing the "is already registered" error, with no registration as result.

The check to be added is "sensor is on-line & active" and this should produce a "is not on-line & active" error, with also no registration as result. This will effectively block all registration in error for non-existing sensors, and prevent all cascading problems due to this error entering your database in the first place.

ATTENTION: checking for data coming in after registration - as you suggested - is too late: in case no data arrives proving it is a wrong number, registration is already done, and must be undone in the whole chain you explained, with all complications you mentioned. Errors should be caught at entry, not later, is one of the first things a programmer learns.

2) OK. Do the good work!

3) OK. Your explanation proves that it is not the use of the Amica module that is causing these "already registered" problems. So point 1) is the way to go. Where exactly is that repository?.

Op zo 20 okt. 2019 om 00:47 schreef Rajko Zschiegner < notifications@github.com>:

At first: We started this project as an attempt to measure air quality in Stuttgart, with only 300 devices. That the project would become that large in such short time no one of us could imagine at that time.

1: Deleting a sensor (i.e. with a wrongly entered ID and other situations) is work in progress. Deleting a sensor is not trivial, as we need some of the informations even the case of a deletion (archiving data some hours or days later). But people should be able to contact us. There are not so many requests regarding this. There will be checks in the near future if we have received any data short after registration. So the registration of "non-existant" chipIDs should be limited. And we check the amount of registration attempts per user account (which needs to be confirmed before a sensor can be registered). This will also limit the chance of a misuse.

Comment to: "This is a simple and effective modification, that doesn't take years to implement." The complete code (excl. the 2 mentioned checks that are done with other tools) of the portal and the API server is open source and published here on Github. So please feel free to add functionality that is missing. We do this project unsalaried in our leisure time. So depending on our available time some things need a little bit longer.

2: Also this is work in progress: We will add an additional field to the device data where users can define an organization that is "supporting" them. This is some kind of giving permission to forward their contact details to that organization.

3: The MAC address contains 3 byte of a vendor specific prefix and 3 byte as a unique number. (This system is used for also for ethernet interfaces). So the MAC addresses of devices of the same vendor should be unique. In our database we use an additional prefix, normally the used MCU/CPU type. You can see some of them in the registration form. So there are different prefixes for Raspi, ESP32, ESP8266, TheThingsNetwork devices and also some alternative firmware versions like "Smogomierz". Raspi has an own cpu serial (which should be unique). The mentioned ESP8266 versions are only different boards with mostly the same ESP8266 module. So the vendor prefix should be identical in all 3 cases and the chipID should be unique.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opendata-stuttgart/sensors-software/issues/257?email_source=notifications&email_token=AJLIM7ALUVVBRN3QJYRM6LTQPOE6JA5CNFSM4FV4A3QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBX6FWA#issuecomment-544203480, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLIM7EYQHQMFG4V5H2GBJDQPOE6JANCNFSM4FV4A3QA .

-- Rik Drabs Vrijwilliger Luchtpijp project van CM / beweging.net

ricki-z commented 5 years ago

The API luftdaten.info and the graphs at madavi.de are running on two completely different systems. While the API is a database, madavi.de ist working with single files per sensor. So looking up the registration info is much faster than searching files at madavi.de. Thats why we can show the message "ist not registered at Luftdaten.info". And madavi.de was meant to be used as a test system. So we can't (shouldn't) use the informations saved at this system. API luftdaten.info is denying access if a system isn't registered yet. For your your solution we would need to save every denied access somewhere so that we could look up if a newly registered sensor has sent something before. And this would also make the system vulnerable for spamming "chipIDs". An attacker could send data for a fake sensor (or an unregistered chipID) first and register that sensor afterwards. That would be only one step more to block a chipID. Even the Opensensemap isn't perfect. You can send data "in the name of" every registered sensor from everywhere.

You can find the repos at https://github.com/opendata-stuttgart/ . In the upper part of the page you can find the repos we are working on at the moment.

RikDrabs commented 5 years ago

First:

The API luftdaten.info and the graphs at madavi.de are running on two completely different systems. While the API is a database, madavi.de ist working with single files per sensor. So looking up the registration info is much faster than searching files at madavi.de. Thats why we can show the message "ist not registered at Luftdaten.info".

And madavi.de was meant to be used as a test system. So we can't (shouldn't) use the informations saved at this system.

API luftdaten.info is denying access if a system isn't registered yet. For your your solution we would need to save every denied access somewhere so that we could look up if a newly registered sensor has sent something before.

And this would also make the system vulnerable for spamming "chipIDs". An attacker could send data for a fake sensor (or an unregistered chipID) first and register that sensor afterwards. That would be only one step more to block a chipID. Even the Opensensemap isn't perfect. You can send data "in the name of" every registered sensor from everywhere.

You can find the repos at https://github.com/opendata-stuttgart/ . In the upper part of the page you can find the repos we are working at at the moment.

Best regards, Rik Drabs, volunteer www.luchtpijp.be

Op zo 20 okt. 2019 om 13:53 schreef Rajko Zschiegner < notifications@github.com>:

The API luftdaten.info and the graphs at madavi.de are running on two completely different systems. While the API is a database, madavi.de ist working with single files per sensor. So looking up the registration info is much faster than searching files at madavi.de. Thats why we can show the message "ist not registered at Luftdaten.info". And madavi.de was meant to be used as a test system. So we can't (shouldn't) use the informations saved at this system. API luftdaten.info is denying access if a system isn't registered yet. For your your solution we would need to save every denied access somewhere so that we could look up if a newly registered sensor has sent something before. And this would also make the system vulnerable for spamming "chipIDs". An attacker could send data for a fake sensor (or an unregistered chipID) first and register that sensor afterwards. That would be only one step more to block a chipID. Even the Opensensemap isn't perfect. You can send data "in the name of" every registered sensor from everywhere.

You can find the repos at https://github.com/opendata-stuttgart/ . In the upper part of the page you can find the repos we are working at at the moment.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/opendata-stuttgart/sensors-software/issues/257?email_source=notifications&email_token=AJLIM7HP4SNODQT4DIPVXSDQPRBDZA5CNFSM4FV4A3QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBYIM3A#issuecomment-544245356, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJLIM7CKNEOAWHQEZGGIL6DQPRBDZANCNFSM4FV4A3QA .

-- Rik Drabs Vrijwilliger Luchtpijp project van CM / beweging.net

ricki-z commented 5 years ago

Only a comment just now: All data is archived daily at http://archive.sensor.community/ (or the former address http://archive.luftdaten.info/), but only for registered devices. And only this data should be downloaded, if you need archived data. Madavi.de (as the name may suggest) is a private server. This server may be shut down and the functionality may be implemented on another system in a completely different way. This site was meant to be used for testing freshly installed devices before those are registered as we started the project. And Madavi.de is used only by systems that use our firmware. Systems developed by others may use another way to test them. I.e. some of the scripts for Raspberry Pi don't send any data to Madavi.de. Those systems couldn't be checked this way.

To communicate with the developers of meine.luftdaten.info please use the Github issues. It's not only one person. And not all of the developers would be available at all time to answer your requests (this project isn't done by a company!). But the issues could be read by all.

cpietsch commented 4 years ago

Hello, I registered my Sensor some years ago and now I do not have access to the email address anymore. When trying to re-register the sensor on https://meine.luftdaten.info/sensors/register I get the Error This sensor ID is already registered. What is the best way to recover this sensor and make a new registration ?

ricki-z commented 4 years ago

Hello @cpietsch , you can send an email to rajko (at) sensor . community with the chipID and the email address that was used originally. I will change the entry to the new email address.

myshyak commented 4 years ago

The registration page is broken today. It gives you an error page after submitting email and password, but block the email for retry. And no email are sent as well

ricki-z commented 4 years ago

@myshyak did you send an email to my account?

myshyak commented 4 years ago

I did. Thanks. Now susscrefully registered Sincerely Yours Arseniy Finberg www.interesniy.kiev.ua +(38050) 334-23-48

пн, 18 мая 2020 г. в 16:22, Rajko Zschiegner notifications@github.com:

@myshyak https://github.com/myshyak did you send an email to my account?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opendata-stuttgart/sensors-software/issues/257#issuecomment-630179350, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABF5VPF7SYR5XUQW263YAH3RSEZBHANCNFSM4FV4A3QA .