search

Safecast / safecastapi

The app that powers api.safecast.org
44 stars 25 forks source link

Handle bGeigieNano drives with mostly good data, but some bad lines/sections #330

Open robouden opened 7 years ago

robouden commented 7 years ago

We need the errors to be visible at the individual drive level, but if you do this right, you want to “spool” all rejected measurements to a separate table and attach an error code why it got rejected. Then we can pull that anytime and format it for a drive (+ some stuff computed at drive load time) More importantly ,we then can run analytics across rejected drive measurements and look for patterns. That then can be turned into a console so we can daily see if any issues are happening in the field.

Easiest in my point of view would be to take the filters from Databot and apply them to the bGiegieNano import filter.

Extension of these issues:

eitoball commented 7 years ago

Bad data example: https://api.safecast.org/en-US/bgeigie_imports/25414

matschaffer commented 7 years ago

Some slack conversation around bad drives starting at https://safecast.slack.com/archives/api/p1475069122000051

eitoball commented 7 years ago

@robouden In #8 case, we are currently ignoring such corrupted lines and processing other lines. Should we process such case as we do now and "spool to separate table"? If so, should we relate spooled data and bGeigie import?

eitoball commented 7 years ago

@robouden How is #165 related to this issue?

matschaffer commented 7 years ago

Assigning to @jsidd since he mentioned trying to give people better info about bad uploads.

robouden commented 7 years ago

Some errors in log could be easy detected if they are no ASCII or wrong date format for example:

below "20L&" is GPS errors: that would completely stop the import instead of delete the line only. $BMRDD,154,20L&-12-14T01:53:10Z,59,4,9662,A,3407.3623,N,13431.3803,E,8.9,A,0.97,206 $BMRDD,154,20L&-12-14T01:57:27Z,46,3,9851,A,3407.5267,N,13431.1781,E,9.6,A,0.97,20A