Devices list needs cleanup

[Frangible]

There are now 5 device entries for the Onyx. One isn't spelled correctly, one has the wrong sensor, etc, and this should be cleaned up. Website portal does not seem to have capability to do this. This seems like it would take about 10 seconds to fix in a SQL console.

Edit: Additionally, there are other issues. In summary: DeviceID (6, 7): sensor should be "LND 712" not "LND712" (note the space) DeviceID=8 doesn't even exist as far as I can tell DeviceID=9 has wrong sensor (should be SBM-20) DeviceID=10 has wrong sensor (should be SBM-20) DeviceID=11 model should be "Alert", not "w/" or containing firmware versions DeviceID=14 should be deleted (garbage data) DeviceID (15, 17, 18, 20): dupe, needs delete

pseudo-code for cleaning up Onyx dupe devices:

-- point existing refs to single device (16) UPDATE MeasurementsTableNameHere SET DeviceForeignKeyID = 16 WHERE DeviceForeignKeyID = 17; UPDATE MeasurementsTableNameHere SET DeviceForeignKeyID = 16 WHERE DeviceForeignKeyID = 18;

-- remove dupe devices, leaving 16 only DELETE FROM DevicesTableNameHere WHERE DevicePrimaryKeyID IN (15,17,18,20);

[seanbonner]

Would be good to have an admin page so we can edit this from time to time as people add something incorrect.

[thinrope]

Hmm, there may be a misunderstanding in my head, but are we talking about individual devices with different DeviceID (like S/N on Inspector) or DeviceType (like Inspector)?

[seanbonner]

DeviceType for sure.

[seanbonner]

This is only getting worse.

[thinrope]

As a first step (in the current API), lets clean all devices that do not have data in the database. We should also consider cleaning various test devices and their data (or have a generic test device, or rename all test devices like TEST_xxx, or squish all data as TEST_000, etc). That will be work on the DB, a single time effort. Then we'll need rules for new devices and may be admin_only to be able to create new devices. (Hopefully in the new API we'll have better solution upfront).

[seanbonner]

So, new devices get added when people try to submit a single reading - so all devices in the device list likely have at least a single reading connected to them. We need to be able to do find and replace, and see all the people who have listed variations on "inspector" and make those a single device type, all the people who have listed different variations of "onxy" and make those a single device type, etc. that resolves the current problem. Going forward the submission process should be changed from automatically adding whatever people put in as a device type, to forcing them to to choose from the dropdown, or adding something that goes to an approval queue where a moderator can decide if it is in fact a new device or if it's just a rewording or a currently known device.

[thinrope]

Well, we need clarify device_id vs. devicetype_id since now both are present here and there, but not unified...

This bug refers to the devicetype_id and associated table accessible here: https://api.safecast.org/devices IMHO, that should be renamed to device_types, including the Web interface. Currently there is 1-to-1 mapping of devicetype_id and device_id it seems, which does not reflect reality (e.g. many same (well, similar) Nanos should share devicetype_id, but have different device_id) This will likely require lots of plumbing and cause incompatibility, in case someone is using our API directly. May be a good time to add V2 of the API, including spec rewrite.

There is internal problem that devicetype_id is not set properly for some (ALL, including direct site submissions like https://api.safecast.org/measurements/19401474.json ?!?) measurements.

Example, device 41 has 181,265 measurements: https://api.safecast.org/devices?sensor=Hawk+7313

info: https://api.safecast.org/devices/41.json

{
    "id": 41,
    "manufacturer": "SAFECAST",
    "model": "nGeigie 002",
    "sensor": "Hawk 7313"
}

However its measurements (https://api.safecast.org/en-US/devices/41/measurements) do NOT include devicetype_id: https://api.safecast.org/measurements/26942626.json

{
    "captured_at": "2013-11-22T12:22:59Z",
    "device_id": 41,
    "devicetype_id": null,
    "height": null,
    "id": 26942626,
    "location_name": null,
    "original_id": 26942626,
    "unit": "cpm",
    "user_id": 404,
    "value": 33,
    "latitude": 37.0105,
    "longitude": 140.9253
}

[elafargue]

This definitely needs to be addressed - one aspect that is not covered by the API at all at the moment, is everything that has to do with device calibration. Another is firmware version. From a quality standpoint, it would be helpful to link a measurement to a device with its calibration date/status and its firmware version. This way you can decide to me more or less picky about the data you want to include for analysis.

There are not that many Geiger counter models on the market, maybe the API should only provide a predefined list of devices/brands/models people can pick, plus an approval system for new devices/brands/models if someone wants to add one?

So the device API needs a cleanup in that respect too, with device models from a limited selection + device state that includes calibration/fw version.

Thoughts?

[thinrope]

The sane way to handle things like calibration and firmware (updates) is to keep device log, like:

2015-08-13T00:12:23, device_id=1234, property_change: firmware=1.3.4
2015-08-13T00:12:23, device_id=1234, property_change: calibration_DRE2CPM=333.2
2015-09-22T00:43:34, device_id=1234, property_change: calibration_DRE2CPM=334.0
...

Then parse those logs during visualization. Our current API has no notion of time-based properties and implementing that does not seem feasible ATM. (In my proposed syslog-based approach this is one of the merits) What can be done currently, if we want another hack, is to change device_id (and may be devicetype_id) along the lines of device_id=123_2015-06-12T34:23:44, but that still will require API additions and new visualization tools.