Closed bitsofbits closed 7 years ago
if we publish this, we can't publish anonymized classified data afterwards, as they would be trivial to correlate...
@redhog , that's a good point. ☹️ . I guess we need to anonymize this as well before we publish it publically.
@redhog , @seacourtaw : thinking about this a bit more, I think it should be sufficient to just fuzz the end of the ranges. What I've implemented is to put the starting edge of the ranges between the last point before the range and first point of the range (but never more than 10 minutes from the start of the range. Similarly for the end of the ranges. I think that this fuzzing, plus the fact that the ranges lump all contiguous points together should be enough to make correlating the ranges to points hard enough that we shouldn't have to worry about it.
I'm have an implementation sketched out here: https://github.com/GlobalFishingWatch/mussidae/pull/4
@redhog , @seacourtaw I were talking about ways to make training data more flexible and we decided to extract the fishing classification information we have in terms of time ranges. Specifically, CSV files with
mmsi
,start_time
,end_time
, andis_fishing
fields. For example:I started roughing out the implementation of this in https://github.com/GlobalFishingWatch/vessel-scoring/tree/convert-fishing-data-to-ranges, although eventually the code will likely live somewhere else.
This format has several advantages: