search

e-mission / e-mission-docs

Repository for docs and issues. If you need help, please file an issue here. Public conversations are better for open source projects than private email.
https://e-mission.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
15 stars 34 forks source link

Pipeline Issues with duty cycling turned off #1092

Open Abby-Wheelis opened 2 months ago

Abby-Wheelis commented 2 months ago

We got a report from uue who had asked people to turn duty cycling off on their phones, and after that point stopped seeing data on the admin dashboard. After investigating, it seems that under these circumstances we are never converting the background/location entries to background/filtered_location entries so the rest of the pipeline is ignoring them (found a log statement indicating that filter_accuracy disabled, early return).

For a principled fix, we'll need to think deeper about how to handle duty cycling and data collection parameters in general.

For now, @shankari proposed 2 potential changes:

  1. add duty cycling to the dynamic config and read it from there to enable/disable filter_accuracy - which we will do later for some other planned work
  2. add option to admin dash to allow people to download raw data to do independent processing -- this could include adding a toggle for processed/raw in trajectories and pulling background/location or analysis/recreated_location, depending

@TeachMeTW @JGreenlee can you please work on (2) change so we can unblock the folks at uue?

Abby-Wheelis commented 2 months ago

Temp fix for (2) is open at #133, linking the two for tracking!