search

HorizonsRC / hydrobot

Python package providing tools for the automation of data processing.
Other
0 stars 0 forks source link

Hybrid workflow documentation #54

Open nicmostert opened 5 months ago

nicmostert commented 5 months ago

We need to fully document the hybrid workflow. For that we need to make sure it is stable and well tested. The current workflow steps are something like this:

  1. Create/activate venv
  2. Install/update hydrobot and deps
  3. Obtain config info for site/measurement combo. Update ranges and such.
  4. Obtain script template which does the following steps:
  5. Set up annalist for logging
  6. Create processor object to obtain data
  7. Run the regular stuff: clip, spike removal, gap handling
  8. Quality encode
  9. Export
  10. Load up in Hilltop Manager for final check and to copy it to the appropriate archive or whatever.

Keep in mind that this is meant to be a hybrid workflow, so it must work piece-wise. By this I mean that you should be able to split it up into multiple steps, e.g:

Hydrobot for clipping and spike removal -> export to Manager for gap filling and ramping and such -> Back into Hydrobot for Quality Encoding -> export to Manager for final archive transformation etc.

Remember to test and document the auditing trail process!

Important: We found an "upstream" bug in Hilltop. When importing/opening data from xml, the audit trail seems to break. After import, Manager needs to be restarted before continuing processing in Manager

nicmostert commented 4 months ago

Removing this from the milestone due to time constraints. Promise we'll do this soon. Very soon. This is very important, we know.