GeoscienceAustralia / ga_sar_workflow

InSAR processing workflow used by Geoscience Australia
Apache License 2.0
3 stars 0 forks source link

Create HOWTO docs for "Add new sensors" #343

Closed truth-quark closed 2 years ago

truth-quark commented 2 years ago

As of Feb 2022, there's missing docs explaining how to add new sensors. This should be added to documentation to help other developers pick up the project.

The overview of the process (snarfed from Mitch), assuming a sensor called blah:

  1. create an insar/sensors/blah.py for your the sensor that implements METADATA / get_data_swath_info / acquire_source_data (like the other sensors, examples are RSAT2, ALOS)
    • source_path is the path to either the raw_dir of data or an archive file (from the input file list???)
    • dst_dir is the some/dir/path/raw_data/{scene_date} dir
  2. Add it to _sensors in insar/sensors/data.py (and other relevant parts like identify_data_source())
  3. add processing code to insar/process_blah_slc.py (the bash port)
  4. add and implement tests/test_process_blah_slc.py (& add test data if necessary)
    • product_path specifies whatever dir(s) are returned from acquire_source_data(), which may be the root of an extracted archive file. This path can affect file searches (which may be painful to debug on a production machine)
  5. add insar/workflow/luigi/blah.py with tasks for running the processing code (similar to rs2/s1/etc in that module)
  6. in insar/workflow/luigi/multilook.py update CreateMultilook.requires() to add a case for the new sensor (multilook is what depends on the sensor-specific SLC processing tasks)
  7. in insar/workflow/luigi/resume.py update ReprocessSingleSLC.run() for the new sensor
  8. Update insar/project.py
  9. test the production code works on Gadi/a GAMMA enabled system
  10. Ask the InSAR team to verify the results of the real data run
  11. Update the relevant README docs for the new sensor.

ALOS or RS2 are useful as templates (they're the simple/normal cases, whereas - S1 is an exceptional case w/ more complicated code due to subswaths and bursts & it has its own special coregistration).

Parts 4/5 above will be documented to some extent (Mitch has workflow documentation in the works, but there's no reference to the code yet - but can/should add it)

Open questions:

truth-quark commented 2 years ago

To add to the HOWTO: adding test data. The general practice for making fake data is:

Ideally the fake data needs to be relatively small.

truth-quark commented 2 years ago

Closing as solved by #364.