Handle GAMMA version migration

[adeane-ga]

We are currently using GAMMA version 20191203, but do have updated versions sitting on the NCI (20210701 is the latest). This issue is related to planning what needs to be done for version migrations.

Software is stored on the NAS at: T:\ops\insar\SOFTWARE\GAMMA

and on GADI: /g/data/dg9/SOFTWARE/dg9-apps/GAMMA

Items for investigation:

• dependencies
• potential GAMMA changes/upgrades/additions
• test data set for migration validation

[mwheeler]

Quickly looking at this now, they seem to have moved from GDAL 2.0 (libgdal.so.20) to 3.0 (libgdal.so.26 - not sure why GDAL "3.0" release is .26 semver binary).

Latest compatible DEA env on NCI is /g/data/v10/public/modules/dea-env/20200713.

[mwheeler]

The DEA environments compatible w/ the latest GAMMA version produce strange OSGEO related errors during DB querying:

Traceback (most recent call last):
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/luigi/worker.py", line 401, in check_complete
    is_complete = task.complete()
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/luigi/task.py", line 822, in complete
    return all(r.complete() for r in flatten(self.requires()))
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/luigi/task.py", line 883, in flatten
    for result in iterator:
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/gamma_insar-0+untagged.1500.g1657b87.dirty-py3.6.egg/insar/scripts/process_gamma.py", line 2521, in requires
    self.sensor
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/gamma_insar-0+untagged.1500.g1657b87.dirty-py3.6.egg/insar/generate_slc_inputs.py", line 254, in query_slc_inputs
    max_date_arg=max_date_arg,
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/gamma_insar-0+untagged.1500.g1657b87.dirty-py3.6.egg/insar/meta_data/s1_slc.py", line 1774, in select_bursts_in_vector
    for extent in spatial_subset.convert2wkt(set3D=False)
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/spatialist/vector.py", line 288, in convert2wkt
    features = self.getfeatures()
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/spatialist/vector.py", line 426, in getfeatures
    features = [x.Clone() for x in self.layer]
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/spatialist/vector.py", line 426, in <listcomp>
    features = [x.Clone() for x in self.layer]
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/osgeo/ogr.py", line 3066, in __iter__
    feature = self.GetNextFeature()
  File "/home/547/mmw547/gamma_insar_install_new/lib/python3.6/site-packages/osgeo/ogr.py", line 1632, in GetNextFeature
    return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError

This (and any bugs that occur after this) will need to be resolved until we can proceed.

[mwheeler]

DEA env related issues have been solved (will make it through in a future PR), however the new GAMMA version has signifiant API breakages, as one small example rasSLC no longer has an LR parameter, which causes errors as all the parameters there after are shifted back one position resulting in an error such as ERROR wr_image(): unsupported raster image format: 0

The first step in an upgrade will require a complete review of every single gamma_insar processing file, every single command, to ensure all parameters are correct since the API has changed / GAMMA API is unstable.

Edit: This also means we can NOT maintain support for multiple GAMMA versions easily - gamma_insar must be tied to a specific version since their API is not stable / backward compatible. If such a need comes up in the future, we will have to maintain multiple versions of processing pipeline paths for each GAMMA version.

[truth-quark]

The first step in an upgrade will require a complete review of every single gamma_insar processing file, every single command, to ensure all parameters are correct since the API has changed / GAMMA API is unstable.

Edit: This also means we can NOT maintain support for multiple GAMMA versions easily - gamma_insar must be tied to a specific version since their API is not stable / backward compatible. If such a need comes up in the future, we will have to maintain multiple versions of processing pipeline paths for each GAMMA version.

GAMMA function signatures changing (and the potential for default args changing) cropped up in this discussion #116. There was a brief discussion on writing some scripts to automate some of this work, but the solution really needs some discussion.

[mwheeler]

The first step in an upgrade will require a complete review of every single gamma_insar processing file, every single command, to ensure all parameters are correct since the API has changed / GAMMA API is unstable. Edit: This also means we can NOT maintain support for multiple GAMMA versions easily - gamma_insar must be tied to a specific version since their API is not stable / backward compatible. If such a need comes up in the future, we will have to maintain multiple versions of processing pipeline paths for each GAMMA version.

GAMMA function signatures changing (and the potential for default args changing) cropped up in this discussion #116. There was a brief discussion on writing some scripts to automate some of this work, but the solution really needs some discussion.

Is there anything that 'can' be discussed? If parameters get removed (eg: this first one we hit was the mirroring LR parameter being dropped), and we rely on that parameter for something... there's no script that can automate the fact mirroring functionality is gone if we used it.

In this first rasSLC case we don't use the removed feature (at least not at this point of failure), but it's a prime example of what the issues are w/ an unstable API - features can be removed at a whim, behaviour could even change w/o the API changing if they've not committed to stability...

Edit: tl;dr - you can't automate away "anything could change in any potential unforseen way"

[truth-quark]

In insar/process_alos_slc.py there is this hardcoded dependency, possibly needing a change when GAMMA is updated:

sensor_antpat = f"{MSP_HOME}/sensors/palsar_ant_20061024.dat"

[mwheeler]

Quick update, we since have an even newer release - 20211201 with even more changes which if/when we decide to upgrade would be the target version.

[mwheeler]

This has been completed at https://github.com/mwheeler/gamma_insar/tree/issues/249 but won't be put into PR, the end goal is #360 which will include this support for 20210701 without dropping support for 20191203