fix one cause for GammaUnknownError: calloc_2d: memory allocation error for data values

MarkusZehner commented 9 months ago

The following is caused by only no-data values in an S1 scene over the DEM/Area of interest to process the data. Using the geometry of the scene's metadata, as in #185, instead of the footprint, would not consider those scenes in a select, reducing unnecessary data processing up to the shell commands below.

which operating system are you using?
Centos
which environment is pyroSAR running in?
Micromamba env
which version of pyroSAR are you using?
0.22.1

which function of pyroSAR did you call with which parameters?

geocode(scene = f, 
            dem = dem,
            outdir = outdir,
            spacing = 10, 
            scaling = 'db', 
            export_extra = ['ls_map_geo', 'pix_area_gamma0_geo'], 
            tmpdir = '/work/tempdir' + site_name, 
            osvdir = '/work/osvs')

GAMMA 20221129
the full error message

 File "/work/dems/process_data.py", line 52, in <module>
    geocode(scene = f, 
    ^^^^^^^^^^^^^^^^^^
  File "/home/micromamba/envs/s1proc/lib/python3.12/site-packages/pyroSAR/gamma/util.py", line 835, in geocode
    gc_map_wrap(image=reference, namespace=n, dem=dem, spacing=spacing, exist_ok=exist_ok,
  File "/home/micromamba/envs/s1proc/lib/python3.12/site-packages/pyroSAR/gamma/util.py", line 1477, in gc_map_wrap
    diff.gc_map2(**gc_map_args)
  File "/home/.pyrosar/gammaparse/diff.py", line 1421, in gc_map2
    process(cmd, logpath=logpath, outdir=outdir, shellscript=shellscript)
  File "/home/micromamba/envs/s1proc/lib/python3.12/site-packages/pyroSAR/gamma/auxil.py", line 360, in process
    gammaErrorHandler(out, err)
  File "/home/micromamba/envs/s1proc/lib/python3.12/site-packages/pyroSAR/gamma/error.py", line 74, in gammaErrorHandler
    raise GammaUnknownError(err_out)
pyroSAR.gamma.error.GammaUnknownError: calloc_2d: memory allocation error for data values

shell commands:

# this script was created automatically by pyroSAR on Wed Nov 29 13:00:45 2023

export base=/work/tempdirAG/S1A__IW___A_20171119T171511
export GAMMA_HOME=/cluster/apps/GAMMA_SOFTWARE/20221129

$GAMMA_HOME/ISP/bin/par_S1_GRD $base/S1A_IW_GRDH_1SDV_20171119T171511_20171119T171536_019337_020C59_1951.SAFE/measurement/s1a-iw-grd-vh-20171119t171511-20171119t171536-019337-020c59-002.tiff $base/S1A_IW_GRDH_1SDV_20171119T171511_20171119T171536_019337_020C59_1951.SAFE/annotation/s1a-iw-grd-vh-20171119t171511-20171119t171536-019337-020c59-002.xml $base/S1A_IW_GRDH_1SDV_20171119T171511_20171119T171536_019337_020C59_1951.SAFE/annotation/calibration/calibration-s1a-iw-grd-vh-20171119t171511-20171119t171536-019337-020c59-002.xml - $base/S1A__IW___A_20171119T171511_VH_grd.par $base/S1A__IW___A_20171119T171511_VH_grd - - - - - 2 - -

$GAMMA_HOME/ISP/bin/par_S1_GRD $base/S1A_IW_GRDH_1SDV_20171119T171511_20171119T171536_019337_020C59_1951.SAFE/measurement/s1a-iw-grd-vv-20171119t171511-20171119t171536-019337-020c59-001.tiff $base/S1A_IW_GRDH_1SDV_20171119T171511_20171119T171536_019337_020C59_1951.SAFE/annotation/s1a-iw-grd-vv-20171119t171511-20171119t171536-019337-020c59-001.xml $base/S1A_IW_GRDH_1SDV_20171119T171511_20171119T171536_019337_020C59_1951.SAFE/annotation/calibration/calibration-s1a-iw-grd-vv-20171119t171511-20171119t171536-019337-020c59-001.xml - $base/S1A__IW___A_20171119T171511_VV_grd.par $base/S1A__IW___A_20171119T171511_VV_grd - - - - - 2 - -

$GAMMA_HOME/ISP/bin/S1_OPOD_vec $base/S1A__IW___A_20171119T171511_VH_grd.par $base/osv/S1A_OPER_AUX_POEORB_OPOD_20210305T045727_V20171118T225942_20171120T005942.EOF -

$GAMMA_HOME/ISP/bin/S1_OPOD_vec $base/S1A__IW___A_20171119T171511_VV_grd.par $base/osv/S1A_OPER_AUX_POEORB_OPOD_20210305T045727_V20171118T225942_20171120T005942.EOF -

$GAMMA_HOME/ISP/bin/multi_look_MLI $base/S1A__IW___A_20171119T171511_VH_grd $base/S1A__IW___A_20171119T171511_VH_grd.par $base/S1A__IW___A_20171119T171511_VH_grd_mli $base/S1A__IW___A_20171119T171511_VH_grd_mli.par 1 1 - - - -

$GAMMA_HOME/ISP/bin/multi_look_MLI $base/S1A__IW___A_20171119T171511_VV_grd $base/S1A__IW___A_20171119T171511_VV_grd.par $base/S1A__IW___A_20171119T171511_VV_grd_mli $base/S1A__IW___A_20171119T171511_VV_grd_mli.par 1 1 - - - -

$GAMMA_HOME/DIFF/bin/gc_map2 $base/S1A__IW___A_20171119T171511_VH_grd_mli.par /work/dems/AG_2019_p50_3x3_corr_swap.par /work/dems/AG_2019_p50_3x3_corr_swap $base/S1A__IW___A_20171119T171511_dem_seg_geo.par $base/S1A__IW___A_20171119T171511_dem_seg_geo $base/S1A__IW___A_20171119T171511_lut_init 0.99959192 0.99959192 $base/S1A__IW___A_20171119T171511_ls_map_geo - $base/S1A__IW___A_20171119T171511_inc_geo - - - - - - - - - - 8 - - - -

MarkusZehner commented 9 months ago

I see a problem with the solution in #185 as there is a backwards incompatability if the geometry column is simply added. I tried to work around this via checks and the special flag to use the geometry. but this is already required at creation of the archive.

Another idea is to just add a refinement to the select method:

find scenes by the footprint and
get only those geometry on the fly from stored metadata and refine search.

The second option would leave the database as-is and reduce the needed ajustments to a read method for the geometry and a call within the select funtion, See #282. What do you think?

johntruckenbrodt commented 4 months ago

I guess the issue is not really closed yet but is being addressed in https://github.com/johntruckenbrodt/pyroSAR/pull/296.

johntruckenbrodt / pyroSAR

fix one cause for GammaUnknownError: calloc_2d: memory allocation error for data values #281