run_05_invertMisreg error

[saeedmhanna1]

Hello all,

I just want to bring this error back, which was previously referred to in this discussion #488.

I am getting this error message after running the 5th run file (invertMisreg):

I am using stripmapStack to create a stack of ENVISAT images.

Any feedback is greatly appreciated. ` /home/jovyan/Envisat_Hama/refineSecondaryTiming/pairs/20031111_20050208 ['_MutableMappingmarker', 'abstractmethods', 'class', 'class_getitem', 'contains', 'del', 'delattr', 'delitem', 'dict', 'dir', 'doc', 'enter', 'eq', 'exit', 'format', 'ge', 'getattribute', 'getitem', 'getstate', 'gt', 'hash', 'init', 'init_subclass', 'iter', 'le', 'len', 'lt', 'module', 'ne', 'new', 'reduce', 'reduce_ex', 'repr', 'reversed', 'setattr', 'setitem', 'sizeof', 'slots', 'str', 'subclasshook', 'weakref', '_abc_impl', '_protocol', 'cache', 'clear', 'close', 'dict', 'get', 'items', 'keyencoding', 'keys', 'pop', 'popitem', 'setdefault', 'sync', 'update', 'values', 'writeback'] Traceback (most recent call last): File "/home/jovyan/.local/envs/insar_analysis/lib/python3.11/shelve.py", line 111, in getitem__ value = self.cache[key]

KeyError: 'azpoly'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jovyan/.local/envs/insar_analysis/share/isce2/stripmapStack//invertMisreg.py", line 207, in <module>
    main()
  File "/home/jovyan/.local/envs/insar_analysis/share/isce2/stripmapStack//invertMisreg.py", line 148, in main
    A, B, Laz, Lrg = design_matrix(pairDirs)
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/envs/insar_analysis/share/isce2/stripmapStack//invertMisreg.py", line 126, in design_matrix
    azOff, rgOff = extract_offset(pairDirs[ni])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/envs/insar_analysis/share/isce2/stripmapStack//invertMisreg.py", line 69, in extract_offset
    azpoly = db['azpoly']
             ~~^^^^^^^^^^
  File "/home/jovyan/.local/envs/insar_analysis/lib/python3.11/shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jovyan/.local/envs/insar_analysis/lib/python3.11/dbm/dumb.py", line 148, in __getitem__
    pos, siz = self._index[key]     # may raise KeyError
               ~~~~~~~~~~~^^^^^
KeyError: b'azpoly'
`

[EJFielding]

Did you try this suggestion? https://github.com/isce-framework/isce2/discussions/488#discussioncomment-3482353

[saeedmhanna1]

The reply you referred to is related to the unpackFrame_ENV. py step.

My error arises after running run_05_invertMisreg and is related to azpoly.

[EJFielding]

Yes, but the azpoly needs to be set in the first unpackFrame step for all the Envisat scenes. If it is not in the metadata database, it might only cause the error much later in the processing.

[saeedmhanna1]

Thanks for the clarification. I had already included that correction in the unpackFrame step before running. Also might be worth adding but when I ran the same step with 4 images it previously worked. now i am trying a bigger stack (27 images from 2002 to 2010) and this error showed up.

[EJFielding]

It seems that the azpoly metadata is missing for at least one of your scenes, from the error message. Either it was missed in the initial unpacking or it was lost in the processing. Did all of your runs before "run_05" complete successfully?

[saeedmhanna1]

All runs files before "run_05" were successful. However, as you mentioned, I expect one of the images is corrupted, since i ran ISCE with fewer images and it worked perfectly. Is there a way to check this azpoly metadata before? especially because "run_05" is taking a lot of time (8 hours in my case for 27 images").

Thanks a lot Eric, Saeed

On Wed, 11 Oct 2023 at 22:26, Eric Fielding @.***> wrote:

It seems that the azpoly metadata is missing for at least one of your scenes, from the error message. Either it was missed in the initial unpacking or it was lost in the processing. Did all of your runs before "run_05" complete successfully?

— Reply to this email directly, view it on GitHub https://github.com/isce-framework/isce2/issues/776#issuecomment-1758482111, or unsubscribe https://github.com/notifications/unsubscribe-auth/BCUJDSQCM37FYRHMILBLDILX636FXANCNFSM6AAAAAA53Z5V2A . You are receiving this because you authored the thread.Message ID: @.***>

[EJFielding]

The error message was for the pair 20031111_20050208 so one of those two images is probably the bad one. Unfortunately, the ISCE2 topsStack workflow stores all the metadata in "shelve" files that are binary files and are not easily checked. You would have to write some Python code to load the database from those files to check them.