Open scottstanie opened 1 year ago
This issue would have been solved in #164 and #168. Can you confirm that you're using the most up-to-date version of COMPASS?
hmmm yes it looks like it's still happening on main:
(mapping-311) [staniewi@aurora COMPASS]$ git pull upstream main
git logFrom github.com:opera-adt/COMPASS
* branch main -> FETCH_HEAD
Already up to date.
(mapping-311) [staniewi@aurora COMPASS]$ git log | head -3
commit 998e203378ffa423c9bbddf3dcd82be5df1922b2
Author: Seongsu Jeong <sjeong.kr@gmail.com>
Date: Fri Aug 25 13:02:21 2023 -0700
(mapping-311) [staniewi@aurora COMPASS]$ s1_cslc.py /home/staniewi/repos/COMPASS/talib-error/geo_runconfig_20230518_t042_088919_iw3.yaml
journal: Starting s1_geocode_slc burst
journal: Starting geocoding of t042_088919_iw3 for 20230518
journal:
-- DEM EPSG: 4326
-- Output EPSG: 4326
journal:
-- Processing block: 1
-- - line start: 0
-- - line end : 111
-- - dopplers near mid far: 0 0 0
journal:
-- Actual DEM bounds used:
-- Top Left: -122.872 37.6109
-- Bottom Right: -121.521 36.9903
-- Spacing: 0.000277778 -0.000277778
-- Dimensions: 4866 2235
Topo progress (block 1/1): 100%
journal: Total convergence: 47880 out of 52392
Warning 1: libjpeg: Corrupt JPEG data: 405 extraneous bytes before marker 0x6e (this warning can be turned as an error by setting GDAL_ERROR_ON_LIBJPEG_WARNING to TRUE)
Traceback (most recent call last):
File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/bin/s1_cslc.py", line 8, in <module>
sys.exit(main())
^^^^^^
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_cslc.py", line 53, in main
run(parser.run_config_path, parser.args.grid_type)
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_cslc.py", line 45, in run
s1_geocode_slc.run(cfg)
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/s1_geocode_slc.py", line 92, in run
cumulative_correction_luts(burst, dem_path=cfg.dem,
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 63, in cumulative_correction_luts
compute_geocoding_correction_luts(burst,
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 212, in compute_geocoding_correction_luts
[open_raster(raster_path) for raster_path in rdr2geo_raster_paths]
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/lut.py", line 212, in <listcomp>
[open_raster(raster_path) for raster_path in rdr2geo_raster_paths]
^^^^^^^^^^^^^^^^^^^^^^^^
File "/u/aurora-r0/staniewi/repos/COMPASS/src/compass/utils/helpers.py", line 351, in open_raster
ds = gdal.Open(filename, gdal.GA_ReadOnly)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/lib/python3.11/site-packages/osgeo/gdal.py", line 5297, in Open
return _gdal.Open(*args)
^^^^^^^^^^^^^^^^^
RuntimeError: libjpeg: Unsupported marker type 0x6e
Maybe there were different gdal versions being tested? I was also getting a RuntimeError
as you said here https://github.com/opera-adt/COMPASS/pull/168/files#r1264289515 . The attribute error caught nothing.
Did we purposely leave out a gdal.UseExceptions()
call in the module? I think that would be the reason for the attribute error- calling something on a None
instead of GDAL raising the actual RuntimeError
which might happen now by default in gdal 3.7
just clarifying: There's a chance that the docker SAS will run correctly as is. But I think if anything running the sas has called gdal.UseExceptions()
, then it will error like mine did.
>>> from osgeo import gdal
# the `Open()` returned None, so it's an attribute error
>>> gdal.Open('asdf').GetRasterBand(1)
ERROR 4: asdf: No such file or directory
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'GetRasterBand'
# a real exception is raised here
>>> gdal.UseExceptions()
>>> gdal.Open('asdf').GetRasterBand(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/u/aurora-r0/staniewi/miniconda3/envs/mapping-311/lib/python3.11/site-packages/osgeo/gdal.py", line 5297, in Open
return _gdal.Open(*args)
^^^^^^^^^^^^^^^^^
RuntimeError: asdf: No such file or directory
Would you mind sharing the runconfig above, I'll try to reproduce the error on my end sometime next week.
Regarding UserExceptions()
:
I don't see anything wrong in using the default way of catching unexpected behavior, although it is not so much pythonic way of error handling. But, I agree your point that the behavior of GDAL can be confusing based on other modules we put in SAS. I think we might need to come up with some agreement about how to handle the GDAL errors.
@taliboliver found a very sneaky bug: Reproduce with:
Apparently the
x.rdr
raster inside thecorrections
folder is getting misread by GDAL. I've seen a similar problem with other binary drivers here https://github.com/OSGeo/gdal/issues/4713 , but I think GDAL uses a heuristic to guess the driver, and if the contents of the file match the heuristic, it can guess wrong.