Error when processing MSI scene

BrandonSmithJ commented 3 years ago

Hi, I'm running into the following error when applying acolite to Sentinel-2:

ERROR 4: : No such file or directory
ERROR 5: GDALWarpOptions.Validate()
  hSrcDS is not set.
/scratch/job/35907215/pipeline/AtmCorr/acolite/sentinel2/l1_convert.py:359: RuntimeWarning: Mean of empty slice
  ave_vza = np.nanmean(ave_vza, axis=2)
/scratch/job/35907215/pipeline/AtmCorr/acolite/sentinel2/l1_convert.py:360: RuntimeWarning: Mean of empty slice
  ave_vaa = np.nanmean(ave_vaa, axis=2)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/scratch/job/35907215/pipeline/AtmCorr/atm_utils.py", line 143, in wrapper
    return function(*args, **kwargs)
  File "/scratch/job/35907215/pipeline/AtmCorr/run_acolite.py", line 61, in run_acolite
    ac.acolite.acolite_run(tmpfile.name)
  File "/scratch/job/35907215/pipeline/AtmCorr/acolite/acolite/acolite_run.py", line 94, in acolite_run
    ret = ac.acolite.acolite_l1r(bundle, setu_l1r)
  File "/scratch/job/35907215/pipeline/AtmCorr/acolite/acolite/acolite_l1r.py", line 76, in acolite_l1r
    verbosity=setu['verbosity'])
  File "/scratch/job/35907215/pipeline/AtmCorr/acolite/sentinel2/l1_convert.py", line 370, in l1_convert
    sza = ac.shared.warp_from_source(target_file, dct_prj, sza) # alt (dct, dct_prj, sza)
  File "/scratch/job/35907215/pipeline/AtmCorr/acolite/shared/warp_from_source.py", line 99, in warp_from_source
    data = ds.ReadAsArray()
AttributeError: 'NoneType' object has no attribute 'ReadAsArray'

I've dug into it a bit and found that by changing the format of ds at line 94 to be MEM instead of VRT, it seems to perform the warp. Unfortunately, that leads to another issue where the original data is shaped (1830, 1830), but the warped data becomes (1830, 1831) - which leads to errors elsewhere in the code base when writing the calculated products to disk.

For what it's worth, the exact scene I'm using to test with is S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916, but the problem occurs with others as well. Any suggestions?

acolite commented 3 years ago

Hi Brandon

The scene processes fine here with a minimal settings file with inputfile, output, and s2_target_res=60 (which is presumably what you use to get 1830 pixels). Do you have any other custom settings?

You get a "hSrcDS is not set" error which maybe means that the file from which the target projection is taken is missing or incomplete. When I run the processing the target_file used to set up the geometry is S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916.SAFE/GRANULE/L1C_T21HXB_A010546_20190314T133942/IMG_DATA/T21HXB_20190314T133219_B01.jp2

The size of this file is 3.2M with MD5 checksum d485fd8d48046ed865a6111a18c29564

Quinten

BrandonSmithJ commented 3 years ago

Hey Quinten, thanks for the reply.

I've ran it again with a minimal config file (just inputfile, output, and s2_target_res=60) with the same results. I've also double checked the file you referenced, and it has the same MD5. The remaining files in the folder all look reasonable enough as far as sizes, but I can get you the remaining checksums if you'd like.

If it helps, here's the log prior to the error:

Running generic ACOLITE processing - Generic Version Public Beta - April 2021
Run ID - 20210924_121023
/pardees/f002/bsmith/Tiles/MSI/20190314/S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916/S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916.SAFE
/pardees/f002/bsmith/Tiles/MSI/20190314/S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916/S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916.SAFE/out/acolite_run_20210924_121023_l1r_settings.txt
Starting conversion of 1 scenes
Starting conversion of /pardees/f002/bsmith/Tiles/MSI/20190314/S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916/S2B_MSIL1C_20190314T133219_N0207_R081_T21HXB_20190314T195916.SAFE
Importing metadata from L1C_T21HXB_A010546_20190314T133942
Reading per pixel geometry
Computing per detector geometry
Detector 4
/pardees/f002/bsmith/AtmCorr/acolite/sentinel2/l1_convert.py:359: RuntimeWarning: Mean of empty slice
  ave_vza = np.nanmean(ave_vza, axis=2)
/pardees/f002/bsmith/AtmCorr/acolite/sentinel2/l1_convert.py:360: RuntimeWarning: Mean of empty slice
  ave_vaa = np.nanmean(ave_vaa, axis=2)
Detector 1
Detector 3
Detector 2

What version of GDAL are you using to test with? I've tried with a single release for each of 1.x, 2.x, and 3.x with no luck.

acolite commented 3 years ago

Hi Brandon

I am using gdal 3.3.1 from conda-forge. (On Mac, could also check the Linux and Windows versions, but I generally have one of the latest from conda-forge.)

Did you update the code recently? I did change some things for warping the S2 data on July 24: https://github.com/acolite/acolite/commit/146b7b689bc3dc27227996670d86e4f49d0664a8

You get the data = ds.ReadAsArray() error at L99 of warp_from_source.py, which may indicate you have the February version of that script. (That line is currently at L102.)

Let me know how it goes!

Quinten

BrandonSmithJ commented 2 years ago

Hi Quinten, sorry about the delay getting back to you. Indeed updating the code appears to have fixed the issue, after I modify the format used in warp_from_source to be MEM rather than VRT. I'm guessing that has to do with the gdal build I'm using though, which unfortunately isn't maintained by me.

In any case, I appreciate your help in resolving this, and your work on building this project in general!

acolite / acolite

Error when processing MSI scene #11