EOReader 0.17.0 and SAR: BUG whith DSPK bands

[TK12331]

When I followed the document of https://eoreader.readthedocs.io/en/latest/notebooks/SAR.html#load-bands, some problems occurred.

First, there are some bool values of band_names returned from s1_prod.has_band(band) are True, while they don't exist in get_existing_bands()

Second, I check the output file and there are only two tif files which causes the code s1_prod.load(ok_bands, resolution=20.) to fail.

Are these normal phenomenon?

[remi-braun]

1. First screenshot: this is normal
   • has_band returns True to every bands that could potentially be loaded
   • get_existing_bands gives only the "satellite bands", not the CLOUDS, indices, or DEM bands
2. Could you display the SNAP bug that caused the DSPK to fail ? Is it something with a predictor ?

[TK12331]

For the question 2, is DSPK supposed to exist in the file?

[remi-braun]

Yes, it creates a VV_DSPK file on disk. This fails for you and I want to know why 😓

I have corrected a bug in the latest version and you may have encountered it...

[TK12331]

I download this repository the day before yesterday, should I try to update the code? I'm restarting running the code now, it may takes some time.

Now the code reprocesses the data for every run, how can I let it use the cache in tmp file?

[remi-braun]

I would like you to give me the full bug stack trace so I can analyze if 😄

In order not to reprocess everything, you should set an output directory and leave remove_tmp in Reader().open to False (its default value).

prod = Reader().open(path, output_path="your/path")

or

prod = Reader().open(path)
prod.output = "your/path"

[TK12331]

After s1_prod = reader.open(s1_path, output_path=f"{s1_path[:-4]}/EO/") I got VH and VV tif without throwing an error, can you tell me which code file should I set breakpoint at? (Or rather the code file that create the VH_DSPK.tif)

[remi-braun]

Yes VV and VH will be OK. I suspect the error to be thrown by SNAP during the despeckling conversion. (sar_products._despeckle_sar) Can you try:

s1_prod.load(VV_DSPK)

That should fail with something like https://forum.step.esa.int/t/exception-found-when-reading-compressed-tif/654/7

[TK12331]

I found that the self._despeckle_sar was skipped, thus didn't run into _despeckle_sar() .

I set dspk=True in _write_sar, and DSPK can be wrote. However, the original tif 'VV' and 'VH' are missing.

Can I just set dspk=True in _write_sar ? Is it correct?

Besides, s1_prod.load(VV_DSPK) would thow 'file not found' error.

[remi-braun]

_despeckle_sar is skipped if you don't try load DSPK bands.

I don't understand what you are currently testing. Can you be clearer ? For example show me your testing snippets ?

[TK12331]

Here is my testing code now.

I found that I did't try to load DSPK in s1_prod.get_band_paths() before because https://eoreader.readthedocs.io/en/latest/notebooks/SAR.html#load-bands don't mention it, thus skipping. I add DSPK now and following error raised when processing DSPK.

RuntimeError (note: full exception trace is shown but execution is paused at: _run_module_as_main) Something went wrong with SNAP! File "D:\SAR_DATA\process\eoreader-main\eoreader\products\sar\sar_product.py", line 421, in get_band_paths band_paths[band] = files.get_file_in_dir( File "C:\Users\10992.conda\envs\DL1\Lib\site-packages\sertit\files.py", line 1119, in get_file_in_dir raise FileNotFoundError(

During handling of the above exception, another exception occurred:

File "D:\SAR_DATA\process\eoreader-main\eoreader\products\sar\sar_product.py", line 852, in _despeckle_sar misc.run_cli(cmd_list) File "C:\Users\10992.conda\envs\DL1\Lib\site-packages\sertit\misc.py", line 355, in run_cli raise RuntimeError(f"Exe {cmd[0]} has failed.")

The above exception was the direct cause of the following exception:

File "D:\SAR_DATA\process\eoreader-main\eoreader\products\sar\sar_product.py", line 854, in _despeckle_sar raise RuntimeError("Something went wrong with SNAP!") from ex File "D:\SAR_DATA\process\eoreader-main\eoreader\products\sar\sar_product.py", line 441, in get_band_paths band_paths[band] = self._despeckle_sar(speckle_band, **kwargs) File "D:\SAR_DATA\process\eoreader-main\K.py", line 17, in s1_prod.get_band_paths([VV, VH, VV_DSPK, VH_DSPK]) File "C:\Users\10992.conda\envs\DL1\Lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\10992.conda\envs\DL1\Lib\runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "C:\Users\10992.conda\envs\DL1\Lib\runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "C:\Users\10992.conda\envs\DL1\Lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\10992.conda\envs\DL1\Lib\runpy.py", line 197, in _run_module_as_main (Current frame) return _run_code(code, main_globals, None,

[TK12331]

If you need more messages, plz make me know, I'm always online.

[remi-braun]

Hello,

The line get_band_path is useless for you, you can directly load the bands 😄 But you will likely have the same issue. Otherwise your code is OK.

Could you give me the SNAP error ? Or maybe it isn't displayed...

What is your sertit version ? For that type:

import sertit
sertit.__version__

I feel all that revolves around an already fixed bug in sar_products.py (the fix will be in 0.18.0 version), that comes from the fact that VV and VH GeoTiff's are saved on disk with an optimized format (LZW + predictor = 3, added in sertit==1.19.2), and sadly SNAP doesn't handle it...

If it's the case, you can fork EOReader and use the 0.18.0 branch until its final release or downgrade sertit to 1.19.1 😓

[TK12331]

The SNAP error isn't displayed... The version of my sertit is '1.19.5'. I'm trying the 0.18.0 branch, hope it will work.

By the way, here are some questions I want to ask:

1. Should operation of Multi_Look should be conducted when processing S1 product? I note that other programs seem to conduct Multi_Look before Terrain_Correction.

2. What are footprint and extent as follows mean? I have thought about it for a long time...

3. Can I directly load tiff file by OpenCV in python for post SAR image processing? Because it seems totally black in vision now... If not, how can I convert Tiff to PNG reasonably?

4. Why the value in the VV band I load (xarray type) is -60 to 25, the value seems strange due to the usual value is 0 to a positive number that depends on the precision ( etc, int16, int32)

Thank you in advance for your help !

[remi-braun]

1/ Multi-look is for SLC data, not GRDH. If you look at complex graphs in eoreader.data you will see MultiLook before Terrain-Correction 😄 2/ The difference between extent and footprint is easier to understand here. Basically, the footprint is the data polygon, when extent includes nodata. 3/ I never used OpenCV and I wouldn't recommend it as it isn't designed to deal with rasters. Moreover, jpg will compress your file and you will lose a lot of information. 4/ VV band is in float32 and in dB. SAR data can have negative values, especially when processed.

You may need to take some courses on SAR data if you want to truly understand what you are doing as this isn't really the place to ask about it, as I am no expert 😉

[remi-braun]

PS: if you don't manage to use 0.18.0 branch, downgrade sertit to 1.19.1 by doing pip install sertit[full]==1.19.1

[TK12331]

Got it, I'm a new researcher in SAR image domain hhh. Thanks for your advice.

I just found that pip install doesn't have 0.18.0 version

And I meet some strange problem when running the code from 0.18.0 branch like

I 'm using python==3.9 and eoreader==0.17.0 now, does the main branch is version 0.17.0? How can I update it to 0.18.0?

[remi-braun]

As I said, 0.18.0 version has not been release yet 😉 Try the downgrade stated hereover 🙏

If you want the last version, you have to do pip install git+https://github.com/sertit/eoreader@0.18.0 And yeah there are some spyndex things that are not released either, so add pip uninstall spyndex && pip install -U --no-deps git+https://github.com/awesome-spectral-indices/spyndex.git

This is why I would advice you to downgrade sertit 😅

[TK12331]

I have downgraded the sertit, and a new problem is raised.

And error something went wrong with snap appeared again...

[remi-braun]

Aaaah that is what I wanted to see in the first time ! SNAP traceback 🥂

Did you recomputed VH and VV files ? Or used the computed with the other version ? If not, you need to delete/recompute them as they are saved on disk with a compression SNAP is not able to read....

(this is the whole bug point 😅 )

[TK12331]

I indeed not recomputed it and used the cache before, I try it right now (about 4mins).

[TK12331]

The DSPK file can be produced now, thank you!

[remi-braun]

Ouuuh yeah, so relieved! 🚀 Sorry you arrived at a bad time 😅

This will all be resolved in 0.18.0...

[remi-braun]

For other people that have the same issue.

TLDR: Attempt to compute DSPK bands ➡️ SNAP fails with a predictor issue ➡️ downgrade sertit to 1.19.1: ✅ pip install sertit[full]==1.19.1

[TK12331]

Ouuuh yeah, so relieved! 🚀 Sorry you arrived at a bad time 😅

This will all be resolved in 0.18.0...

I'm looking forward to it hhhh.

[remi-braun]

Solved in release 0.18.0 🚀