Problems processing S3 Images

[FranGutierrezC]

Hi everyone! I have some troubles running the processing for S3 images and I would appreciate some help. Before the execution of the .sh, I updated the sen-et-scripts (https://github.com/DHI-GRAS/sen-et-snap-scripts) and cheked that the graphs of the graphs folder were the corrected ones.
When I execute de sh main.sh file from git bash I got all the outputs from S2, however for the S3 data, I only receive the observation geometry, lst and mask. In the Output folder “out” generated, the only output file obtained is the longwave_ir. Also, the folder to save the ET.tiff also is missing. As I said before, im troubling trying to solve the issue. I copy the part of the error for one of the images that im using.

Im using python 3.6, and working with "S2B_MSIL2A_20221204T143729_N0400_R096_T19KDP_20221204T183146" and "S3B_SL_2_LST____20221202T141651_20221202T141951_20221203T124713_0179_073_224_3420_PS2_O_NT_004"

I will be attentive to any questions or additional information that is necessary to resolve this problem, I greatly appreciate your time.

(…) Warp the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Usage: warp_to_template.py [OPTIONS] Try "warp_to_template.py --help" for help.

Error: Invalid value for "--template": File "D:\PROYECTO_B\S2\2022_12_04\S2_2022_12_04_resampled.dim" does not exist. Sharpening the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Usage: data_mining_sharpener.py [OPTIONS] Try "data_mining_sharpener.py --help" for help.

Error: Invalid value for "--high_res_geom": File "D:\PROYECTO_B\S3\2022_12_02\S3_2022_12_02_hr_vza.dim" does not exist. Computing the meteo data for the day 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: org.hsqldb.persist.Logger: dataFileCache open start INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\AETOS~1.I7-\AppData\Local\Temp\tmpyl8kes3z.tif Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". Warning 1: Recode from UTF-8 to CP_ACP failed with the error: "Invalid argument". air_temperature Saved MEM Saved MEM Saved MEM vp Saved MEM ap Saved MEM ws Saved MEM cssr Saved MEM adsi Saved MEM Computing the longwave radiation for the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: org.hsqldb.persist.Logger: dataFileCache open start Computing the shortwave radiation for the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Usage: net_shortwave_radiation.py [OPTIONS] Try "net_shortwave_radiation.py --help" for help.

Error: Invalid value for "--sza_product": File "D:\PROYECTO_B\S3\2022_12_02\S3_2022_12_02_hr_vza.dim" does not exist. Computing the energy fluxes for the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Usage: energy_fluxes.py [OPTIONS] Try "energy_fluxes.py --help" for help.

Error: Invalid value for "--lst": File "D:\PROYECTO_B\S3\2022_12_02\S3_2022_12_02_lst_sharpened.dim" does not exist. Computing the evapotranspiration for the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Usage: daily_evapotranspiration.py [OPTIONS] Try "daily_evapotranspiration.py --help" for help.

Error: Invalid value for "--ief_file": File "D:\PROYECTO_B\out\2022_12_02\2022_12_02_instantaneous_fluxes.dim" does not exist. Transformation of the evapotranspiration for the image S3 2022_12_02 14:17 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Executing processing graph done.

Error: [NodeId: Read] Specified 'file' [D:\PROYECTO_B\et\dim\2022_12_02_daily_evapotranspiration.dim] does not exist. Finish the computation of the evapotranspiration for the image S3 2022_12_02 14:17

automatization_sen-et_issue.txt

[LorenzoStucchi]

Hi, the first error in the log is the code of 1_script_gpt_s3.sh, here:

         Processing the image S3 2022_12_02
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: D:\PROYECTO_B\S3\2022_12_02\S3B_SL_2_LST____20221202T141651_20221202T141951_20221203T124713_0179_073_224_3420_PS2_O_NT_004.SEN3 (Acceso denegado)
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: Could not find ''.
 done.

Error: [NodeId: Subset] Operator 'SubsetOp': Value for 'Source Bands' is invalid: 'sat_zenith_tn'

I'm not understanding this part of your previous text:

Before the execution of the .sh, I updated the sen-et-scripts (https://github.com/DHI-GRAS/sen-et-snap-scripts) and cheked that the graphs of the graphs folder were the corrected ones.

It is not needed to change nothing, the correct graph are the one present in the folder graph of this repo

[FranGutierrezC]

Thanks for your reply! I notice this error off "access denied" to all the folders of sentinel-3 imagen. However I've cheked that the folders had the permision, tried to use the chmod +x to give complete access and also I've been execute gitbash as administrator, but none of this have work. Have you any ideas to solve the issue? What I cant understand is that if the folder of S3 had access denied, why I can obtein the products of lst, mask and observed geometry? About the scripts, I updated the ones of the sen-et-scripts folders because ive found some difference with the ones in this repo (https://github.com/DHI-GRAS/sen-et-snap-scripts)

[LorenzoStucchi]

Yes, there are differences between the scripts, but the onesin this repo are the ones for this code. Try to do it with the scripts in this repo I made the change in the other repo because they were wrong https://github.com/DHI-GRAS/sen-et-snap-scripts/commits?author=LorenzoStucchi

[FranGutierrezC]

Ok thanks, I'll make a test with those scripts. About the access denied to the folder, do you have any ideas to solve it? or had you this similar problem before? i've trying to solve it but nothing works

[LorenzoStucchi]

It could be an issue that is not critical, let's try with the scripts if they solve the issue.

[FranGutierrezC]

Hi! I've try the solution by replacing the scripts of the link that you gave me, however im still getting the same error of access denied to the folders .SEN3 of the images that I'm using. I also attach the output "4 script S3" to check if there's a problem in the generation of the outputs.

Aetos.i7-10700@Fran-geolab MINGW64 /d/SEN-ET_automatization (master) $ sh main.sh Path files created! input/s2_paths.txt input/s3_paths.txt

real 0m0.138s user 0m0.000s sys 0m0.000s

    Scripts created!
    output\1_script_gpt_s3.sh
    output\2_script_gpt_s2.sh

real 0m0.171s user 0m0.000s sys 0m0.000s

     Processing the image S3 2022_12_02

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Executing processing graph WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: D:\Proyecto_B\S3\2022_12_02\S3B_SL_2_LST____20221202T141651_20221202T141951_20221203T124713_0179_073_224_3420_PS2_O_NT_004.SEN3 (Access denied) WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: Could not find ''. INFO: org.hsqldb.persist.Logger: dataFileCache open start .13%.27%..38%.48%..62%..76%.90% done.

4_script_S3.txt

[LorenzoStucchi]

It looks that the access is denied but then the comand is done. The product is created? The 4_script_S3.txt is the same as one that I have as example. Which version of snap are you using?

[LorenzoStucchi]

I tested the code only on snap 8, maybe could be something related to snap 9, I did not tested on this version

[FranGutierrezC]

Trying to find the issue, i've notice that the script .xml of sentinel_2_preprocessing doesn't save the output of the resample of the bands, is that correct or is not necesary in the workflow? Because i check that in the "warp_to_template" starts mi problem. And I got all de inputs necesary except for the resample.dim file . When I checked and open the .xml file from snap I realized that theres no box in the graph builder that writes the "Write_resample" as with the other products of S2. I think that having that file _resample.dim I could get the others missing outputs. I would appreciate some guidance to solve the issue. Thanks so much for your time so far

sentinel_2_pre_processing_xml.txt

[LorenzoStucchi]

Ok, found the issue, the variable have been rename but not all the changes have been uploaded, I have added them now.

So now, the warp_to_template.py takes 3 input and provides 1 output:

1. --source: that is obs_geometry.dim that comes from the sentinel_3_pre_processing.xml at the node Write_obs_geometry
2. --template: that is mask.dim that comes from the sentinel_2_pre_processing.xml at the node Write_mask
3. --resample_algoritm: that is directly set to near

Now, it should work

[LorenzoStucchi]

fix in https://github.com/LorenzoStucchi/SEN-ET_automatization/commit/165e2c1590c3aec9661d3b0576ff2891ed1a38d9

[LorenzoStucchi]

Thank you for the help in understanding the problem!

As you said, there is a lack of info on the documentation. You could try to contact the authors directly via mail.

However, as far as I remember, I believe that the lake has been removed because it has been recognized as water, and the model does not compute direct ET from water.

[LorenzoStucchi]

I don't know if have you ever experience a similar problem with multitemporal analysis of images, any guidane will help me. Thanks again, and my apologize for the insistence.

My usage was for computing a year of data. So I wrote the code to choose the S2 image closest to the S3 of reference. You could try to provide fewer S2 images and see how these affect the results.

Also, the usage of S2A and S2B could provide different results, maybe staying just on one of the two. Same for S3A and S3B

[FranGutierrezC]

Thanks for your answer Lorenzo, now it makes sense. I'll work with S2A because those images are the ones that gave me the output that i expected, and I will select only one sensor for S3. I appreciate your help