search

LorenzoStucchi / SEN-ET_automatization

Automatization of the SEN-et SNAP plugin for multi-temporal analysis
GNU General Public License v3.0
7 stars 1 forks source link

[SEN-ET_automatization]"NameError: name 'valid_list_s2_images' is not defined" during code run #6

Closed coomomo closed 1 year ago

coomomo commented 1 year ago

Hello, I encountered "NameError: name 'valid_list_s2_images' is not defined" when running the code "0_path_read.py". I don't know how to fix this error, I hope I can get some help, thanks!

LorenzoStucchi commented 1 year ago

Dear @coomomo, the error seems connected to the fact that there are no Sentinel-2 images in the indicated directory. Did you follow the steps indicated in the README.md and download the satellite images?

coomomo commented 1 year ago

Thank you very much for your answer. I have solved this problem now, the reason for this problem was that I had unrecognized characters in the folder name. But now I seem to have another problem😣, in "2_read_times_s3.py", "for day in date_s3: path = 'E:\\TEMP\\MAINFOLDER\' + "s3\\" + day + "\\S3" + day + "_mask.dim" for line in open(path, 'r').readlines(): if "PRODUCT_SCENE_RASTER_START_TIME" in line: data_start = line.split(">")[1].split("<")[0]" The time obtained by this code is the time of S2 and not S3, I have tried to modify it many times, but still no success. By the way, I would like to ask you if you are writing in IOS system? I hope to get your reply. Thanks!😀

Lorenzo Stucchi @.***> 于2022年11月24日周四 21:34写道：

Dear @coomomo https://github.com/coomomo, the error seems connected to the fact that there are no Sentinel-2 images in the indicated directory. Did you follow the steps indicated in the README.md https://github.com/LorenzoStucchi/SEN-ET_automatization#folder-structure and download the satellite images?

— Reply to this email directly, view it on GitHub https://github.com/LorenzoStucchi/SEN-ET_automatization/issues/6#issuecomment-1326463252, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWNCJ7FEJTSN246NWQNPCQ3WJ5VGLANCNFSM6AAAAAASI52THI . You are receiving this because you were mentioned.Message ID: @.***>

LorenzoStucchi commented 1 year ago

As before I believe this should be an issue with the path, try to check if the path exists.

It is searching the file connected with the S3 images and only that, so maybe your file are in the wrong location

coomomo commented 1 year ago

I solved the S3 time problem yesterday, thanks for your guidance.😀 But now 0_path_read.py is wrong again, in lines 15-18 of "0_path_read.py"

sub_folders = [name for name in os.listdir(general_path) if os.path.isdir(os.path.join(general_path, name))] for folder in sub_folders: temp_path = general_path + "\" + folder

Since " general_path=E:\TEMP\ and sub_folders=MAIN_FOLDER, why is folder still equal to MAIN_FOLDER? I print(temp_path), and the result is "E:\TEMP\\MAIN_FOLDER", not "E:\TEMP\\MAIN_FOLDER\\s2" or "E:\TEMP\\MAIN_FOLDER\s3". I think the temp_path should be equal to "E:\TEMP\\MAIN_FOLDER\\s2" in order to read the s2 images, now the code error is "valid_list_s2_images ""valid_list_s3 images" does not exist. But I checked the code in lines 1-14, and I think it is correct. I hope I can get your help again, I've been struggling with this problem for a long time and can't solve it.😣 I've put a screenshot of my code in the attachment, thank you very much again!

Lorenzo Stucchi @.***> 于2022年11月25日周五 00:04写道：

As before I believe this should be an issue with the path, try to check if the path exists.

It is searching the file connected with the S3 images and only that, so maybe your file are in the wrong location

— Reply to this email directly, view it on GitHub https://github.com/LorenzoStucchi/SEN-ET_automatization/issues/6#issuecomment-1326633195, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWNCJ7FHSA3MGU7ETYPRMNLWJ6GZ3ANCNFSM6AAAAAASI52THI . You are receiving this because you were mentioned.Message ID: @.***>

LorenzoStucchi commented 1 year ago

I cannot see your image, try to attach them on github issue and not in the email

coomomo commented 1 year ago

Thank you very much for your previous answer, I have solved the path problem. But there are still several problems in the program that I have tried many times and still can't solve.I hope I can get your help，Thanks again.

` Processing the image S2 2021_01_18 INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.0.0 from distribution will be used. (f1) INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP. INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP. Executing processing graph INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32648 WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist INFO: org.hsqldb.persist.Logger: dataFileCache open start WARNING: org.esa.s2tbx.dataio.s2.ortho.metadata.S2OrthoMetadata: Warning: missing file E:\TEMP\MAIN_FOLDER\s2\2021_01_18\S2B_MSIL2A_20210118T033059_N0214_R018_T48QXM_20210118T062400.SAFE\GRANULE\L2A_T48QXM_A020207_20210118T033053\QI_DATA\L2A_T48QXM_20210118T033059_DDV_20 m.jp2

WARNING: org.esa.s2tbx.dataio.s2.ortho.metadata.S2OrthoMetadata: Warning: no image files found for band quality_dense_dark_vegetation

SEVERE: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when reading C:\Users\Administrator.snap\auxdata\s2tbx\biophysical\3_0\S2B\LAI_Cw\LAI_Cw_TestCases WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available.`

Sharpening the image S3 2021_01_18 03:03 INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used. INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP. INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. INFO: Deriving solar illumination conditions... INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP. INFO: org.hsqldb.persist.Logger: dataFileCache open start INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmph1zuweee.tif Saved C:\Users\ADMINI~1\AppData\Local\Temp\tmph1zuweee_cos_theta.tif INFO: Preparing high-resolution data... INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmp_lsb7_ij.tif Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. Warning 1: TIFFReadDirectory:Sum of Photometric type-related color channels and ExtraSamples doesn't match SamplesPerPixel. Defining non-color channels as ExtraSamples. INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmpinzg6xr1.tif INFO: org.esa.snap.dataio.bigtiff.BigGeoTiffProductWriter: writing to output file C:\Users\ADMINI~1\AppData\Local\Temp\tmpn1tefd__.tif INFO: Training regressor... Traceback (most recent call last): File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\sen-et-snap-scripts\data_mining_sharpener.py", line 216, in <module> main() File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 764, in __call__ return self.main(*args, **kwargs) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 717, in main rv = self.invoke(ctx) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\click\core.py", line 555, in invoke return callback(*args, **kwargs) File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\sen-et-snap-scripts\data_mining_sharpener.py", line 97, in main disaggregator.trainSharpener() File "C:\Users\Administrator\.snap\auxdata\sen-et-conda-Win64\lib\site-packages\pyDMS\pyDMS.py", line 332, in trainSharpener qualityPix = np.logical_and(qualityPix, ~np.isnan(data_LR)) ValueError: operands could not be broadcast together with shapes (5490,5490) (96,96)

coomomo commented 1 year ago

Although "WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available." appears, but this process has result output, I'm not sure if the result is correct. And "Sharpen LST" has no result output

coomomo commented 1 year ago

I am using windows 10, snap8.0, python3.4

LorenzoStucchi commented 1 year ago

Although "WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available." appears, but this process has result output, I'm not sure if the result is correct. And "Sharpen LST" has no result output

This is a SNAP warning, it is ok and the results is valid

coomomo commented 1 year ago

Although "WARNING: org.esa.s2tbx.biophysical.BiophysicalAuxdata: Error when loading coefficients TEST_CASES for variable LAI_Cw model S2B. They won't be available." appears, but this process has result output, I'm not sure if the result is correct. And "Sharpen LST" has no result output

This is a SNAP warning, it is ok and the results is valid

Thank you. "ValueError: operands could not be broadcast together with shapes" when I try to run "Sharpen LST". How should I correct this mistake?

LorenzoStucchi commented 1 year ago

Thank you. "ValueError: operands could not be broadcast together with shapes" when I try to run "Sharpen LST". How should I correct this mistake?

Sorry, I don't have enough information to provide help. Do you insert all the correct inputs?

LorenzoStucchi commented 1 year ago

@coomomo did you solve the found issue? Do you need more help?

coomomo commented 1 year ago

@LorenzoStucchi
The problem I encountered regarding "ValueError: operands could not be broadcast together with shapes (5490,5490) (96,96)" has been solved. My solution was to manually operate in SNAP when doing "warp to template", "sharpen LST" and the input file must be "graph" The input file must be the "LST", "MASK" and "observation geometry" from the "graph" folder of sentinel_3_pre_processing.xml. I think the reason for this one error may be that the study area is not in the right range. Thank you again for your previous help.