Issues in Data Preparation and Modelling

abhisheksinghIN commented 3 months ago

After implementing sentinel2_prepare.py;

test1

But during implementing meraner_processor.py, the following issues are;

root@3dfcdc1d2ec0:/workspace/decloud/decloud/production# python meraner_processor.py --in_s2 workspace/decloud/data/S2_PREPARE/T32TPS/ --in_dem workspace/decloud/data/DEM_PREPARE/T32TPS.tif --output workspace/decloud/data/output/ --savedmodel workspace/decloud/data/model/merunet_occitanie_pretrained.zip 2024-06-24 08:49:39 (INFO) [pyOTB] Successfully loaded 126 OTB applications 2024-06-24 08:49:39 INFO Init. S2_TILED product 2024-06-24 08:49:39 INFO Product is not of type s2 (Exception: Not a S2_TILED product (expected 1 files ending with _FRE_10m.tif)) 2024-06-24 08:49:39 INFO Init. S2_THEIA product 2024-06-24 08:49:39 INFO Input type is a directory 2024-06-24 08:49:39 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_THEIA product : _FRE_B2.tif is mi 2024-06-24 08:49:39 INFO Init. S2_ESA product 2024-06-24 08:49:39 INFO Input type is a directory 2024-06-24 08:49:39 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_ESA product : _B02_10m.jp2 is mis Traceback (most recent call last): File "/workspace/decloud/decloud/production/meraner_processor.py", line 223, in sys.exit(main()) File "/workspace/decloud/decloud/production/meraner_processor.py", line 217, in main meraner_processor(params.il_s1, params.in_s2, params.savedmodel, params.in_dem, File "/workspace/decloud/decloud/production/meraner_processor.py", line 73, in meraner_processor s2t = s2t_product.get_raster_10m() AttributeError: 'NoneType' object has no attribute 'get_raster_10m'

abhisheksinghIN commented 2 months ago

After S1_prepare.py; data directory: workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/

After S2_prepare.py, data directory: workspace/decloud/data/S1_PREPARE/T32TPS/

After the python meraner_processor.py, This is following error while using the merunet_occitanie_pretrained model;

root@3dfcdc1d2ec0:/workspace/decloud/decloud/production# python meraner_processor.py --in_s2 workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/ --il_s1 workspace/decloud/data/S1_PREPARE/T32TPS/ --in_dem workspace/decloud/data/DEM_PREPARE/T32TPS.tif --output workspace/decloud/data/output/ --savedmodel workspace/decloud/data/model/merunet_occitanie_pretrained/ 2024-07-15 13:35:23 (INFO) [pyOTB] Successfully loaded 126 OTB applications 2024-07-15 13:35:23 INFO Init. S2_TILED product 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: Not a S2_TILED product (expected 1 files ending with _FRE_10m.tif)) 2024-07-15 13:35:23 INFO Init. S2_THEIA product 2024-07-15 13:35:23 INFO Input type is a directory 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_THEIA product : _FRE_B2.tif is missing) 2024-07-15 13:35:23 INFO Init. S2_ESA product 2024-07-15 13:35:23 INFO Input type is a directory 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_ESA product : _B02_10m.jp2 is missing) 2024-07-15 13:35:23 INFO Init. S1_TILED product 2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: Not a S1_TILED product. File workspace/decloud/data/S1_PREPARE/T32TPS/ not found) 2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: S1_THEIA.init() got an unexpected keyword argument 'verbose') Traceback (most recent call last): File "/workspace/decloud/decloud/production/meraner_processor.py", line 183, in sys.exit(main()) File "/workspace/decloud/decloud/production/meraner_processor.py", line 177, in main meraner_processor(params.il_s1, params.in_s2, params.savedmodel, params.in_dem, File "/workspace/decloud/decloud/production/meraner_processor.py", line 68, in meraner_processor input_s1_products.sort(key=_closest_date, reverse=True) File "/workspace/decloud/decloud/production/meraner_processor.py", line 66, in _closest_date return abs(s2t_product.get_timestamp() - x.get_timestamp()) AttributeError: 'NoneType' object has no attribute 'get_timestamp'

Can you please verify the pre-processed files and why I am facing this error? Did you update the pre-processing scripts through which we have different filenames saved in S1_PREPARE and S2_PREPARE and the pre-trained model is expecting different filenames?

remicres commented 1 month ago

Hi @abhisheksinghIN ,

I think that you must provide the absolute paths. Let me know if this fixes the issue

BernradMaillard commented 1 month ago

After S1_prepare.py; data directory: workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/

After S2_prepare.py, data directory: workspace/decloud/data/S1_PREPARE/T32TPS/

After the python meraner_processor.py, This is following error while using the merunet_occitanie_pretrained model;

root@3dfcdc1d2ec0:/workspace/decloud/decloud/production# python meraner_processor.py --in_s2 workspace/decloud/data/S2_PREPARE/T32TPS/SENTINEL2A_20230821-101805-444_L2A_T32TPS_D_V3-1/ --il_s1 workspace/decloud/data/S1_PREPARE/T32TPS/ --in_dem workspace/decloud/data/DEM_PREPARE/T32TPS.tif --output workspace/decloud/data/output/ --savedmodel workspace/decloud/data/model/merunet_occitanie_pretrained/ 2024-07-15 13:35:23 (INFO) [pyOTB] Successfully loaded 126 OTB applications 2024-07-15 13:35:23 INFO Init. S2_TILED product 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: Not a S2_TILED product (expected 1 files ending with _FRE_10m.tif)) 2024-07-15 13:35:23 INFO Init. S2_THEIA product 2024-07-15 13:35:23 INFO Input type is a directory 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_THEIA product : _FRE_B2.tif is missing) 2024-07-15 13:35:23 INFO Init. S2_ESA product 2024-07-15 13:35:23 INFO Input type is a directory 2024-07-15 13:35:23 INFO Product is not of type s2 (Exception: workspace/decloud/data/S2_PREPARE/T32TPS/ not a S2_ESA product : _B02_10m.jp2 is missing) 2024-07-15 13:35:23 INFO Init. S1_TILED product 2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: Not a S1_TILED product. File workspace/decloud/data/S1_PREPARE/T32TPS/ not found) 2024-07-15 13:35:23 INFO Product is not of type s1 (Exception: S1_THEIA.init() got an unexpected keyword argument 'verbose') Traceback (most recent call last): File "/workspace/decloud/decloud/production/meraner_processor.py", line 183, in sys.exit(main()) File "/workspace/decloud/decloud/production/meraner_processor.py", line 177, in main meraner_processor(params.il_s1, params.in_s2, params.savedmodel, params.in_dem, File "/workspace/decloud/decloud/production/meraner_processor.py", line 68, in meraner_processor input_s1_products.sort(key=_closest_date, reverse=True) File "/workspace/decloud/decloud/production/meraner_processor.py", line 66, in _closest_date return abs(s2t_product.get_timestamp() - x.get_timestamp()) AttributeError: 'NoneType' object has no attribute 'get_timestamp'

Can you please verify the pre-processed files and why I am facing this error? Did you update the pre-processing scripts through which we have different filenames saved in S1_PREPARE and S2_PREPARE and the pre-trained model is expecting different filenames?

provide the relative paths, this worked for me.

 meraner_processor \
  --il_s1 ./images/S1/S1_PREPARE/s1a_33PUK_vvvh_ASC_161_20231029txxxxxx_from-10to3dB.tif \
  --in_s2 ./images/S2/S2_PREPARE/T33PUK/SENTINEL2B_20231030-093821-936_L2A_T33PUK_C_V3-1/ \
  --in_dem ./images/T33PUK.tif \
  --output reconstructed.tif \
  --savedmodel ./merunet_burkina_pretrained \
  --pad 256

CNES / decloud

Issues in Data Preparation and Modelling #14

provide the relative paths, this worked for me.