search

cordmaur / WaterDetect

Water Detect Algorithm
Apache License 2.0
194 stars 37 forks source link

WaterDetect: Unable to clip with a shapefile and 8 gb RAM crashed without it #27

Closed Dave0178 closed 2 months ago

Dave0178 commented 9 months ago

Hello :-)

I'm trying to run this : 

!waterdetect -GC

water_mask = wd.DWWaterDetect.run_water_detect(input_folder="C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE",
                                         output_folder="C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\output\\water_detect_test",
                                         shape_file="C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\spatial_extent.shp",
                                         single_mode=True,
                                         product='S2_S2COR'
                                        )

or this : 

input_folder="C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE"

output_folder="C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\output\\water_detect_test"
os.makedirs(output_folder, exist_ok=True)

shape_file="C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\spatial_extent.shp"

water_mask = wd.DWWaterDetect.run_water_detect(input_folder=input_folder,
                                         output_folder=output_folder,
                                         shape_file=shape_file,
                                         single_mode=True,
                                         product='S2_S2COR'
                                        )

but it seems it can't clip : 

Copying [C:\Users\david\Documents\py\myenv\Lib\site-packages\WaterDetect.ini](file:///C:/Users/david/Documents/py/myenv/Lib/site-packages/WaterDetect.ini) into current dir.
WaterDetect.ini copied into [c:\Users\david\Documents\py\myenv\nakambe_2022.](file:///C:/Users/david/Documents/py/myenv/nakambe_2022.)
Loading configuration file WaterDetect.ini
File WaterDetect.ini verified.
Folder [C:\Users\david\Documents\py\myenv\nakambe_2022\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE](file:///C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE) verified.
File [C:\Users\david\Documents\py\myenv\nakambe_2022\spatial_extent.shp](file:///C:/Users/david/Documents/py/myenv/nakambe_2022/spatial_extent.shp) verified.
Folder [C:\Users\david\Documents\py\myenv\nakambe_2022\output\water_detect_test](file:///C:/Users/david/Documents/py/myenv/nakambe_2022/output/water_detect_test) verified.
Starting WaterDetection version: 1.5.15
Opening image in folder [C:\Users\david\Documents\py\myenv\nakambe_2022\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE](file:///C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE)
Retrieving bands for product S2_S2COR
35 were found:
Loading band: Blue
Loading band: Green
Loading band: Red
Loading band: Mir
Loading band: Mir2
Loading band: RedEdg1
Loading band: RedEdg2
Loading band: RedEdg3
Loading band: Nir
Loading band: Nir2
****** ERROR ********
'NoneType' object has no attribute 'FlushCache'

Without the clipping option, it ran for an hour and crashed due to lack of ram: 

Copying C:\Users\david\Documents\py\myenv\Lib\site-packages\WaterDetect.ini into current dir.Loading configuration file WaterDetect.ini

WaterDetect.ini copied into c:\Users\david\Documents\py\myenv\nakambe_2022.
File WaterDetect.ini verified.
Folder C:\Users\david\Documents\py\myenv\nakambe_2022\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE verified.
Folder C:\Users\david\Documents\py\myenv\nakambe_2022\output\water_detect_test verified.
Starting WaterDetection version: 1.5.15
Opening image in folder C:\Users\david\Documents\py\myenv\nakambe_2022\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE
Retrieving bands for product S2_S2COR
35 were found:
Loading band: Blue
Loading band: Green
Loading band: Red
Loading band: Mir
Loading band: Mir2
Loading band: RedEdg1
Loading band: RedEdg2
Loading band: RedEdg3
Loading band: Nir
Loading band: Nir2
Invalid pixels (0.04494427689357368) < maximum (0.95).
Saving final MASK
Saving image: C:/Users/david/Documents/py/myenv/nakambe_2022/output/water_detect_test/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200_invalid_mask.tif
Calculating water mask in single mode. Just the first band_combination is processed
Calculating clusters for the following combination of bands:
['mndwi', 'ndwi', 'Mir2']
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Selection of best number of clusters using Calinski-Harabasz Index:
**** ERROR DURING CLUSTERING ****
unable to allocate array data

Also,

bands=loader.get_bands_files() gave me FileNotFoundError: [WinError 3] Le chemin d’accès spécifié est introuvable: 'C:\\Users\\david\\Documents\\py\\myenv\\nakambe_2022\\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE\\AUX_DATA\\GRANULE

which is curious because I didn't specified ...SAFE\\AUX_DATA\\GRANULE.

It seems it can't browse correctly. If I delete the \\AUX_DATA folder, then the same error occured but with another path that wasn't specified neither : ...SAFE\\DATASTRIP\\GRANULE, it just tried to reach a folder \\GRANULE in the first folder in the folder \\...SAFE.

I tried also those: 

import waterdetect as wd
from pathlib import Path
input_folder=Path("C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE")
loader=wd.InputOutput.DWLoader(input_folder=input_folder, product='S2_S2COR' )

Folder C:\Users\david\Documents\py\myenv\nakambe_2022\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE verified.

!dir "C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE"

 Le volume dans le lecteur C s'appelle Windows-SSD
 Le num�ro de s�rie du volume est 9417-7BFA

 R�pertoire de C:\Users\david\Documents\py\myenv\nakambe_2022\S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE

30-12-23  18:29    <DIR>          .
04-01-24  15:11    <DIR>          ..
06-06-22  18:40    <DIR>          AUX_DATA
06-06-22  18:40    <DIR>          DATASTRIP
06-06-22  18:40    <DIR>          GRANULE
06-06-22  18:40    <DIR>          HTML
06-06-22  18:37            18.628 INSPIRE.xml
06-06-22  18:37            96.247 manifest.safe
06-06-22  18:37            53.370 MTD_MSIL2A.xml
06-06-22  18:40    <DIR>          rep_info
06-06-22  18:40            42.484 S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200-ql.jpg
               4 fichier(s)          210.729 octets
               7 R�p(s)   9.808.646.144 octets libres
import os
print(os.listdir(input_folder))

from pathlib import Path
p = Path(input_folder)
print([x for x in p.iterdir() if x.is_dir()])

['AUX_DATA', 'DATASTRIP', 'GRANULE', 'HTML', 'INSPIRE.xml', 'manifest.safe', 'MTD_MSIL2A.xml', 'rep_info', 'S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200-ql.jpg'] [WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/AUX_DATA'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/DATASTRIP'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/GRANULE'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/HTML'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/rep_info')]

loader.images [WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/AUX_DATA'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/DATASTRIP'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/GRANULE'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/HTML'), WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/rep_info')]

img_folder=loader.current_image_folder
img_folder

WindowsPath('C:/Users/david/Documents/py/myenv/nakambe_2022/S2A_MSIL2A_20220101T102431_N0301_R065_T30PZU_20220101T133200.SAFE/AUX_DATA') [str(f) for f in img_folder.iterdir()] []

Any idea?

Thanx,

David

cordmaur commented 7 months ago

@Dave0178,

As you have only 8Gb of RAM, I suggest you to clip the area using a third party software (e.g., QGIS) and then load the arrays manually and pass them to waterdetect as explained in the session Running without GDAL of the tutorial:

https://www.geocorner.net/post/water-detection-in-high-resolution-satellite-images-using-the-waterdetect-python-package

Here is the gist for it: Link to Gist