Closed JacobJeppesen closed 2 months ago
Hello @JacobJeppesen Thanks for using Floodpy. Firstly I would like to focus on the issue you have with the provided example notebook. It seems strange that you don't have a result at 2 hours given your configuration. Could you please verify that your downloaded S1 images are not corrupted. You can open them in SNAP desktop and see if you could visualize the intensity bands. Please let me know if this is the case.
Hi @kleok
Thank you for making Floodpy! :slightly_smiling_face:
I just opened a couple of the Sentinel-1 files in SNAP. They seem to be fine, and I can visualize the bands. All Sentinel-1 zip files are 1.8GB.
Just to be sure I did everything correctly, I just tried again from scratch:
FLOODPY_gpu_env.yml
, and activated the environment.Floodpyapp_notebook.ipynb
notebook.
# The path of your project. Make sure you have enough free space disk on the specific location.
projectfolder = '/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023'
src_dir = '/home/jhj/Downloads/floodpy_issue_test/FLOODPY/floodpy/'
snap_orbit_dir = '/home/jhj/.snap/auxdata/Orbits/Sentinel-1'
GPTBIN_PATH = '/home/jhj/esa-snap/bin/gpt'
5. Created the project folder `/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023`
6. Ran all cells in the notebook.
The issue was the same, where it got stuck on the Coregistration step. How long should the processing time approximately be, if it worked?
I'd normally change credentials in the notebook btw., but this time I wanted to make as few changes as possible, so I just used the defaults.
I continued debugging, and made this change to enable logging from ESA SNAP:
def perform_pair_preprocessing_2GRD_2GRD(gptcommand, primary1, primary2, secondary1, secondary2, outfile, Subset_AOI, xml_file, overwrite):
"""
This function performs the preprocessing of the given pair from the
input S1 SLC stack.
"""
if not overwrite:
if os.path.exists(outfile+'.h5'):
return 0
argvs=[gptcommand, '-e',
xml_file,
'-Pfilein1='+primary1,
'-Pfilein2='+primary2,
'-Pfilein3='+secondary1,
'-Pfilein4='+secondary2,
'-Ppolygon='+_shapely_to_snap_polygon(Subset_AOI),
'-Pfileout='+outfile,]
print_esa_snap_logs = True
if print_esa_snap_logs:
# Append argument for logging from ESA SNAP
argvs.append('-J-Dsnap.log.level=FINE')
# Print the command for manual execution
print("Running command:")
print(" ".join(argvs))
# Run the command while printing ESA SNAP logs to stdout in real-time
with subprocess.Popen(argvs, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) as process:
for stdout_line in iter(process.stdout.readline, ""):
print(stdout_line, end="")
process.stdout.close()
return_code = process.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, argvs)
else:
subprocess.check_call(argvs, stdout=subprocess.DEVNULL, stderr=subprocess.STDOUT)
return 0
It gave me the following command for the graph processing:
/home/jhj/esa-snap/bin/gpt -e /home/jhj/Downloads/floodpy_issue_test/FLOODPY/floodpy/Preprocessing_S1_data/Graphs/preprocessing_pair_primary_2GRD_secondary_2GRD.xml -Pfilein1=/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023/Sentinel_1_data/S1A_IW_GRDH_1SDV_20230906T043947_20230906T044012_050202_060AD8_440E.SAFE.zip -Pfilein2=/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023/Sentinel_1_data/S1A_IW_GRDH_1SDV_20230906T044012_20230906T044037_050202_060AD8_49FE.SAFE.zip -Pfilein3=/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023/Sentinel_1_data/S1A_IW_GRDH_1SDV_20230708T043943_20230708T044008_049327_05EE7E_A3D4.SAFE.zip -Pfilein4=/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023/Sentinel_1_data/S1A_IW_GRDH_1SDV_20230708T044008_20230708T044033_049327_05EE7E_C11B.SAFE.zip -Ppolygon=POLYGON ((21.82 39.35, 21.82 39.65, 22.3 39.65, 22.3 39.35, 21.82 39.35, 21.82 39.35)) -Pfileout=/home/jhj/Downloads/floodpy_issue_test/FLOODPY/projects/Thessalia_Floods_2023/Preprocessed_20230906T043947/20230708T043956 -J-Dsnap.log.level=FINE
Running it showed that it's getting stuck on RemoveGRDBorderNoiseOp
:
.
.
.
FINE: org.esa.snap.core.gpf.internal.OperatorContext: Tile cache assigned to OperatorImageTileStack[Write,Sigma0_VV_mst_06Sep2023]
FINE: org.esa.snap.core.gpf.internal.OperatorContext: Tile cache assigned to OperatorImageTileStack[Write,Sigma0_VH_slv1_08Jul2023]
FINE: org.esa.snap.core.gpf.internal.OperatorContext: Tile cache assigned to OperatorImageTileStack[Write,Sigma0_VV_slv2_08Jul2023]
..20%....30%FINE: org.esa.snap.core.dataio.AbstractProductReader: Finish reading the product: input: /home/jhj/.snap/auxdata/dem/Copernicus 30m Global DEM/Copernicus_DSM_COG_10_N39_00_E021_00_DEM.tif, reader: org.esa.snap.dataio.geotiff.GeoTiffProductReader, size: 3600x3600, elapsed time: 0.095 seconds.
FINE: org.esa.snap.core.dataio.AbstractProductReader: Finish reading the product: input: /home/jhj/.snap/auxdata/dem/Copernicus 30m Global DEM/Copernicus_DSM_COG_10_N39_00_E022_00_DEM.tif, reader: org.esa.snap.dataio.geotiff.GeoTiffProductReader, size: 3600x3600, elapsed time: 0.008 seconds.
..FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: topBorder = 433
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: bottomBorder = 16262
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: leftBorder = 268
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: rightBorder = 26469
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: topBorder = 436
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: bottomBorder = 16252
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: leftBorder = 268
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: rightBorder = 26469
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: topBorder = 219
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: bottomBorder = 16671
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: leftBorder = 189
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: rightBorder = 26459
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: topBorder = 272
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: bottomBorder = 16670
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: leftBorder = 189
FINE: org.esa.s1tbx.calibration.gpf.RemoveGRDBorderNoiseOp: rightBorder = 26459
..40%....50%....60%..
I don't think I've encountered this before. The noise removal normally runs very fast. Have you ever had issues similar to this? It's basically just sitting there, running full load on all cores. However, it is very slowly progressing, so I'll try to let it run for longer tomorrow, and see what happens. It seems like it'll take ~4 hours for this step, but I'll let it have it :slightly_smiling_face:
Hey @JacobJeppesen,
Thanks for taking the time to dive into this issue. I think that your steps are correct and I believe that the issue is related to processing using ESA-SNAP. For now let's try to resolve this issue using SNAP9 even if the latest version is SNAP10.
Given that the expected completion time for each pair (given that we have 2GRDs for our AOI) is about 10 min, I believe the huge running time is related to the maximum memory usage of SNAP. I suggest you to open /home/jhj/esa-snap/bin/gpt.vmoptions
and adjust the maximum memory usage from 512 MB to 32 Gb. Please ensure that -Xmx512m
is commented and add at the end of the file -Xmx32G
.
Let me know if this worked. Happy to help :smile:
The -Xmx32G
parameter didn't do the trick, but it got me on the right track. Opened SNAP and went through Tools -> Options -> Performance -> System -> Compute -> Apply. That updated /home/jhj/esa-snap/bin/gpt.vmoptions
to:
# Enter one VM parameter per line
# For example, to adjust the maximum memory usage to 512 MB, uncomment the following line:
# -Xmx512m
# To include another file, uncomment the following line:
# -include-options [path to other .vmoption file]
-Xmx90099m
-Xms2048m
-XX:+AggressiveOpts
-Xverify:none
-Dnetbeans.mainclass=org.esa.snap.main.Main
-Dsun.java2d.noddraw=true
-Dsun.awt.nopixfmt=true
-Dsun.java2d.dpiaware=false
Now it all runs smoothly.
Thanks for the help! :slightly_smiling_face:
Just tested the other region, where I had the LIA error, and it works perfectly after updating with the code from your latest commit :+1:
Thanks again! :smiley:
I'm trying to run the example notebooks, but I can't get any further than the pre-processing. When running
Floodpy_app.create_S1_stack(overwrite=False)
, I get this message under the cell in Jupyter:It seems to be stuck there, and letting it run for ~2 hours doesn't results in any progress. Looking at
htop
, it's runninggpt
from SNAP with lots of cores active. I've only changed paths and credentials at the top, without any further changes to the notebook. I've tried SNAP 9.0 and 10.0, running Ubuntu 22.04 on an AMD 5950x with 128GB RAM.Then I tried a different region, by changing the location to:
I kept the flood and pre-flood start/end dates the sames, and used
sel_flood_date = '2023-09-08T17:07:51.075000000'
. This gave me the following error instead inFloodpy_app.create_S1_stack(overwrite=False)
:I tried a couple of different regions too, but got the exact same error.
Have anyone encountered similar issues? And have any ideas on how to fix them? :slightly_smiling_face: