Small issues with DTMCreation.py

olivierhagolle commented 5 years ago

Hi Peter, I have found two little issues with DTMCreation.py

first, each time it is started, it unzips all the SWBD masks, which takes some time (1 or 2mn on hal)
then it crashes when merging the SRTM files: SRTM files are unzipped in the tmp dir, but the tif files are searched in the SRTM_dir (see below)

My command line was:

python DTMCreation.py -k tiles.kml -g 35TLL -s /work/CESBIO/projects/Maja/SRTM/ -w /work/CESBIO/projects/Maja/SWBD/ -o .

I think the error is line 436 of lib_mnt, but i don't fully understand the function tempfile.mkstem In case, I have put all SRTM and SWBD files in /work/CESBIO/projects/Maja/SRTM /work/CESBIO/projects/Maja/SWBD Merci Olivier

liste_fic_mnt ['srtm_41_03.tif', 'srtm_42_03.tif']
FIC: srtm_41_03.tif
<class 'str'> <class 'str'>
/work/CESBIO/projects/Maja/SRTM//srtm_41_03.tif
Archive:  /work/CESBIO/projects/Maja/SRTM//srtm_41_03.zip
  inflating: /tmp/35TLL/35TLL_02n_hz5r/readme.txt  
  inflating: /tmp/35TLL/35TLL_02n_hz5r/srtm_41_03.hdr  
  inflating: /tmp/35TLL/35TLL_02n_hz5r/srtm_41_03.tfw  
  inflating: /tmp/35TLL/35TLL_02n_hz5r/srtm_41_03.tif  
FIC: srtm_42_03.tif
<class 'str'> <class 'str'>
/work/CESBIO/projects/Maja/SRTM//srtm_42_03.tif
Archive:  /work/CESBIO/projects/Maja/SRTM//srtm_42_03.zip
  inflating: /tmp/35TLL/35TLL_02n_hz5r/readme.txt  
  inflating: /tmp/35TLL/35TLL_02n_hz5r/srtm_42_03.hdr  
  inflating: /tmp/35TLL/35TLL_02n_hz5r/srtm_42_03.tfw  
  inflating: /tmp/35TLL/35TLL_02n_hz5r/srtm_42_03.tif  
gdal_merge.py -o /tmp/35TLL/35TLL_02n_hz5r/mnt_35TLLe3xxik1t.tif /work/CESBIO/projects/Maja/SRTM/srtm_41_03.tif  /work/CESBIO/projects/Maja/SRTM/srtm_42_03.tif 
ERROR 4: /work/CESBIO/projects/Maja/SRTM/srtm_41_03.tif: No such file or directory
ERROR 4: /work/CESBIO/projects/Maja/SRTM/srtm_42_03.tif: No such file or directory

petket-5 commented 5 years ago

I will have a look into this.

petket-5 commented 5 years ago

Pushed changes to 181205_start_maja_reprog

olivierhagolle commented 5 years ago

Hi, I tested, and it has improved, Thanks ! But it crashes later on (sorry)

I used

module add gdal_openjpeg/2.3.0 wgich also loads python 3.5
python DTMCreation.py -k tiles.kml -g 35TLL -s /work/CESBIO/projects/Maja/SRTM/ -w /work/CESBIO/projects/Maja/SWBD/ -o .

Now it crashes when providing the 10m images:

###decoupage 10m
gdalwarp  -overwrite -r cubic -srcnodata -32768 -dstnodata 0 -of ENVI -tr 10 10 -te 300000 4990200 409800 5100000 -t_srs EPSG:32635 /tmp/35TLLt8hjwmt_/tmpu182jzf3/mnt_35TLL3yd15887nodata0.tif /tmp/35TLLt8hjwmt_/tmpu182jzf3_10m.mnt

Creating output file that is 10980P x 10980L.
ERROR 1: Image file is too small
gdalwarp  -overwrite -r cubic -ot Float32 -srcnodata -32768 -dstnodata 0 -of ENVI -tr 10 10 -te 300000 4990200 409800 5100000 -t_srs EPSG:32635 /tmp/35TLLt8hjwmt_/tmpu182jzf3/mnt_35TLL3yd15887nodata0.tif /tmp/35TLLt8hjwmt_/tmpu182jzf3_10mfloat.mnt

Creating output file that is 10980P x 10980L.
ERROR 1: Image file is too small
gdalwarp  -overwrite -r cubic -ot Float32 -srcnodata -32768 -dstnodata 0 -of ENVI -tr 10 10 -te 300000 4990200 409800 5100000 -t_srs EPSG:32635 /tmp/35TLLt8hjwmt_/tmpu182jzf3_90mfloat.dz_dl /tmp/35TLLt8hjwmt_/tmpu182jzf3_10mfloat.dz_dl

Creating output file that is 10980P x 10980L.
ERROR 1: Image file is too small
gdalwarp  -overwrite -r cubic -ot Float32 -srcnodata -32768 -dstnodata 0 -of ENVI -tr 10 10 -te 300000 4990200 409800 5100000 -t_srs EPSG:32635 /tmp/35TLLt8hjwmt_/tmpu182jzf3_90mfloat.dz_dc /tmp/35TLLt8hjwmt_/tmpu182jzf3_10mfloat.dz_dc

Creating output file that is 10980P x 10980L.
ERROR 1: Image file is too small
/tmp/35TLLt8hjwmt_/tmpu182jzf3_10m
241120800
Traceback (most recent call last):
  File "DTMCreation.py", line 302, in <module>
    creator.run(args.out, args.tempout)
  File "DTMCreation.py", line 269, in run
    water_zipped = self.WaterZipped)
  File "/work/CESBIO/users/hagolle/Start_Maja_Gitlab/prepare_mnt/../prepare_mnt/tuilage_mnt_eau_S2.py", line 227, in run
    mnt_full.calcul_pente_aspect_fic()
  File "/work/CESBIO/users/hagolle/Start_Maja_Gitlab/prepare_mnt/../prepare_mnt/lib_mnt.py", line 362, in calcul_pente_aspect_fic
    dz_dl = (np.fromfile(fic_dz_dl, type_donnee)).reshape(nblig, nbcol)
ValueError: cannot reshape array of size 0 into shape (10980,10980)
[hagolle@visu03 prepare_mnt]$ module add gdal_openjpeg/2.3.0^C

olivierhagolle commented 5 years ago

It seems to be a gdal bug http://earthdef.caltech.edu/boards/4/topics/2462

olivierhagolle commented 5 years ago

Effectivement, ça fonctionne mieux avec python 2.7 et gdal 2.2.1 Mais le masque d'eau n'est pas bon:

gdal_translate -of GTIFF  /tmp/pbs.1116525.admin01/35TLLgrpfwn/tmpgUSMgW_240m.eau ./S2__TEST_AUX_REFDE2_T35TLL_0001/S2__TEST_AUX_REFDE2_T35TLL_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T35TLL_0001_MSK.TIF
ERROR 4: /tmp/pbs.1116525.admin01/35TLLgrpfwn/tmpgUSMgW_240m.eau: No such file or directory
Finished DTM creation for 35TLL
[hagolle@node039 prepare_mnt]$ gdalinfo ./S2__TEST_AUX_REFDE2_T35TLL_0001/S2__TEST_AUX_REFDE2_T35TLL_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T35TLL_0001_MSK.TIF
ERROR 4: ./S2__TEST_AUX_REFDE2_T35TLL_0001/S2__TEST_AUX_REFDE2_T35TLL_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T35TLL_0001_MSK.TIF: No such file or directory
gdalinfo failed - unable to open './S2__TEST_AUX_REFDE2_T35TLL_0001/S2__TEST_AUX_REFDE2_T35TLL_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T35TLL_0001_MSK.TIF'.
[hagolle@node039 prepare_mnt]$ ls ./S2__TEST_AUX_REFDE2_T35TLL_0001/S2__TEST_AUX_REFDE2_T35TLL_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T35TLL_0001_MSK.TIF
ls: cannot access ./S2__TEST_AUX_REFDE2_T35TLL_0001/S2__TEST_AUX_REFDE2_T35TLL_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T35TLL_0001_MSK.TIF: No such file or directory
[hagolle@node039 prepare_mnt]$ ls  /tmp/pbs.1116525.admin01/35TLLgrpfwn/tmpgUSMgW_240m.eau
ls: cannot access /tmp/pbs.1116525.admin01/35TLLgrpfwn/tmpgUSMgW_240m.eau: No such file or directory

olivierhagolle commented 5 years ago

OK, looks perfect now thanks Peter ! Olivier

olivierhagolle / Start_maja

Small issues with DTMCreation.py #30