Trouble saving outputs to file

[GinaONeil]

Hello,

I am a graduate researcher at the University of Virginia and I am working on designing/optimizing a terrain analysis model used to identify wetlands. I am just beginning to move this model out of ArcGIS and I think the pyDEM library is a great tool to calculate the TWI and its derivatives for my study sites. I'm starting off just working with the test_NN064_013_elev.tif file before processing the elevation tiles for my study sites. Using the test tif, I try to create a DEMProcessor object, calculate the slope and directions, plot the slope array, and save the slope array to a file. My code is below:

filename_to_elevation_geotiff = 'test_NN064_013_elev.tif'

dem_proc = DEMProcessor(filename_to_elevation_geotiff)

mag, aspect = dem_proc.calc_slopes_directions()

print dem_proc.mag

plt.imshow(dem_proc.direction)

if not os.path.exists('mag'): os.mkdir('mag') dem_proc.save_slope('.') print 'done'

---

When I run this code, I get the following output:

starting slope/direction calculation [[ 1.57748599e-05 1.57748599e-05 1.56843030e-05 ..., 1.56843030e-05 1.57748599e-05 1.57748599e-05] [ 1.57748599e-05 1.57748599e-05 1.56843030e-05 ..., 1.56843030e-05 1.57748599e-05 1.57748599e-05] [ 1.58631500e-05 1.58631500e-05 1.57733022e-05 ..., 1.57733022e-05 1.58631500e-05 1.58631500e-05] ..., [ 1.56804838e-05 1.56804838e-05 1.55954034e-05 ..., 1.55954034e-05 1.56804838e-05 1.56804838e-05] [ 1.55911111e-05 1.55911111e-05 1.55055366e-05 ..., 1.55055366e-05 1.55911111e-05 1.55911111e-05] [ 1.55911111e-05 1.55911111e-05 1.55055366e-05 ..., 1.55055366e-05 1.55911111e-05 1.55911111e-05]] <<<<<<<< gdalwarp -multi -wm 2000 -co BIGTIFF=YES -of GTiff -co compress=lzw -co TILED=YES -wo OPTIMIZE_SIZE=YES -r near -t_srs EPSG:4326 .\mag\N45W-73_N46W-72_mag_tmp.tif .\mag\N45W-73_N46W-72_mag.tif >>>>>>>> done

A plot is created (attached) and a "mag" folder is created in my directory. However, this folder remains empty and I cannot find the newly created mag geotiff anywhere on my computer.

We think that this line :

https://github.com/creare-com/pydem/blob/master/pydem/dem_processing.py#L682

which removes the tmp file may be deleting the output of the save_slope method. We have tried to edit the dem_processing.py file to comment out line 682, but the edits (or other test edits) do not seem to affect my code, and I get the same result.

What do you think is causing this?

Thanks, Gina

[empeeu]

In line just above, try subprocess.call(cmd, shell=True)

What OS are you using?

[GinaONeil]

Thanks, actually it looks I had to reload the dem_processing module by closing and reopening spyder, when I did this the changes I made to the dem_processing.py file showed up. I am using Windows.

So now, it looks like the only way I can save the mag file is by commenting out https://github.com/creare-com/pydem/blob/master/pydem/dem_processing.py#L682 which creates a mag tmp_file. This change works with or without the "subprocess.call(cmd, shell=True)" addition that was suggested.

[mpu-creare]

Since you've managed to create the 'tmp' file, can you run the command from above from a windows command prompt?

gdalwarp -multi -wm 2000 -co BIGTIFF=YES -of GTiff -co compress=lzw -co TILED=YES -wo OPTIMIZE_SIZE=YES -r near -t_srs EPSG:4326 .\mag\N45W-73_N46W-72_mag_tmp.tif .\mag\N45W-73_N46W-72_mag.tif

I have a suspicion of what the problem might be: I assume you're using anaconda for you python installation? If that's the case, you may have to set your GDAL_DATA environmental variable to something like this GDAL_DATA=<Anaconda_Install_Directory> \Library\share\gdal

There's a lot of assumptions baked into that suggestion.

[GinaONeil]

Yes, I am using anaconda. Okay, I set that environmental variable and have checked that is is defined. When I try to run gdalwarp -multi -wm 2000 -co BIGTIFF=YES -of GTiff -co compress=lzw -co TILED=YES -wo OPTIMIZE_SIZE=YES -r near -t_srs EPSG:4326 .\mag\N45W-73_N46W-72_mag_tmp.tif .\mag\N45W-73_N46W-72_mag.tif

I get: 'gdalwarp' is not recognized as an internal or external command, operable program or batch file.

[mpu-creare]

Ah! Well, that would be a problem. Did you install Anaconda as a system admin? Or for the specific user only? I'm guessing the latter. That's good to know.

Basically windows can't find the gdal executables. You have to add the Anaconda bin directory to your windows path, something like PATH=<Anaconda Install Dir>/Library/bin/

[GinaONeil]

Got it! I was able to run that command from the windows command prompt. The output was:

0...10...20...30...40...50...60...70...80...90...100 - done.

and a mag tiff file was created in the correct directory, identical to the tmp file that was created before. So is this a necessary step to create a non-tmp file of pydem outputs?

[mpu-creare]

Glad you got it working.

This step doesn't always do something, but it:

1. Inpaints any nan regions
2. Compresses the file
3. Projects the file to the CRS as specified by DEMProcessor.save_projection

Feel free to modify the code to server your own needs. I'm closing this one because I'm assuming you're all set.

[GinaONeil]

Thanks, I appreciate all the help!

[diafu]

Hi mpu-creare. Thanks for this great tool. I have a comment related to this issue. I agree with Gina, here. I used your TWIDinf command line to save a .tif file for twi. However, it did not work until I did what was recommended by Gina: commented L682 in dem_processing.py. It might work fixing this, many can save time trying to solve the problem. Again, thanks for the tool!

[mpu-creare]

Thanks for the comment @diafu

I'm working on a new version that's Python 3 compatible... but I'll consider a hotfix, intermediate version if I don't get it out later this month.

[diafu]

Great if it can be done. Thanks again!