Enhancement: colour scale in geotiffs produced by GDAL

[mcgarth]

Currently Geotiffs produced by GDAL in PyRate are in grey-scale. It would be better if a rainbow colour scale was applied with all images scaled for min-max of the entire dataset (i.e. all input ifgs or all time series slices) so quickviews of the geotiffs can be quickly compared.

[basaks]

Garrick I believe has already done this.

[gp8899]

I have done this but it's using regular tiff files. I discussed with Matt that for the prepifg output, since pyrate is using the data in those files, we can't change the data at all. The best we can do with those files is to modify ColorInterp=Gray.

For the output file I've been simply creating a regular tiff with RGB data with each channel having an integer between 0 and 255. This gives a good visualization of data, but it's not good in the sense that the file contains no information of the actual values each pixel takes (e.g. for the ratemap, it doesn't give you the actual m/s value for each pixel)...

There must be some way to define a color funtion in the tiff file that keeps the actual data (e.g. m/s) in the file, but when opened with an image viewer, applies the appropriate color map.

I will look into this.

[basaks]

Garrick, Here is a starter script.

Usage is documented in the script too.

sydney_trimmed.tif can be converted into something like this with this script:

[basaks]

I have added optional auto colour option. This is how the dem looks like with auto colour:

Script updated with new instructions. It can handle both dem and ifgs.

[gp8899]

@basaks I just sent you an email Matt forward me about having gdaldem functionality in a python module (rasterio). This may be better than calling the executable with subprocess. I'm not sure if this module supports the auto color map though.

Also. I'm wondering if it would be good to have the RGBA channels as well as the float channel for ifgs and pyrate output. I think you can do this. I'll look into it.

[basaks]

rasterio is a nice wrapper. I have thought of using it in PyRate for other functionalities too.

Let's wait for what Matt has to say before we do more work on this? What we already have in gdaldem might be all we need for now.

[neginmoghaddam]

@basaks :Using gdaldem.py for plotting the outcome of run_pyrate.py in rhe-compute and pycharm both gave me this outcome for either auto or custom color scale: The command that I used in "Pyrate" folder: python pyrate/utils/gdaldem.py /nas/gemd/insar/Users/Moghaddam/surat_test/plot/20080808-20080901_HH_4rlks_utm.tif auto out_tif.tif

The message: Traceback (most recent call last): File "pyrate/utils/gdaldem.py", line 26, in from pyrate.shared import Ifg, DEM ImportError: No module named pyrate.shared

[basaks]

Can PyRate be found in your PYTHONPATH? Have you done this?

export PYRATEPATH="/home/sudipta/GA/PyRate"
export PYTHONPATH=$PYRATEPATH/:$PYTHONPATH

[neginmoghaddam]

I can not see my .bashrc in my unix folder. I dont know what happened to it. it was already in my unix folder that i simply source it before. !!! However, I used the following set for the path again

export HTTPS_PROXY=http://u15197:pass@proxy.ga.gov.au:8080 export HTTP_PROXY=http://u15197:pass@proxy.ga.gov.au:8080 export PYTHONPATH=/nas/users/u15197/unix/PyRate:$PYTHONPATH

No change....

[basaks]

I just did python pyrate/utils/gdaldem.py tests/sydney_test/dem/sydney_trimmed.tif auto out_tif.tif It works. You have to fix your paths and environment.

Are prepifg and run_pyrate working?

Ask @gp8899 .

[neginmoghaddam]

@basaks : yes of course.

[neginmoghaddam]

@basaks: In rhe-compute i set the environment and run the same code that you used above, but the same error showed up:

(pyrate_new)-bash-4.2$ cd PyRate/ (pyrate_new)-bash-4.2$ python pyrate/utils/gdaldem.py tests/sydney_test/dem/sydney_trimmed.tif auto out_tif.tif Traceback (most recent call last): File "pyrate/utils/gdaldem.py", line 26, in from pyrate.shared import Ifg, DEM ImportError: No module named pyrate.shared

[basaks]

on rhecompute fire up a python shell..and use that command:

ipython
from pyrate.shared import Ifg, DEM

If that does not work, this script won't work either. Is run_prepifg and run_pyrate working?

[neginmoghaddam]

Yes, run_prepifg and run_pyrate both are working for the new dataset.

[basaks]

on nci

python pyrate/utils/gdaldem.py tests/sydney_test/dem/sydney_trimmed.tif auto out_tif.tif

auto generating color file

color file contents
==================================================
222 black

252 blue

282 yellow

311 orange

341 red

342 white

==================================================
0...10...20...30...40...50...60...70...80...90...100 - done.

Did you do those?

ipython
from pyrate.shared import Ifg, DEM

What's the output?

[neginmoghaddam]

In rhe-compute when I do : ipython it gives me the following trackback error: pyrate_new)-bash-4.2$ ipython Traceback (most recent call last): File "/usr/bin/ipython", line 5, in start_ipython() File "/usr/lib/python2.7/site-packages/IPython/init.py", line 120, in start_ipython return launch_new_instance(argv=argv, _kwargs) File "/usr/lib/python2.7/site-packages/IPython/config/application.py", line 573, in launch_instance app.initialize(argv) File "", line 2, in initialize File "/usr/lib/python2.7/site-packages/IPython/config/application.py", line 75, in catch_config_error return method(app, args, _kwargs) File "/usr/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 332, in initialize self.init_shell() File "/usr/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 348, in init_shell ipython_dir=self.ipython_dir, user_ns=self.user_ns) File "/usr/lib/python2.7/site-packages/IPython/config/configurable.py", line 337, in instance inst = cls(_args, _kwargs) File "/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 494, in init self.init_history() File "/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 1607, in init_history self.history_manager = HistoryManager(shell=self, parent=self) File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 505, in init traits) File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 199, in init** self.init_db() File "", line 2, in init_db File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 92, in catch_corrupt_db self.init_db() File "", line 2, in init_db File "/usr/lib/python2.7/site-packages/IPython/core/history.py", line 91, in catch_corrupt_db os.rename(self.hist_file, newpath) OSError: [Errno 13] Permission denied

If you suspect this is an IPython bug, please report it at: https://github.com/ipython/ipython/issues or send an email to the mailing list at ipython-dev@scipy.org

You can print a more detailed traceback right now with "%tb", or use "%debug" to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via: c.Application.verbose_crash=True

[basaks]

conda install ipython

Then redo.

[neginmoghaddam]

I ran the command... it could not install them and wanted me to do "conda create -n my_pyrate_new --clone=/nas/users/u15197/unix/anaconda2/envs/pyrate_new" I did and then I sourced the new environment " my_pyrate_new" then i ran the install command again but it gave me bunch of errors: (my_pyrate_new)-bash-4.2$ conda install ipython Fetching package metadata: .... Solving package specifications: ................An unexpected error has occurred, please consider sending the following traceback to the conda GitHub issue tracker at:

https://github.com/conda/conda/issues

Include the output of the command 'conda info' in your report.

Traceback (most recent call last): File "/nas/users/u15197/unix/.conda/envs/my_pyrate_new/bin/conda", line 5, in sys.exit(main()) File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/cli/main.py", line 173, in main args_func(args, p) File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/cli/main.py", line 180, in args_func args.func(args, p) File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/cli/main_install.py", line 45, in execute install.install(args, parser, 'install') File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/cli/install.py", line 333, in install update_deps=args.update_deps) File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/plan.py", line 438, in install_actions actions = ensure_linked_actions(smh, prefix) File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/plan.py", line 273, in ensure_linked_actions elif install.try_hard_link(config.pkgs_dirs[0], prefix, dist): File "/nas/users/u15197/unix/anaconda2/lib/python2.7/site-packages/conda/install.py", line 437, in try_hard_link assert not isfile(dst), dst AssertionError: /nas/users/u15197/unix/.conda/envs/my_pyrate_new/.tmp-openssl-1.0.2h-0

[basaks]

Don't worry about ipython. In your conda environment, you should be able to use:

python
from pyrate.shared import Ifg, DEM

If that does not work, I can't explain why you can use run_pyrate/prepifg, but not utils/gdaldem.

[neginmoghaddam]

Sure. This is the outcome of calling the command directly in python (pyrate_new)-bash-4.2$ python Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:08:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org

from pyrate.shared import Ifg, DEM Traceback (most recent call last): File "", line 1, in ImportError: No module named pyrate.shared

[basaks]

Use nci for now :)

Use these after you login:

module rm intel-fc intel-cc
module load intel-mkl
module load python/2.7.6
module load python/2.7.6-matplotlib
module rm openmpi/1.6.3
module load openmpi/1.8
module load gdal/1.11.1-python
module load grib_api/1.12.3
module load pypar/30May-2.7.6-1.8
export PYRATEPATH=/g/data/dg9/sudipta/PyRate
PYTHONPATH=$PYRATEPATH:$PYTHONPATH
PYTHONPATH=/g/data/dg9/sudipta/PyRate:/g/data/dg9/sudipta/site-packages:$PYTHONPATH
export PYTHONPATH

Then

cd PyRate
python pyrate/utils/gdaldem.py tests/sydney_test/dem/sydney_trimmed.tif auto out_tif.tif

Works fine.

[neginmoghaddam]

I used the " inspect_output.py". It was working pretty good for my plotting purpose.

[basaks]

That's good, but you should address why you can't use from pyrate.shared import Ifg, DEM

Chances are you will will get stuck again.

[neginmoghaddam]

@basaks I logged into the nci... and used the ones you mentioned... seems that I did not get there...

[nfm547@raijin1 ~]$ module rm intel-fc intel-cc [nfm547@raijin1 ~]$ module load intel-mkl [nfm547@raijin1 ~]$ module load python/2.7.6 [nfm547@raijin1 ~]$ module load python/2.7.6-matplotlib [nfm547@raijin1 ~]$ module rm openmpi/1.6.3 [nfm547@raijin1 ~]$ module load openmpi/1.8 [nfm547@raijin1 ~]$ module load gdal/1.11.1-python [nfm547@raijin1 ~]$ module load grib_api/1.12.3 [nfm547@raijin1 ~]$ module load pypar/30May-2.7.6-1.8 [nfm547@raijin1 ~]$ export PYRATEPATH=/g/data/dg9/sudipta/PyRate [nfm547@raijin1 ~]$ PYTHONPATH=$PYRATEPATH:$PYTHONPATH [nfm547@raijin1 ~]$ PYTHONPATH=/g/data/dg9/sudipta/PyRate:/g/data/dg9/sudipta/site-packages:$PYTHONPATH [nfm547@raijin1 ~]$ export PYTHONPATH [nfm547@raijin1 ~]$ cd PyRate bash: cd: PyRate: No such file or directory

[basaks]

Change these lines to your PyRate location in NCI.

export PYRATEPATH=/g/data/dg9/sudipta/PyRate

If you have accesss to /g/data/dg9/sudipta/PyRate:/g/data/dg9/sudipta/site-packages, then cd in your PyRate in NCI.

[neginmoghaddam]

I have access to /g/data/dg9/sudipta/PyRate but I dont have my own "Pyrate location" in NCI. Maybe I need to set up there...

[basaks]

Yes, just do a git clone Pyrate in NCI.

[neginmoghaddam]

ok I cloned it through this command : git clone /g/data/dg9/sudipta/PyRate into my own directory in NCI. Now i will go up to follow the instruction.

[neginmoghaddam]

@basaks : Successful running the gdaldem.py using test data... :)

Now for my case, should I put the images I want to plot on NCI. Could be directed to the /nas/gemd/.. for instance...

[basaks]

yea, put the full path for the output file, or you can use scp to transfer to your pc.

[neginmoghaddam]

So far it is working for the files similar to this type: date1-date2_HH_4rlks_utm_5rlks_4cr.tif but for the "linearrate" it gives me the following error that is related to the values probably in this file that is not compatible with the colorscale..

[nfm547@gamma PyRate]$ python pyrate/utils/gdaldem.py images/linrate.tif auto images/Linerate_out.tif

auto generating color file /apps/python/2.7.6/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /apps/python/2.7.6/lib/python2.7/site-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) Traceback (most recent call last): File "pyrate/utils/gdaldem.py", line 62, in color_file = gen_color_file(input_file) File "pyrate/utils/gdaldem.py", line 49, in gen_color_file ' ' + c + '\n') ValueError: cannot convert float NaN to integer

[basaks]

It's likely that you phase_data is all nan's, i.e. zeros and then nan_converted.

[neginmoghaddam]

The result for plotting looks good. I might try the custome color scale.

[neginmoghaddam]

Hmm. Might be since I had a number of parameters zero in my configuration file including tscal.

[basaks]

I think we can close this issue now. There are two different implementations now for this.