search

GenericMappingTools / pygmt

A Python interface for the Generic Mapping Tools.
https://www.pygmt.org
BSD 3-Clause "New" or "Revised" License
747 stars 216 forks source link

load_earth_relief() failed to find earth relief files #972

Closed core-man closed 3 years ago

core-man commented 3 years ago

Description of the problem

As I am working on #966, I first ran the examples on pygmt.datasets.load_earth_relief. However, it seems that load_earth_relief() may have a bug to find low-resolution earth relief files, which seems to be related to which.

Full code that generated the error

>>> from pygmt.datasets import load_earth_relief
>>> grid = load_earth_relief()

Full error message

gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]
gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 425, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/datasets/earth_relief.py", line 111, in load_earth_relief
    fname = which(f"@earth_relief_{resolution}{reg}", download="a")
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 281, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/src/which.py", line 55, in which
    raise FileNotFoundError("File '{}' not found.".format(fname))
FileNotFoundError: File '@earth_relief_01d' not found.

I also tested high-resolution earth relief. load_earth_relief() uses grdcut to handle it instead of which.

>>> from pygmt.datasets import load_earth_relief
>>> grid = load_earth_relief("05m", region=[120, 160, 30, 60], registration="gridline")
grdblend [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]
grdblend [NOTICE]: Earth Relief at 5x5 arc minutes from Gaussian Cartesian filtering (9 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
grdblend [NOTICE]:   -> Download 180x180 degree grid tile (earth_relief_05m_g): S90E000

Both high- and low-resolution earth relief files have been downloaded.

# Check GMT Earth data directory
$ ls ~/.gmt/server/earth/earth_relief
earth_relief_01d_p.grd  earth_relief_05m_g
$ ls ~/.gmt/server/earth/earth_relief/earth_relief_05m_g 
S90E000.earth_relief_05m_g.nc

Then I used gmt which command in the terminal to check those files:

# High resolution @earth_relief_05m_g
$ gmt which @earth_relief_05m_g                 
gmtwhich [ERROR]: Tile @S90W180.earth_relief_05m_g.nc not found!
/home/core-man/.gmt/server/earth/earth_relief/earth_relief_05m_g/S90E000.earth_relief_05m_g.nc
$ gmt which @S90E000.earth_relief_05m_g.nc
/home/core-man/.gmt/server/earth/earth_relief/earth_relief_05m_g/S90E000.earth_relief_05m_g.nc

# Low resolution @earth_relief_01d
$ gmt which @earth_relief_01d_p.grd                            
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ gmt which @earth_relief_01d_p    
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ gmt which @earth_relief_01d  
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

System information

Please paste the output of python -c "import pygmt; pygmt.show_versions()":

PyGMT information:
  version: v0.3.1.dev34+g4db28ef
System information:
  python: 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:46)  [GCC 9.3.0]
  executable: /home/core-man/.anaconda3/envs/pygmt/bin/python
  machine: Linux-3.10.0-862.11.6.el7.x86_64-x86_64-with-glibc2.17
Dependency information:
  numpy: 1.20.1
  pandas: 1.2.2
  xarray: 0.16.2
  netCDF4: 1.5.6
  packaging: 20.9
  ghostscript: 9.53.3
  gmt: 6.1.1
GMT library information:
  binary dir: /home/core-man/.anaconda3/envs/pygmt/bin
  cores: 8
  grid layout: rows
  library path: /home/core-man/.anaconda3/envs/pygmt/lib/libgmt.so
  padding: 2
  plugin dir: /home/core-man/.anaconda3/envs/pygmt/lib/gmt/plugins
  share dir: /home/core-man/.anaconda3/envs/pygmt/share/gmt
  version: 6.1.1
welcome[bot] commented 3 years ago

👋 Thanks for opening your first issue here! Please make sure you filled out the template with as much detail as possible. You might also want to take a look at our contributing guidelines and code of conduct.

seisman commented 3 years ago

Could you please post the output of

gmt --version
gmt which @earth_relief_01d_p.grd -Vd
core-man commented 3 years ago

@seisman 

$ gmt --version
6.1.1
$ gmt which @earth_relief_01d_p.grd -Vd
gmt [DEBUG]: Obtained the ppid from parent: 105618
gmt [DEBUG]: Enter: gmtinit_new_GMT_ctrl
gmt [DEBUG]: GMT->session.SHAREDIR = /home/core-man/.anaconda3/envs/pygmt/share/gmt
gmt [DEBUG]: GMT->session.HOMEDIR = /home/core-man
gmt [DEBUG]: GMT->session.USERDIR = /home/core-man/.gmt [created]
gmt [DEBUG]: GMT->session.CACHEDIR = /home/core-man/.gmt/cache [created]
gmt [DEBUG]: GMT: 0. Will try to find subdir=postscriptlight stem = PSL_custom_fonts suffix=.txt
gmt [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
gmt [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /home/core-man/.gmt
gmt [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /home/core-man/.gmt/postscriptlight
gmt [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /home/core-man/.anaconda3/envs/pygmt/share/gmt/postscriptlight
gmt [DEBUG]: GMT: 5. gmt_getsharepath trying SHAREDIR /home/core-man/.anaconda3/envs/pygmt/share/gmt
gmt [DEBUG]: GMT: 6. gmt_getsharepath failed
gmt [DEBUG]: Map distance calculation will be Cartesian
gmt [DEBUG]: Exit:  gmtinit_new_GMT_ctrl
gmt [DEBUG]: Enter: New_PSL_Ctrl
gmt [DEBUG]: Exit:  New_PSL_Ctrl
gmt [DEBUG]: Enter: gmt_manage_workflow
gmt [DEBUG]: Exit : gmt_manage_workflow
gmt [DEBUG]: Enter: PSL_beginsession
gmt [DEBUG]: Exit : PSL_beginsession
gmt [DEBUG]: Enter: PSL_setdefaults
gmt [DEBUG]: Exit : PSL_setdefaults
gmt [DEBUG]: Enter: gmtlib_io_init
gmt [DEBUG]: Exit : gmtlib_io_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit:  gmt_hash_init
gmt [DEBUG]: Enter: gmt_hash_init
gmt [DEBUG]: Exit:  gmt_hash_init
gmt [DEBUG]: Enter: gmt_reload_settings
gmt [DEBUG]: The PROJ_GEODESIC set to Vincenty
gmt [DEBUG]: Look for file /home/core-man/gmt.conf
gmt [DEBUG]: Look for file /home/core-man/.gmt/gmt.conf
gmt [DEBUG]: Look for file /home/core-man/.gmt/server/gmt.conf
gmt [DEBUG]: Look for file /home/core-man/.gmt/cache/gmt.conf
gmt [DEBUG]: Could not find file gmt.conf
gmt [DEBUG]: Exit:  gmt_reload_settings
gmt [DEBUG]: Enter: gmtlib_plot_C_format
gmt [DEBUG]: Exit:  gmtlib_plot_C_format
gmt [DEBUG]: Enter: gmtinit_get_history
gmt [DEBUG]: Initialize FFTW with 8 threads.
gmt [DEBUG]: GMT_Create_Session initialized GMT structure
gmt [DEBUG]: Loading core GMT shared library: libgmt.so
gmt [DEBUG]: Shared Library # 0 (core). Path = libgmt.so
gmt [DEBUG]: Loading GMT plugins from: /home/core-man/.anaconda3/envs/pygmt/lib/gmt/plugins
gmt [DEBUG]: Shared Library # 1 (supplements). Path = /home/core-man/.anaconda3/envs/pygmt/lib/gmt/plugins/supplements.so
gmt [DEBUG]: Local file /home/core-man/.gmt/server/gmt_data_server.txt found
gmt [DEBUG]: File /home/core-man/.gmt/server/gmt_data_server.txt less than 24 hours old, refresh is premature.
gmt [DEBUG]: Load contents from /home/core-man/.gmt/server/gmt_data_server.txt
gmt [DEBUG]: Local file /home/core-man/.gmt/server/gmt_hash_server.txt found
gmt [DEBUG]: File /home/core-man/.gmt/server/gmt_hash_server.txt less than 24 hours old, refresh is premature.
gmt [DEBUG]: Map distance calculation will be using great circle approximation with authalic auxiliary latitudes and authalic (R_2) radius = 6371007.1809 m, in meter.
gmt [DEBUG]: Revised options: @earth_relief_01d_p -Vd
gmtwhich [DEBUG]: gmtapi_init_export: Passed family = Data Table and geometry = Text
gmtwhich [DEBUG]: Object ID 0 : Registered Data Table Stream 7fae37791400 as an Output resource with geometry Text [n_objects = 1]
gmtwhich [DEBUG]: gmtapi_init_export: Added stdout to registered destinations
gmtwhich [DEBUG]: GMT_Init_IO: Returned first Output object ID = 0
gmtwhich [DEBUG]: GMT_Begin_IO: Initialize record-by-record access for Output
gmtwhich [DEBUG]: gmtapi_next_io_source: Selected object 0
gmtwhich [INFORMATION]: Writing Data Table to Standard Output stream
gmtwhich [DEBUG]: GMT_Begin_IO: Output resource access is now enabled [record-by-record]
gmtwhich [DEBUG]: Look for file earth_relief_01d_p.grd in /home/core-man/.gmt
gmtwhich [DEBUG]: Look for file earth_relief_01d_p.grd in /home/core-man/.gmt/cache
gmtwhich [DEBUG]: Look for file earth_relief_01d_p.grd in /home/core-man/.gmt/server
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
gmtwhich [DEBUG]: GMT_End_IO: Output resource access is now disabled
gmtwhich [DEBUG]: gmtlib_unregister_io: Unregistering object no 0 [n_objects = 0]
gmtwhich (gmtlib_free_tmp_arrays): tried to free unallocated memory
gmt [DEBUG]: Entering GMT_Destroy_Session
seisman commented 3 years ago

It's weird. I don't see anything wrong in the output.

seisman commented 3 years ago

@PaulWessel Do you have any idea why gmt which can't find the remote file even though it's already downloaded?

PaulWessel commented 3 years ago

On the face of it, no. But if I delete my copy and run that command I get the same error. I think you need -G if you want to download a file via gmt which though.

seisman commented 3 years ago

@core-man Is it possible that the earth_relief_01d_p.grd file is not complete due to unstable internet connections? Could you check the file size and md5sum?

core-man commented 3 years ago

@core-man Is it possible that the earth_relief_01d_p.grd file is not complete due to unstable internet connections? Could you check the file size and md5sum?

@seisman

Download earth relief grid via load_earth_relief():

$ python
>>> from pygmt.datasets import load_earth_relief
>>> grid = load_earth_relief()
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]
gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 425, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/datasets/earth_relief.py", line 111, in load_earth_relief
    fname = which(f"@earth_relief_{resolution}{reg}", download="a")
  File "/home/core-man/src/pygmt/pygmt/helpers/decorators.py", line 281, in new_module
    return module_func(*args, **kwargs)
  File "/home/core-man/src/pygmt/pygmt/src/which.py", line 55, in which
    raise FileNotFoundError("File '{}' not found.".format(fname))
FileNotFoundError: File '@earth_relief_01d' not found.

Rename the grid to be earth_relief_01d_p_load.grd:

$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd
$ mv ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p_load.grd

Then I can use the grdimage to download and plot @earth_relief_01d.

import pygmt

fig = pygmt.Figure()
fig.grdimage(grid="@earth_relief_01d")
fig.show()
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd  earth_relief_01d_p_load.grd

The two grids are the same.

$ du -h ~/.gmt/server/earth/earth_relief/*
116K    /home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
116K    /home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd

$ diff ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd

$ md5sum ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd
74a884c902015dda516d17605f317efe  /home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
74a884c902015dda516d17605f317efe  /home/core-man/.gmt/server/earth/earth_relief/earth_relief_01d_p_load.grd
core-man commented 3 years ago

On the face of it, no. But if I delete my copy and run that command I get the same error. I think you need -G if you want to download a file via gmt which though.

@seisman @PaulWessel Yes, I can use -G to download and find the grid.

# work in ~/workspace folder
$ cd ~/workspace

# use `-Gl` option
$ ls
$ gmt which -Gl @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls
earth_relief_01d_p.grd

# Use `-Ga` option
$ ls ~/.gmt/server/earth/earth_relief/
$ gmt which -Ga @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

# those two grids are the same
$ diff ./earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd

However, there could still exist some bugs. A strange thing occurred. If the grid is the current folder is deleted and the grid in ~/.gmt/server/earth/earth_relief/ is kept, neither gmt which -Gl nor gmt which -Ga can work.

# I am still in ~/workspace
$ ls
earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

# remove the grid in the current folder
$ rm ./earth_relief_01d_p.grd

# gmt which cannot work
$ gmt which -Gl @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ gmt which -Ga @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

I removed ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd. I then ran commands in the above first window again and they work.

# I am still in ~/workspace which is empty now
$ ls
$ rm ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/

# use `-Gl` option
$ ls
$ gmt which -Gl @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls
earth_relief_01d_p.grd

# Use `-Ga` option
$ ls ~/.gmt/server/earth/earth_relief/
$ gmt which -Ga @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
earth_relief_01d_p.grd
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

Finally, I removed the grids in the current folder and ~/.gmt/server/earth/earth_relief/. gmt which -Ga can download the grid but cannot find the grid.

# I am still in ~/workspace
$ rm earth_relief_01d_p.grd ~/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd

$ gmt which @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ ls
$ ls ~/.gmt/server/earth/earth_relief/

$ gmt which -Ga @earth_relief_01d_p
gmtwhich [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
gmtwhich [NOTICE]:   -> Download grid file [115K]: earth_relief_01d_p.grd
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

$ ls
$ ls ~/.gmt/server/earth/earth_relief/
earth_relief_01d_p.grd

$ gmt which @earth_relief_01d_p 
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!

$ gmt which -Gl @earth_relief_01d_p
gmtwhich [ERROR]: File earth_relief_01d_p.grd not found!
$ ls

In summary:

  1. If there is the same grid in ~/.gmt/server/earth/earth_relief/, gmt which -Gl cannot work.
  2. If there is no the same grid in the current folder, gmt which -Ga can download the grid to ~/.gmt/server/earth/earth_relief but cannot find it.
  3. If there is the same grid in the current folder, gmt which -Ga can download the grid to ~/.gmt/server/earth/earth_relief and can find a grid with the same name.
  4. It seems that the file gmt which -Ga tries to find is the grid in the current folder instead of ~/.gmt/server/earth/earth_relief.
seisman commented 3 years ago

Thanks for the detailed reports. However, I followed your steps but still can't reproduce your issue. I'm using GMT 6.1.1 (installed using conda) on macOS.

core-man commented 3 years ago

Thanks for the detailed reports. However, I followed your steps but still can't reproduce your issue. I'm using GMT 6.1.1 (installed using conda) on macOS.

I use GMT 6.1.1 (installed using conda) CentOS7. I don't have this issue on my macOS either.

seisman commented 3 years ago

I just tried GMT 6.1.1 on CentOS 7.8. Still can't reproduce your issue.

core-man commented 3 years ago

I just tried GMT 6.1.1 on CentOS 7.8. Still can't reproduce your issue.

I use CentOS 7.5. I also use the GMT built from the source code (6.2.0_416e03f_2020.08.18) instead of conda and it also has the same bug.

If only I have the bug, it should be okay cause I also work on macOS. Maybe it is caused by an unknown/strange thing in my CentOS. Shall we close this issue or open it until someone also have this issue?

seisman commented 3 years ago

I will close this one. Feel free to reopen it if someone else can reproduce the issue or have a clue why it doesn't work.

maxrjones commented 3 years ago

I don't have a way to reproduce the issue, but I wanted to point out that this seems similar in some ways to an issue that I had earlier which could not be reproduced by other users: https://github.com/GenericMappingTools/gmt/issues/4522. So, it could be related to the specific user environment rather than the OS.