Recent changes in master branch break PyGMT grdimage

[seisman]

Description of the problem

PyGMT is running a daily cron job for GMT master branch. All the tests passed yesterday (with commit bdb7bfcb4d3839b1b043171973a1bb7dc926a4bf) but two grdimage tests failed today (with commit 0c8971e79138a3ca78029c10a3c7641c473168e9).

Here are all the commits between the two commits tested by PyGMT: https://github.com/GenericMappingTools/gmt/compare/bdb7bfcb4d3839b1b043171973a1bb7dc926a4bf...0c8971e79138a3ca78029c10a3c7641c473168e9. Commits f210ad3, 7d3f78f and 0a66bc9 may break PyGMT.

Here is an example failing test in PyGMT. The test plots the 01d earth relief data, but the output is:

[PaulWessel]

Sorry, dinnertime. Anyway you can isolate out the commit that gave GMTAPI_MEMFILE_LEN and see? Is PyGMT calling GMT_Open_VirtualFile and getting those strings? Perhaps there is an extra character at the end so my count fails?

[seisman]

It crashes in commit f210ad39e57f4a3544d63686e1a8fbbb8a057d26 but works in previous commit.

[seisman]

@PaulWessel Here is the PyGMT script. The script reads the 01 earth relief data into xarray and pass it to grdimage.

import pygmt

grid = pygmt.datasets.load_earth_relief("01d")
fig = pygmt.Figure()
fig.grdimage(grid, projection="W0/6i", cmap='earth', V='d')
fig.savefig("map.pdf")

The above script provides the black/white plot, but if I change earth to geo or relief, it can produce the correct plot. (I'm using the GMT master branch). Something wrong with earth.cpt?

Here are the verbose messages for earth or geo CPTs:

• geo.log
• earth.log

Please see if you can find anything that makes no sense.

[PaulWessel]

There are lots of these messages:

grdimage (gmtsupport_free_range): tried to free unallocated memory

Are you passing the earth cpt as a virtual file or just a text string that ends up as a -Cearth option to grdimage?

[seisman]

Are you passing the earth cpt as a virtual file or just a text string that ends up as a -Cearth option to grdimage?

Just a text string.

I saw these messages for geo, but not for earth.

grdimage [INFORMATION]: Auto-stretching CPT file geo to fit data range -8182 to 5651.5
grdimage [DEBUG]: Found CPT hinge at z' = 0 for slice k = 8!
grdimage [DEBUG]: gmtlib_get_graphics_item: Fig: 1 Subplot: 2 Panel: () Inset: 0
grdimage [DEBUG]: Object ID 6 : Registered CPT File /Users/seisman/.gmt/sessions/gmt_session.93093/gmt.1.cpt as an Output resource with geometry Non-Geographical [n_objects = 4]
grdimage [DEBUG]: gmtapi_begin_io: Output resource access is now enabled [container]
grdimage [DEBUG]: gmtapi_export_palette: Passed ID = 6 and mode = 0
grdimage [INFORMATION]: Write CPT to File /Users/seisman/.gmt/sessions/gmt_session.93093/gmt.1.cpt
grdimage [DEBUG]: Writing CPT to /Users/seisman/.gmt/sessions/gmt_session.93093/gmt.1.cpt
grdimage [DEBUG]: GMT_End_IO: Output resource access is now disabled
grdimage [INFORMATION]: Save current CPT file to /Users/seisman/.gmt/sessions/gmt_session.93093/gmt.1.cpt !

[PaulWessel]

So the command line version has the same messages for earth. In your case, it behaves like it is not a master table. I have added new debug messages to clarify this for us in the put-strings branch.

[seisman]

I'm using the put_strings branch.

For geo,

grdimage [DEBUG]: gmt_get_filename: In: geo Out: geo
grdimage [DEBUG]: Look for file geo in /Users/seisman/.gmt
grdimage [DEBUG]: Look for file geo in /Users/seisman/.gmt/cache
grdimage [DEBUG]: Look for file geo in /Users/seisman/.gmt/server
grdimage [DEBUG]: CPT argument geo understood to be a master table
grdimage [DEBUG]: GMT: 0. Will try to find subdir=cpt stem = geo suffix=.cpt
grdimage [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
grdimage [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /Users/seisman/.gmt
grdimage [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /Users/seisman/.gmt/cpt
grdimage [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /Users/seisman/local/GMT/share/cpt
grdimage [DEBUG]: gmt_get_filename: In: /Users/seisman/local/GMT/share/cpt/geo.cpt Out: /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: Given full path to file /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: gmt_get_filename: In: /Users/seisman/local/GMT/share/cpt/geo.cpt Out: /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: gmt_get_filename: In: /Users/seisman/local/GMT/share/cpt/geo.cpt Out: /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: Found readable file /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: Object ID 5 : Registered CPT File /Users/seisman/local/GMT/share/cpt/geo.cpt as an Input resource with geometry Non-Geographical [n_objects = 3]
grdimage [DEBUG]: gmtapi_begin_io: Input resource access is now enabled [container]
grdimage [DEBUG]: gmtapi_import_palette: Passed ID = 5 and mode = 8
grdimage [INFORMATION]: Reading CPT from File /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: Reading CPT from /Users/seisman/local/GMT/share/cpt/geo.cpt
grdimage [DEBUG]: GMT_End_IO: Input resource access is now disabled
grdimage [INFORMATION]: Auto-stretching CPT file geo to fit data range -8182 to 5651.5
grdimage [DEBUG]: Found CPT hinge at z' = 0 for slice k = 8!
grdimage [DEBUG]: gmtlib_get_graphics_item: Fig: 1 Subplot: 2 Panel: () Inset: 0
grdimage [DEBUG]: Object ID 6 : Registered CPT File /Users/seisman/.gmt/sessions/gmt_session.96183/gmt.1.cpt as an Output resource with geometry Non-Geographical [n_objects = 4]
grdimage [DEBUG]: gmtapi_begin_io: Output resource access is now enabled [container]
grdimage [DEBUG]: gmtapi_export_palette: Passed ID = 6 and mode = 0
grdimage [INFORMATION]: Write CPT to File /Users/seisman/.gmt/sessions/gmt_session.96183/gmt.1.cpt
grdimage [DEBUG]: Writing CPT to /Users/seisman/.gmt/sessions/gmt_session.96183/gmt.1.cpt
grdimage [DEBUG]: GMT_End_IO: Output resource access is now disabled

For earth,

grdimage [DEBUG]: gmt_get_filename: In: earth Out: earth
grdimage [DEBUG]: Look for file earth in /Users/seisman/.gmt
grdimage [DEBUG]: Look for file earth in /Users/seisman/.gmt/cache
grdimage [DEBUG]: Look for file earth in /Users/seisman/.gmt/server
grdimage [DEBUG]: Found readable file /Users/seisman/.gmt/server/earth
grdimage [DEBUG]: CPT argument earth understood to be a regular CPT table
grdimage [DEBUG]: GMT: 0. Will try to find subdir=cpt stem = earth suffix=.cpt
grdimage [DEBUG]: GMT: 1. gmt_getsharepath trying current dir
grdimage [DEBUG]: GMT: 2. gmt_getsharepath trying USERDIR /Users/seisman/.gmt
grdimage [DEBUG]: GMT: 3. gmt_getsharepath trying USERDIR subdir /Users/seisman/.gmt/cpt
grdimage [DEBUG]: GMT: 4. gmt_getsharepath trying SHAREDIR subdir /Users/seisman/local/GMT/share/cpt
grdimage [DEBUG]: gmt_get_filename: In: /Users/seisman/local/GMT/share/cpt/earth.cpt Out: /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: Given full path to file /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: gmt_get_filename: In: /Users/seisman/local/GMT/share/cpt/earth.cpt Out: /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: gmt_get_filename: In: /Users/seisman/local/GMT/share/cpt/earth.cpt Out: /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: Found readable file /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: Object ID 5 : Registered CPT File /Users/seisman/local/GMT/share/cpt/earth.cpt as an Input resource with geometry Non-Geographical [n_objects = 3]
grdimage [DEBUG]: gmtapi_begin_io: Input resource access is now enabled [container]
grdimage [DEBUG]: gmtapi_import_palette: Passed ID = 5 and mode = 0
grdimage [INFORMATION]: Reading CPT from File /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: Reading CPT from /Users/seisman/local/GMT/share/cpt/earth.cpt
grdimage [DEBUG]: GMT_End_IO: Input resource access is now disabled

I think Found readable file /Users/seisman/.gmt/server/earth is the key issue.

[PaulWessel]

Hm, OK. So our check for access to a file does not distinguish files from directories. I think this requires a stat call in gmt_getdatapath and gmt_getsharepath.

[PaulWessel]

Added check for directory. See if this gets caught with your case.

[seisman]

Now it works, but you should open new PR for it, and backport it to 6.1.

[seisman]

One more thing, when I run gmt grdimage @earth_relief_01d -Cearth -JW0/10c -Vd -pdf map, I see:

grdimage [DEBUG]: gmt_get_filename: In: -180/180/-90/90 Out: -180/180/-90/90
grdimage [DEBUG]: Look for file -180/180/-90/90 in /Users/seisman/.gmt
grdimage [DEBUG]: Look for file -180/180/-90/90 in /Users/seisman/.gmt/cache
grdimage [DEBUG]: Look for file -180/180/-90/90 in /Users/seisman/.gmt/server
grdimage [DEBUG]: Got regular w/e/s/n for region (-180/180/-90/90)
grdimage [DEBUG]: Remote file @earth_relief_01d_p.grd exists locally as /Users/seisman/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
grdimage [DEBUG]: Replace file @earth_relief_01d_p.grd with /Users/seisman/.gmt/server/earth/earth_relief/earth_relief_01d_p.grd
grdimage [DEBUG]: gmt_get_filename: In: earth Out: earth
grdimage [DEBUG]: Look for file earth.grd in /Users/seisman/.gmt
grdimage [DEBUG]: Look for file earth.grd in /Users/seisman/.gmt/cache
grdimage [DEBUG]: Look for file earth.grd in /Users/seisman/.gmt/server

Why GMT looks for file earth.grd in command line version?

[PaulWessel]

I get that too:

grdimage [DEBUG]: Look for file earth.grd in /Users/pwessel/.gmt/cache grdimage [DEBUG]: Look for file earth.grd in /Users/pwessel/.gmt/server

but no search for geo.grd, etc. So a similar issue - will need to track that down too.

[PaulWessel]

Testing a fix but suddenly libcur?

gmt grdimage @earth_relief_01d -Cearth -JW0/10c -V -pdf map begin [INFORMATION]: Creating a workflow directory /Users/pwessel/.gmt/sessions/gmt_session.729 grdimage [NOTICE]: Remote data courtesy of GMT data server OCEANIA [https://oceania.generic-mapping-tools.org]

grdimage [NOTICE]: Earth Relief at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019]. grdimage [NOTICE]: -> Download grid file [115K]: earth_relief_01d_p.grd grdimage [INFORMATION]: Downloading file https://oceania.generic-mapping-tools.org/cache/earth_relief_01d_p.grd ... grdimage [ERROR]: Libcurl Error: HTTP response code said error

I ssh'ed in and gmtserver is alive. Do the grdimage work for you. This was working minutes ago

[PaulWessel]

Sorry, I see the cache directory in that path. OK hold on

[PaulWessel]

Sorry, need more time to work on this and have a dr appointment downtown. I opened a WIP PR so you can see where I am on this. It still is failing for a few cases and end up looking in cache for a remote file.

[PaulWessel]

Think I fixed it - please give that WIP PR a spin. Back in a few hours.

[PaulWessel]

Please close if indeed fixed.