Cache data are stored in the wrong directory if GMT_DATA_SERVER is set via environmental variable or via "gmt set"

[seisman]

When GMT is built with the default GMT_DATA_SERVER settings, the default data server is oceania.

$ gmt --version
6.6.0_6d36f98_2024.04.09

$ gmt --show-dataserver
http://oceania.generic-mapping-tools.org

For testing, we want to use the static data server, so that the GMT tests won't fail due to updates of cache files or remote files. This can be done by either setting the environmental variable GMT_DATA_SERVER or via CLI option --GMT_DATA_SERVER=static, but it doesn't work well.

$ export GMT_DATA_SERVER=static   # Set via environmental variable

$ gmt --show-dataserver
http://static.generic-mapping-tools.org   # The URL is correct

Now running gmt which to download a remote file. The debug messages are long, but the most important messages are shown below:

$ gmt which -Ga @earth_relief_01d_g -Vd
...
gmtwhich [DEBUG]: Get remote file http://static.generic-mapping-tools.org/server/earth/earth_relief/earth_relief_01d_g.grd and write to /Users/seisman/.gmt/server/earth/earth_relief/earth_relief_01d_g.grd
gmtwhich [DEBUG]: Download http://static.generic-mapping-tools.org/server/earth/earth_relief/earth_relief_01d_g.grd to /Users/seisman/.gmt/server/earth/earth_relief/earth_relief_01d_g.grd
gmtwhich [NOTICE]: Remote data courtesy of GMT data server static [http://static.generic-mapping-tools.org]
...

As you can see, GMT tries to download from the static data server, but save the file into ~/.gmt/server/ rather than ~/.gmt/static/server.

The same thing happens when set GMT_DATA_SERVER configuration:

$ gmt set GMT_DATA_SERVER=static
$ gmt --show-dataserver
http://static.generic-mapping-tools.org

$ gmt which -Ga @earth_relief_01d_g -Vd
$ gmt which -Ga @earth_relief_01d_g -Vd --GMT_DATA_SERVER=static

Similarly,

export GMT_DATA_SERVER=static
$ gmt get -Dcache -V
gmtget [INFORMATION]: Downloading file http://static.generic-mapping-tools.org/gmt_data_server.txt ...
gmtget [INFORMATION]: Downloading file http://static.generic-mapping-tools.org/gmt_hash_server.txt ...
gmtget [ERROR]: Unable to access or read /Users/seisman/.gmt/server/gmt_hash_server.txt

gmt_hash_server.txt is downloaded to ~/.gmt/gmt_hash_server.txt, but GMT tries to read it in ~/.gmt/server/gmt_hash_server.txt.

I think this explains why @joa-quim has trouble using the static server https://github.com/GenericMappingTools/gmt/pull/8425#issuecomment-2040848702.

[joa-quim]

It might be that, but I've added this line in gmtest.in (which probably the best solution for this)

export GMT_DATA_SERVER="static"

and I see that in my .gmt the static files are under C:\Users\j\.gmt\static

I also don't understand why ou Linux CI tests consistently fail the texture tests while those pass on Win and Mac (they fail for me too due to different grids/images/PSs)

[seisman]

It might be that, but I've added this line in gmtest.in (which probably the best solution for this)

export GMT_DATA_SERVER="static"

and I see that in my .gmt the static files are under C:\Users\j\.gmt\static

I don't think it will work. As I understand it, currently the only solution is setting set (GMT_DATA_SERVER "static") in cmake/ConfigUser.cmake. So we need to fix the bug to make it work.

I also don't understand why ou Linux CI tests consistently fail the texture tests while those pass on Win and Mac (they fail for me too due to different grids/images/PSs)

For unknown reasons, the Linux CI (sometimes the Windows CI) fails to download some files from the GMT data server, thus the tests may fail. Hopefully, PR #8426 can solve the issue temporarily.

[joa-quim]

Yes, you are right gmtest.in is ingested by cmake at build time but I have done the same change in the build/test/gmtin and same tests keep failing. And I have the static files under C:\Users\j\.gmt\static. I keep suspecting that DVC is not updating my PS files, and neither those of the Linux CI. The texture failures are sistematic, not random failures due to internet connections fluctuations.

[seisman]

Yes, you are right gmtest.in is ingested by cmake at build time but I have done the same change in the build/test/gmtin and same tests keep failing. And I have the static files under C:\Users\j\.gmt\static. I keep suspecting that DVC is not updating my PS files, and neither those of the Linux CI. The texture failures are sistematic, not random failures due to internet connections fluctuations.

But in the latest CI run, all tests pass on Windows (https://github.com/GenericMappingTools/gmt/actions/runs/8614489245/job/23608035110) except the known failures.

[joa-quim]

Yes, I saw it but Linux had those failures I mentioned. And both Win and Mac passed all the tests, despite the issue with the static location.

[seisman]

Here are the failures in the latest Linux CI run (https://github.com/GenericMappingTools/gmt/actions/runs/8630998725/job/23658513524?pr=8426) in PR #8426:

    509 - test/grdfill/gridfill.sh (Failed)
    521 - test/grdgdal/gdal_nn.sh (Failed)
    552 - test/grdimage/image_vartrans.sh (Failed)
    647 - test/grdvector/shrink.sh (Failed)
    702 - test/mapproject/proj4.sh (Failed)
    738 - test/nearneighbor/nat_nn.sh (Failed)
    754 - test/potential/firmoviscous.sh (Timeout)
    755 - test/potential/firmoviscous2.sh (Timeout)

texture tests no longer fail on Linux.