Testing of new data sets before public release

[PaulWessel]

Pinging @GenericMappingTools/gmt-team. I have uploaded new remote data grids to the test server. These are described here. Note that until we complete the dvc move for test images we have not yet updated the _earthrelief files (they are still based off SRTM15+ v2.1 from 2019). Once that is settled I will update those files to 2.3 and add the synbath grid as well. Because of this, if you run all the GMT tests they still all pass since all the DEM images are made using the same DEM as was used in the originals (once we update the DEM we will have ~60 failures...).

In the meantime, I hope you can find time to play a bit with the new datasets. To do so, and not mess up your other remote files, I suggest you do this:

Edit your ConfigUserAdvanced.cmake file (perhaps after updating via the Template) and uncomment this line:

# set (GMT_DATA_SERVER "test")

then rebuild GMT. To preserve your existing cache and server files, simply set them aside via

mv ~/.gmt/server ~/.gmt/server_orig
mv ~/.gmt/cache ~/.gmt/cache_orig

so that you will only use files from the test server. You can now try out making maps with @earth_{gebco,gebcosi,geoid,mag,mag4km,vgg,wdmam} - they all have their own default CPTs that will be downloaded via the cache.

Let me know if you find anything odd. The default CPTs are the ones recommended by the data provider, except Gebco which has not replied. I am therefore using the same geo CPT for all the reliefs.

Once you have finished testing, remove the new directories and rename the old:

rm -f /.gmt/server  ~/.gmt/cache 
mv ~/.gmt/server_orig ~/.gmt/server 
mv ~/.gmt/cache_orig ~/.gmt/cache

[Esteban82]

@Paul I could only make maps from the GEBCO grids and look fine.

For the other grids (geoid,mag,mag4km,vgg,wdmam) I got this message: Are those grids available? Or am I doing something wrong?

gmt grdimage @earth_mag -JW15c -Bg0 -Rd -png geoid

grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
grdimage [ERROR]: File /home/federico/.gmt/cache/earth_mag was not found
grdimage [ERROR]: Cannot find file /home/federico/.gmt/cache/earth_mag
grdimage [ERROR]: File /home/federico/.gmt/cache/earth_mag was not found
grdimage [ERROR]: Cannot find file /home/federico/.gmt/cache/earth_mag
grdimage [ERROR]: File /home/federico/.gmt/cache/earth_mag not found
[Session gmt (0)]: Error returned from GMT API: GMT_FILE_NOT_FOUND (16)

[PaulWessel]

Interesting, it works if we specify a resolution:

gmt grdimage @earth_mag_06m -JM12c -RFR -B -pdf t

I will have a look.

[Esteban82]

I have an error with finer resolutions

gmt grdimage @earth_gebco -JM15c -Bg0 -png gebco -R44/50/33/40
grdblend [NOTICE]: Remote data courtesy of GMT data server test [http://test.generic-mapping-tools.org]

grdblend [NOTICE]: GEBCO Earth Relief at 3x3 arc seconds tiles provided by SRTMGL3 (land only) [NASA/USGS].
grdblend [NOTICE]:   -> Download 1x1 degree grid tile (earth_gebco_03s_g): N33E044
grdblend [ERROR]: Libcurl Error: HTTP response code said error
grdconvert [ERROR]: File /home/federico/.gmt/server/earth/earth_gebco/earth_gebco_03s_g/N33E044.earth_gebco_03s_g.jp2 was not found
grdconvert [ERROR]: Cannot find file /home/federico/.gmt/server/earth/earth_gebco/earth_gebco_03s_g/N33E044.earth_gebco_03s_g.jp2
grdblend [ERROR]: ERROR - Unable to convert SRTM file /home/federico/.gmt/server/earth/earth_gebco/earth_gebco_03s_g/N33E044.earth_gebco_03s_g.jp2 to compressed netCDF format
grdblend [ERROR]: Unable to obtain remote file @N33E044.earth_gebco_03s_g.nc
grdblend [NOTICE]:   -> Download 1x1 degree grid tile (earth_gebco_03s_g): N33E044
grdblend [ERROR]: Libcurl Error: HTTP response code said error
...

[PaulWessel]

Thanks, this part is fixed via https://github.com/GenericMappingTools/gmt/pull/6151 but I also have to fix a few things with the server files. Some tiles got labelled N30E300.earth_mag_02m_p.jp2 instead of N30W060.earth_mag_02m_p.jp2 for instance and thus could not be found. But separate from the bug I fixed. For example

gmt grdimage @earth_faa -JM12c -RFR -B -pdf t

now works but mag runs into missing filenames which I will fix next.

[PaulWessel]

OK, the mag files should be fine now.

[Esteban82]

In mag there is no data south of -60º. Is it a limitation of the original data?
In the docs only says that: "This version is observed at sea level over oceanic regions and have no data over land."

gmt grdimage @earth_mag_06m -JW15c -Bg0 -png mag -Rd

[Esteban82]

Now I can download the rest of the grids.

I think that there is typo for the wdmam cpt. Grdimage searchs for earth_wdman.cpt (with N, not M).

gmt grdimage @earth_wdmam -JW15c -Bg0 -Rd -png wdmam

grdimage [ERROR]: GMT_Read_Data: File not found: earth_wdman.cpt

[PaulWessel]

I will check on the 60S issue. The 1s and 3s SRTM tiles issue will take more time and is unresolved.

I have fixed the wdman.cpt typo. You will need to delete the gmt_data_server.txt file in your .gmt/server directory so it can be downloaded again with the fix. Let me know if that works.

[PaulWessel]

The EMAG2 website only shows maps that stop at 60S and show nothing below 60S. Seems like a cutoff. The upward continued mag4km has data at the poles.

[PaulWessel]

The gebco issue should be fixed by https://github.com/GenericMappingTools/gmt/pull/6153.

[Esteban82]

Now, all the issues that I reported are working fine.

[Esteban82]

The EMAG2 website only shows maps that stop at 60S and show nothing below 60S. Seems like a cutoff. The upward continued mag4km has data at the poles.

Yes, I check it. It seems that there is no official data below 60S.

[maxrjones]

Why are there some redundant grid files at https://oceania.generic-mapping-tools.org/ and https://oceania.generic-mapping-tools.org/server/earth/earth_relief/ (e.g., earth_relief_01d.grd, earth_relief_60m.grd)?

[maxrjones]

This may not be a cause for concern since it relates to non-updated files (earth_relief_01d_g.grd), but test/api/apimat2grd.sh fails when using the test server.

[maxrjones]

I'm running into issues where the download is attempted using the cache dir rather than the appropriate server path:

(base) DarthMac:new-datasets meghanj$ gmt grdcut @earth_geoid_01d.grd -Rg -Ggeoid_01d.nc -V
grdcut [NOTICE]: Remote data courtesy of GMT data server test [http://test.generic-mapping-tools.org]

grdcut [NOTICE]: EGM2008 geoid at 1x1 arc degrees from Gaussian Cartesian filtering (111 km fullwidth) of the 1m grid [Pavlis et al., 2012].
grdcut [NOTICE]:   -> Download grid file [99K]: earth_geoid_01d_g.grd
grdcut [INFORMATION]: Downloading file http://test.generic-mapping-tools.org/cache/earth_geoid_01d.grd ...
grdcut [ERROR]: Libcurl Error: HTTP response code said error

[PaulWessel]

Can confirm, will debug.

[maxrjones]

The earth_geoid colormap is missing: grdimage [ERROR]: GMT_Read_Data: File not found: earth_geoid.cpt.

[PaulWessel]

Yes it is. I must have made an earth_geoid.cpt at home and forgot to add it. I will do that later. I know I have made geoid maps at home.

[PaulWessel]

Actually, no such thing. I was using turbo. So I have replaced @earth_geoid.cpt with turbo in the gmt_data_server.txt file and you will need to remove the old one or wait 24 hours..

[maxrjones]

Running into another issue with the earth_geoid dataset:

gmt grdimage @earth_geoid_05m_g -Rg -JH15c -B -png test

grdblend [NOTICE]: EGM2008 geoid at 5x5 arc minutes from Gaussian Cartesian filtering (9 km fullwidth) of the 1m grid [Pavlis et al., 2012].
grdblend [NOTICE]:   -> Download 180x180 degree grid tile (earth_geoid_05m_g): S90W180
grdblend [ERROR]: Libcurl Error: HTTP response code said error
grdblend [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
grdconvert [ERROR]: File /Users/meghanj/.gmt/server/earth/earth_geoid/earth_geoid_05m_g/S90W180.earth_geoid_05m_g.jp2 was not found

[PaulWessel]

That does it - I must have messed up something in the srv_tiler.sh script or similar. I have noticed many tiles got named wrongly (some had E330 instead of W030 and now this with being named E180 but GMT looks for W180... I may have to redo the processing once I find the culprit.

[maxrjones]

OK. Another issue: I am confused why the following script tries to read a colormap from stdin rather than using the session cpt (this example works for the other datasets):

gmt begin test png
gmt grdimage @earth_mag_01d_g -Rg -JH15c -B
gmt colorbar -V
gmt end show

[PaulWessel]

All files for earth_geoid, earth_mag and earth_mag4km have been updated after fixing the -Rg vs -Rd issue. Your first geoid example now works.

[PaulWessel]

The cpt stdin issue seems to be a new bug (?) and affects older remote files as well. Will be looking at that now.

[PaulWessel]

OK. Another issue: I am confused why the following script tries to read a colormap from stdin rather than using the session cpt (this example works for the other datasets):

gmt begin test png
gmt grdimage @earth_mag_01d_g -Rg -JH15c -B
gmt colorbar -V
gmt end show

This should be fixed by https://github.com/GenericMappingTools/gmt/pull/6171.

[maxrjones]

The results for earth_mag05m look off for both pixel and gridline:  Here's the same for earth_mag06m: 

[PaulWessel]

Oddly the 6m grid has the specified scale of 0.2 and offset of 800 while the 5m grids have 0.03125 and 100, which what the values are for earth_vgg. Not sure how that is possible but rerunning earth_mag now.

[PaulWessel]

Mystery solved and fixed: For the tiled resolutions we use JP2000 format on the server - which has no header etc. When gmt downloads and converts these to netCDF locally it needs to know stuff - which it gets from gmt_data_server.txt. If the author of that file forgets to set the metadata for the three mag data to, say, that of VGG then we notice once tiles are selected. Also fixed gebco which has a scale of 1 since the authors of that data set made the decision to only store integer meters.

Solution for you is to wipe your gmt_server_data.txt file as well as the earth/* directory associated with mag, mag4km, wdmam, gebco, and gebcosi.

I made 05 and 06 maps for all these and all look fine now.

[PaulWessel]

I will make a test script for doing these 5 vs 6 maps for all data since a good and only check for this problem.

[maxrjones]

I will make a test script for doing these 5 vs 6 maps for all data since a good and only check for this problem.

Great, I think we could store useful test scripts like this in https://github.com/GenericMappingTools/remote-datasets/tree/main/scripts.

[maxrjones]

A few questions/comments from running grdinfo on the new/updated datasets:

1. Why do some include the command (e.g., earth_faa) while others do not (e.g., earth_mag)?
2. For the gebco grids, "GEBCO Compilation Group, 2020 GEBCO 2021 Grid" should be "GEBCO Compilation Group, 2021 GEBCO 2021 Grid". Also, the DOI agrees with that written on their site, but still leads to a 404 error. Is there anything we can do here?
3. Most have a remark that makes it clear how the grid relates to the data source, except earth_faa and earth_vgg (e.g., Obtained by Gaussian Cartesian filtering (111.2 km fullwidth) from /tmp/19294/extend.grd. Should from tmp... be more descriptive?
4. The earth_synbath and earth_relief preprint citation is 2021 (https://doi.org/10.1002/essoar.10508279.2). Does the remark list 2022 instead because the publish date will change for the final version? If so will the DOI still be the same for the published version and preprint?
5. The year for earth_vgg citation is 2021 but the remark lists 2019 (https://doi.org/10.1016/j.asr.2019.09.011).

[maxrjones]

For the gridline registered grids, there is often empty space between the map border and grid edge:

gmt grdimage @earth_synbath_01d_g -R110/140/20/35 -JB125/20/25/45/12c -B -png test_synbath_g

[PaulWessel]

A few questions/comments from running grdinfo on the new/updated datasets:

1. Why do some include the command (e.g., earth_faa) while others do not (e.g., earth_mag)?

Unclear, but there is a different path since the EMAG etc files are zipped ascii tables that needs some preprocessing first.

2. For the gebco grids, "GEBCO Compilation Group, 2020 GEBCO 2021 Grid" should be "GEBCO Compilation Group, 2021 GEBCO 2021 Grid". Also, the DOI agrees with that written on their site, but still leads to a 404 error. Is there anything we can do here?

I have sent a request to the host so I guess we can wait a few days to see if it can get resolved. I have updated the year in the recipe.

3. Most have a remark that makes it clear how the grid relates to the data source, except earth_faa and earth_vgg (e.g., Obtained by Gaussian Cartesian filtering (111.2 km fullwidth) from /tmp/19294/extend.grd. Should from tmp... be more descriptive?

I need to make a temp file to extend the data to the poles. I could change extend.grd to the original file name instead.

4. The earth_synbath and earth_relief preprint citation is 2021 (https://doi.org/10.1002/essoar.10508279.2). Does the remark list 2022 instead because the publish date will change for the final version? If so will the DOI still be the same for the published version and preprint?

Hm, it is accepted for publication in early 2022 so normally that would mean a 2022 publication year - despite what the preprint says, no?

5. The year for earth_vgg citation is 2021 but the remark lists 2019 (https://doi.org/10.1016/j.asr.2019.09.011).

David updates the grids at least once, sometimes twice a year as data continues to come in, but does not publish papers each time. So 2019 is correct but I should add "Version 31" to the titles I think.

[PaulWessel]

For the gridline registered grids, there is often empty space between the map border and grid edge:

gmt grdimage @earth_synbath_01d_g -R110/140/20/35 -JB125/20/25/45/12c -B -png test_synbath_g

Probably related to recent change of the double-padding removal. I will try to debug this and see why this case gets short-changed.

[PaulWessel]

Since this is a GMT bug, maybe create an issue on the GMT issues site using earth_relief instead?

[PaulWessel]

I will rebuild VGG, FAA but wait on Gebco. I will see if I can learn why mag has no command.

[maxrjones]

The PyGMT tests that use the SRTM data are failing with the test server due to tiles not being found. Can you try this to see if it works for you?

gmt grdcut @srtm_relief_03s_g -R135/136/35/36 -Gtest.nc

[anbj]

Any plans to include the International Bathymetric Chart of the Southern Ocean (IBCSO) and International Bathymetric Chart of the Arctic Ocean (IBCAO)? Or are these included in one of the other data sets?

[PaulWessel]

I forgot to copy over the STRM tiles after I deleted and rebuilt the new earth_relief files... Now fixed.

[PaulWessel]

Any plans to include the International Bathymetric Chart of the Southern Ocean (IBCSO) and International Bathymetric Chart of the Arctic Ocean (IBCAO)? Or are these included in one of the other data sets?

I thought they are in the GEBCO grid but perhaps not? Anyway, they are not global grids.

[anbj]

Any plans to include the International Bathymetric Chart of the Southern Ocean (IBCSO) and International Bathymetric Chart of the Arctic Ocean (IBCAO)? Or are these included in one of the other data sets?

I thought they are in the GEBCO grid but perhaps not? Anyway, they are not global grids.

I think you are right; they're included in GEBCO.

[PaulWessel]

FAA and VGG files updated on test server

[maxrjones]

@GenericMappingTools/pygmt-maintainers @GenericMappingTools/gmt-maintainers, the plan is to update earth_relief and add the new datasets to the server on Jan 27, 2022.

[Esteban82]

I have a problem with @earth_wdmam_02m. I got the below message. It works with other grids and resolutions (03m, 04m, 05m).

mt grdimage @earth_wdmam_02m -Rd -png mpa
grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
grdimage [ERROR]: File /home/federico/.gmt/cache/earth_wdmam_02m was not found
grdimage [ERROR]: Cannot find file /home/federico/.gmt/cache/earth_wdmam_02m
grdimage [ERROR]: File /home/federico/.gmt/cache/earth_wdmam_02m was not found
grdimage [ERROR]: Cannot find file /home/federico/.gmt/cache/earth_wdmam_02m
grdimage [ERROR]: File /home/federico/.gmt/cache/earth_wdmam_02m not found
[Session gmt (0)]: Error returned from GMT API: GMT_FILE_NOT_FOUND (16)

[Esteban82]

Is it possible that the 02m grid doesn't exist? And that the original version is 03m?

[PaulWessel]

Correct, there is no such thing as 02m. I see our docs is in error - will fix.