search

GenericMappingTools / gmtserver-admin

Cache data and script for managing the GMT data server
GNU Lesser General Public License v3.0
7 stars 3 forks source link

Cannot get past one tile #88

Closed PaulWessel closed 3 years ago

PaulWessel commented 3 years ago

I want to download all the 30 second tiles:

gmt grdinfo @earth_relief_30s -G

However, I cannot get past this tile: N60W135, even if restarting after failure:

gmt grdinfo @earth_relief_30s -G    
grdblend [NOTICE]: Remote data courtesy of GMT data server oceania [http://oceania.generic-mapping-tools.org]

grdblend [NOTICE]: Earth Relief at 30x30 arc seconds from Gaussian Cartesian filtering (1.0 km fullwidth) of SRTM15+V2.1 [Tozer et al., 2019].
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W135
grdblend [ERROR]: Libcurl Error: Couldn't resolve host name
grdblend [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
grdconvert [ERROR]: File /Users/pwessel/.gmt/server/earth/earth_relief/earth_relief_30s_p/N60W135.earth_relief_30s_p.jp2 was not found
grdconvert [ERROR]: Cannot find file /Users/pwessel/.gmt/server/earth/earth_relief/earth_relief_30s_p/N60W135.earth_relief_30s_p.jp2
grdblend [ERROR]: ERROR - Unable to convert SRTM file /Users/pwessel/.gmt/server/earth/earth_relief/earth_relief_30s_p/N60W135.earth_relief_30s_p.jp2 to compressed netCDF format

It got 243 tiles before then and I have 300 Gb of free space on my drive. Permissions on the server are all consistent across the tiles. But above it says cannot resolve host name. Network is up.

Any ideas? Pinging @joa-quim @seisman @meghanrjones

PaulWessel commented 3 years ago

Starting up a gain selecting the pixel files ends at exactly the same tile:

grdconvert [ERROR]: File /Users/pwessel/.gmt/server/earth/earth_relief/earth_relief_30s_g/N60W135.earth_relief_30s_g.jp2 was not found

This is attempting to download tile number 244 in both cases.

seisman commented 3 years ago

I just started a fresh downlaod, and can't reproduce your issue:

grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N45E090
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N45E105
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N45E120
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N45E135
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N45E150
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N45E165
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W180
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W165
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W150
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W135
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W120
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W105
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W090
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W075
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W060
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W045
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W030
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60W015
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60E000
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60E015
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60E030
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60E045
grdblend [NOTICE]:   -> Download 15x15 degree grid tile (earth_relief_30s_p): N60E060
maxrjones commented 3 years ago

I am getting the same issue. @seisman, I do not see it having downloaded N60W135 in your example.

seisman commented 3 years ago

It's downloaded: image

PaulWessel commented 3 years ago

Still haveing the same issue in terminal, but in debugging in Xcode it is now running past that step. Strange!

PaulWessel commented 3 years ago

Xcode got all 288. Now I run the grdblend command in the pre script manuallly:

gmt grdgradient @earth_relief_30s -A90 -Nt2.5 -Gearth_relief_30s+2.5_int.nc
`ERROR 4: /Users/pwessel/.gmt/server/earth/earth_relief/earth_relief_30s_p/N60E015.earth_relief_30s_p.nc: Too many open files`

This is after a reboot to clean up whatever open files the system thinks it has. Opening 288 files should not be an issue since the limit is huge.

PaulWessel commented 3 years ago

Stupid, stupid, stupid.

Of course, when one upgrades the macOS, previously set ulimits evaoprate, returning to the cosy limit of 256 open files. My tile list has 288. This is an idiotically low limit. @meghanrjones you can do

ulimit -n 10240

but there are more permanent ways - just have to search a bit more.

joa-quim commented 3 years ago

Hmmm, but why are all those files opened at same time?

PaulWessel commented 3 years ago

Just the way grdblend was built: It loops over all the files and opens all files that overlap with the final -R. I agree this is not needed - they could be opened the first time they are accessed. I will look at that. But macos has silly default limits so this could be an issue for the common man on macOS.