Add lunar DEM file to remote server

[willschlitzer]

Per a discussion at the forum, I'm hoping to add a lunar topographic relief grid to the GMT remote server.

[PaulWessel]

Great. Just a heads up: I am actually trying to do some of the legwork for you and will do a PR soon to add the moon recipe. I thought I at least run in through a few resolutions before adding it.

[willschlitzer]

Thanks @PaulWessel! I'll hold off any work until your PR.

[PaulWessel]

PR safely updated and seems to work well.Unless you are on a mac with lots of RAM it would be good to see how well the process runs for you. Expect many hours.

[willschlitzer]

Currently running the script and it's converting the TIFF to a netCDF. You weren't joking about the RAM requirements; currently using 24 out of 32 GB. I'm on an Ubuntu 20.04 gaming desktop with an 8th gen i7, so I'm hoping it won't take too long. I did run into an issue the first time I ran the script; it stopped downloading after 4.6 GB and I received curl: (18) transfer closed with 3917136203 bytes remaining to read, so I'm not sure if that was a connection issue or something else. I deleted the downloaded portion and it seems to have completed the download the second time around!

[PaulWessel]

Interesting with curl doing that. Probably a glitch in the matrix somewhere as we say. If the downsampler step works well you will find the folder moon/moon_relief inside the staging folder and it will have all the grids at various resolutions. If you then run the tiler script then all the 5 min and higher resolutions will be split into tiles so there will then be directories with JP2 tiles inside that folder while the original global grids will be moved up to stain/tiled as safekeeping (we do not use those again or store them on the server).

[willschlitzer]

I just received this error and the process quit. Here is what I'm seeing on my terminal:

srv_downsampler_grid.sh: Must download original source https://planetarymaps.usgs.gov/mosaic/Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8100M  100 8100M    0     0  28.7M      0  0:04:41  0:04:41 --:--:-- 28.1M
srv_downsampler_grid.sh: Must convert original tif source to Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc
scripts/srv_downsampler_grid.sh: line 117: [: NaN: integer expression expected
Convert Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_14s_p.grd=ns+s0.5+o0
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 21766 Killed                  gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}

It looks like the download went fine but it ran into an issue when trying to run grdfilter. It looks like my home environment still has GMT 6.0.0, so I'm trying it with my PyGMT development environment that has GMT 6.4.0.

[willschlitzer]

Same issue with GMT 6.4:

moon/moon_relief/moon_relief_14s_p.grd exists - skipping
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 22132 Killed                  gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}

Any idea what the culprit may be? Here is my environment info:

GMT library information:
  binary version: 6.4.0
  cores: 12
  grid layout: rows
  library path: /home/will/miniconda3/envs/pygmt-dev/lib/libgmt.so
  padding: 2
  plugin dir: /home/will/miniconda3/envs/pygmt-dev/lib/gmt/plugins
  share dir: /home/will/miniconda3/envs/pygmt-dev/share/gmt
  version: 6.4.0

[PaulWessel]

OK, I think you will actually need to run from GMT master - I do not know how to do that within conda since I am from the olden days. But I have made GMT source changes the last 24 hours in responses to issues with the moon processing. So master is a requirement for now.

[joa-quim]

The problem with big files is that it's not enough to have enough RAM. There needs to exist a contiguous chunk of memory to hold the entire array. Maybe if you reboot and start from a cleanest as possible state it may make a difference ... or not.

[PaulWessel]

True, @joa-quim but also a must to use master since I fixed some issues. They may be macOS-specific though - hard to tell. The crash is in the section where I had to run grdfilter separately on the N and S hemispheres otherwise I got strange horizontal lines that @Esteban82 reported. Doing it this way (until I can find some subtle trouble) works - at least for me - and is only done on the 30s or finer resolutions. This and other issues are most likely some subtle 64-bit counting issue but hard to debug.

[willschlitzer]

Installed gmt from master (showing 6.5.0_e43114b_2023.01.04) and rebooted; still got the same result. It also used all of my RAM (32 GB) for about a minute prior to crashing.

Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167:  6018 Killed                  gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}

Unfortunately, I don't have any Apple hardware to try this out with. I'm not sure what troubleshooting to try next.

[PaulWessel]

Thanks for trying. What I guess you can try is to run the script with full verbose (bash -xv .....) so that you can see that command with all the actual arguments. Then copy paste and verify it still crashes (it should) but add -Vd so we might learn when it crashes. Then, just mess with -I and go from 15s to 30s to 01m and see if it stops crashing. I would think so as this seems to be related to trouble with 64-bit counting and not so much lack of memory since you have 32 Gb.

[PaulWessel]

THe 14 arc second grid as 4-byte float internally is about 16 Gb, and then it is the output grid at 15s which is almost 14 Gb. So just those two allocations push 30 Gb of memory. For "fun" I am trying this comment (which does not produce the right answer hence the S and N workaround:

gmt grdfilter moon_relief_14s_p.grd -Fg0.18 -I15s -rp -D1 -Gshit.nc=ns -V

and activity monitor says this process is using 45.68 Gb! I am not sure what the extra 15 Gb but maybe a weight grid for something. I will have to debug to understand that 3rd blob.

[PaulWessel]

Third blob is the area weight: Each node on a sphere controls less and less area going towards the poles so it is used in the convolution weighting. Obviously this could be rewritten as just a few columns instead of the entire grid since stuff is constant along rows except at the ends (If not 360).

[PaulWessel]

Note that this grdfilter improvement reduces total memory by about what the input grid needs. So instead of needing 32 Gb for a 16Gb input grid it is now just a hair over 16 Gb. So perhaps @willschlitzer's crashes is related to needing over 40 Gb to do the job.

[willschlitzer]

The grdfilter update appears to have worked; my computer made it through moon_relief_15s_g and went on to moon_relief_15s_p seemingly without issue (although it did get up to 29.1 GB of RAM). I'll plan on letting the script run entirely later this afternoon or overnight.

[willschlitzer]

@PaulWessel The srv_downsampler_grid.sh script appears to have worked on my Ubuntu machine, currently running srv_tiler.sh. Assuming it works, what else needs to be done, other than setting the default CPT?

[PaulWessel]

Get, I think that is it. As for the other data CPTs we have a mix of using GMT master FPTs (e.g, geo for DEM) but earth_faa.cpt for free-air anomalies. So for the moon I think we should create a moon_relief.cpt file that has a reasonable range to match the actual data range (Not exact but slightly rounded, .e.g., -9200/10800 for instance, and we add that to the cache system and with recipe saying @moon_relief.cpt we are OK. I did set that # DST_CPT=@moon_relief.cpt before you ran your job I hink, depending on if you updated. Once the tiler finishes (of perhaps it is just after the downsample finished) you should find a file called moon_relief_server.txt in the staging directory. It has all the meta data about each resolugion+registration and should list @moon_relief if so.

[PaulWessel]

So I made moon_relief.cpt simply via gmt makecpt -Cgray -T-9200/10800 > moon_relief.cpt and it looks like this. OK?

[PaulWessel]

@PaulWessel The srv_downsampler_grid.sh script appears to have worked on my Ubuntu machine, currently running srv_tiler.sh. Assuming it works, what else needs to be done, other than setting the default CPT?

The remaining parts are not quite automated and some requires ability to authentically in on the SOEST server. Maybe @maxrjones and @Esteban82 can remind me to what extent they were able to do uploads and work on oceania.soest.hawaii.edu? We had hoped that instead of using a computer in Hawaii, especially with me in Norway, we would have made a deal with UNAVCO or IRIS on hosting our data but they are merging and we are small fries in their workload so I have not been able to make that request yet. Meantime we are using my SOEST server. Because logging into the UH/SOEST network is needed and because they do not really want some "community account" like gmtadmin that we could all share around, I am not sure what the best next step is. You can learn most of what we do from README and MAINTENANCE but I am sure it is not complete.

But at some point we will need to transfer the 6+ Gb of tiles and files to oceania, update the test gmt_data_server.txt file, and then set GMT_DATA_SERVER to test.generic-mapping.tools.org and kick the tires, i.e., make a few maps to see if things actually work of if we forgot something important....

[willschlitzer]

Get, I think that is it. As for the other data CPTs we have a mix of using GMT master FPTs (e.g, geo for DEM) but earth_faa.cpt for free-air anomalies. So for the moon I think we should create a moon_relief.cpt file that has a reasonable range to match the actual data range (Not exact but slightly rounded, .e.g., -9200/10800 for instance, and we add that to the cache system and with recipe saying @moon_relief.cpt we are OK. I did set that # DST_CPT=@moon_relief.cpt before you ran your job I hink, depending on if you updated.

Sounds good about the new CPT. I'm still showing the default CPT as geo, I'll run git pull once the tiler script finishes.

Once the tiler finishes (of perhaps it is just after the downsample finished) you should find a file called moon_relief_server.txt in the staging directory. It has all the meta data about each resolugion+registration and should list @moon_relief if so.

It looks like moon_relief_server.txt was created as soon as the tiler script began running, and metadata is being added as the grids are being created (currently on the 15s grids).

[willschlitzer]

@PaulWessel The srv_downsampler_grid.sh script appears to have worked on my Ubuntu machine, currently running srv_tiler.sh. Assuming it works, what else needs to be done, other than setting the default CPT?

The remaining parts are not quite automated and some requires ability to authentically in on the SOEST server. Maybe @maxrjones and @Esteban82 can remind me to what extent they were able to do uploads and work on oceania.soest.hawaii.edu? We had hoped that instead of using a computer in Hawaii, especially with me in Norway, we would have made a deal with UNAVCO or IRIS on hosting our data but they are merging and we are small fries in their workload so I have not been able to make that request yet. Meantime we are using my SOEST server. Because logging into the UH/SOEST network is needed and because they do not really want some "community account" like gmtadmin that we could all share around, I am not sure what the best next step is. You can learn most of what we do from README and MAINTENANCE but I am sure it is not complete.

But at some point we will need to transfer the 6+ Gb of tiles and files to oceania, update the test gmt_data_server.txt file, and then set GMT_DATA_SERVER to test.generic-mapping.tools.org and kick the tires, i.e., make a few maps to see if things actually work of if we forgot something important....

I'll work on creating some plots, since it seems like I won't be able to do much with the hosting server. If I move the moon folder from staging to the .gmt/server folder and copy the contents of moon_relief_server.txt to gmt_data_server.txt, will I be able to call the grids with the standard remote file syntax (e.g. @moon_relief_30m_g) when using GMT/PyGMT?

[joa-quim]

That moon_relief.cpt is too dark. Maybe set minimum at ~100

[PaulWessel]

Yes that might work, but remember GMT may check the server every day and decide to override your files... I would do this:

1. Make a backup copy of the server directory, e.g., cp -r server server_orig
2. Now add in the moon_relief_server.txt stuff into the gmt_data_server.txt.
3. Maybe set GMT_DATA_UPDATE_INTERVAL to 0 to prevent any update while you play with the moon.

[PaulWessel]

That moon_relief.cpt is too dark. Maybe set minimum at ~100

Not sure what is best here. That gives 100-255 as the gray range and with any intensity shade we have basically less visible range. Gray is funny in that the hue actually changes with intensity, whereas for colors the hue stays constant and only saturation/value change.

[willschlitzer]

I've been unable to use any of the .grd files when they're saved to my .gmt/server/moon/moon_relief/ folder. I'm getting the error:

grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01d_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01d_g.grd

Is there anything else I need to do to test out using the grids?

Additionally, what's the next step for getting the staged files on the server? I'm assuming they ultimately end up in a moon folder in /export/gmtserver/gmt/data/server?

[maxrjones]

I've been unable to use any of the .grd files when they're saved to my .gmt/server/moon/moon_relief/ folder. I'm getting the error:

We've been testing out the grids using a different directory on the server, not sure about local testing apart from providing the absolute file path to GMT. I'll share instructions in the next couple days about getting the staged files on the server.

[PaulWessel]

Thanks @maxrjones. I think we loaded stuff up on the /gmt/test side and used GMT_DATA_SERVER=test.generic-mapping-tools.org as source and our course it is updating the gmt_data_server.txt with the snippets taht got made during the tiling stage. I can help document that later if needed but I hope you and @Esteban82 might remember.

[Esteban82]

I can help document that later if needed but I hope you and @Esteban82 might remember.

Later I can share my own notes.

[Esteban82]

These are my notes. I am not sure how to test the data from the server (last item).

Enter the server

1. In the terminal type: ssh esteban82@gmtserver.SOEST.Hawaii.edu

2. Write password 12345678...

If the password is ok, then a message like this will appear: Last login: Tue Aug 16 16:19:32 2022 from 190.246.68.195

Check that you have the necessary permissions to be able to work with the remote data set

1. After you enter the server, type: groups Expected output: gmt (among others).

How to upload data to test folder (TEST)

1. Enter the server.

2. Upload data to the TEST folder (for example earth free anomaly)

user=esteban82
local=/home/federico/Github/GenericMappingTools/gmtserver-admin/staging/earth/earth_faa
server_TEST=/export/gmtserver/gmt/test/server/earth
scp -r $local $user@gmtserver.SOEST.Hawaii.edu:$server

3. Update file with information

scp /home/federico/Software/Github/gmtserver-admin/staging/earth_faa_server.txt esteban82@gmtserver.SOEST.Hawaii.edu:/export/gmtserver/gmt/test

scp /home/federico/Software/Github/gmtserver-admin/information/gmt_data_server.txt esteban82@gmtserver.SOEST.Hawaii.edu:/export/gmtserver/gmt/test

4. Test data from the server.

[willschlitzer]

It looks like I have the correct permissions. Should I copy the local staging/moon/ folder (and included files) to the server's test folder?

[Esteban82]

I would think so.

Also, I don't know if we should a create the moon folder in test server first (/export/gmtserver/gmt/test/server/moon)

[willschlitzer]

I believe all files have been copied to the appropriate folders per @Esteban82's guidance. Is there a way to point GMT at the test folder for remote downloads to test out the new grid files?

[Esteban82]

I think you should this script.

Test the new files (e.g., https://github.com/GenericMappingTools/remote-datasets/blob/main/scripts/remote_map_check.sh)

Source: https://github.com/GenericMappingTools/gmtserver-admin/issues/150

[Esteban82]

Is there a way to point GMT at the test folder for remote downloads to test out the new grid files?

Try this command: gmt set GMT_DATA_SERVER=test

And then this to check the server: gmt --show-dataserver

Desired output: http://test.generic-mapping-tools.org

[Esteban82]

I enter the test folder (http://test.generic-mapping-tools.org/). I think that the last file should be deleted.

[willschlitzer]

I enter the test folder (http://test.generic-mapping-tools.org/). I think that the last file should be deleted.

@Esteban82 Sounds good, I deleted moon_relief_server.txt. I was under the impression that it needed to be copied to the server from this line in your notes:

scp /home/federico/Software/Github/gmtserver-admin/staging/earth_faa_server.txt esteban82@gmtserver.SOEST.Hawaii.edu:/export/gmtserver/gmt/test

[PaulWessel]

The GMT C code only looks for gmt_data_server.txt which needs to have all those sections that the tiler builds I think. There is no looking for other planet files directly - it finds that by reading the gmt_data_server.txt file which lists what's on tap.

[Esteban82]

So, @willschlitzer do you manage to test the moon relief?

[willschlitzer]

I'm showing that I'm using the test server, but I'm getting this when plotting @moon_relief_01d_g.grd:

grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01d_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01d_g.grd

[Esteban82]

I got the same error. I fixed it by deleting the file gmt_data_server.txt from your pc.

[willschlitzer]

Same error after deleting gmt_data_server.txt. However, looking at the gmt_data_server.txt that was re-downloaded when I called grdimage, I'm not seeing any moon relief file information. Are you seeing moon relief data in the version you downloaded?

[Esteban82]

Maybe you should also delete the file gmt_hash_server.txt. The files are located in ther server dir in your user dir. You can find this last dir with gmt --show-userdir.

[Esteban82]

Are you seeing moon relief data in the version you downloaded?

Yes, I see the info and it works.

Type this to use test folder. gmt set GMT_DATA_SERVER=test

[willschlitzer]

It appears to have worked on my machine! What's the next step for validating this?

[willschlitzer]

@Esteban82 Have you tried any of the tiled moon resolution grids? I'm having no issues with 01d, 30m and 15m (all gridline registration), but getting an issue with remote files:

grdimage [NOTICE]: Remote data courtesy of GMT data server test [http://test.generic-mapping-tools.org/]
grdimage [NOTICE]: LOLA Moon Relief at 01x01 arc minutes reduced by Gaussian Cartesian filtering (0.6 km fullwidth) [Mazarico et al., 2013].
grdimage [NOTICE]:   -> Download grid file [257M]: moon_relief_01m_g/
grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01m_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01m_g.grd

[Esteban82]

Hi Will. No, I haven't try. Now I am on holydays so I won't be abble to test it until March.

[maxrjones]

@Esteban82 Have you tried any of the tiled moon resolution grids? I'm having no issues with 01d, 30m and 15m (all gridline registration), but getting an issue with remote files:

grdimage [NOTICE]: Remote data courtesy of GMT data server test [http://test.generic-mapping-tools.org/]
grdimage [NOTICE]: LOLA Moon Relief at 01x01 arc minutes reduced by Gaussian Cartesian filtering (0.6 km fullwidth) [Mazarico et al., 2013].
grdimage [NOTICE]:   -> Download grid file [257M]: moon_relief_01m_g/
grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01m_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01m_g.grd

Hi Will, unfortunately don't have time to check this out myself right now, but here are the steps I would take to debug:

1. Check if the file organization is the same for the moon data: http://test.generic-mapping-tools.org/server/moon/ and an earth_relief variety that does not error: https://oceania.generic-mapping-tools.org/server/earth/earth_relief/ (using GMT_DATA_SERVER=oceania for the earth_relief comparison since the test data for earth_relief is unstable).
2. Run with -Vd on an earth_relief_01m_g (using oceania server) that works and compare to the error message for the failing earth_moon file (using test server).

Beyond that, if it turns out to be a GMT bug rather than an issue with the remote files, it'd be necessary to dust off the GMT debugging setup.

[seisman]

I've tried @moon_relief_01d_g, @moon_relief_01d_p, @moon_relief_05m_p, @moon_relief_05m_g, and @moon_relief_01m_g. They all work for me.

[Esteban82]

So, I think we could approve this.