Closed willschlitzer closed 8 months ago
Great. Just a heads up: I am actually trying to do some of the legwork for you and will do a PR soon to add the moon recipe. I thought I at least run in through a few resolutions before adding it.
Thanks @PaulWessel! I'll hold off any work until your PR.
PR safely updated and seems to work well.Unless you are on a mac with lots of RAM it would be good to see how well the process runs for you. Expect many hours.
Currently running the script and it's converting the TIFF to a netCDF. You weren't joking about the RAM requirements; currently using 24 out of 32 GB. I'm on an Ubuntu 20.04 gaming desktop with an 8th gen i7, so I'm hoping it won't take too long. I did run into an issue the first time I ran the script; it stopped downloading after 4.6 GB and I received curl: (18) transfer closed with 3917136203 bytes remaining to read
, so I'm not sure if that was a connection issue or something else. I deleted the downloaded portion and it seems to have completed the download the second time around!
Interesting with curl doing that. Probably a glitch in the matrix somewhere as we say. If the downsampler step works well you will find the folder moon/moon_relief inside the staging folder and it will have all the grids at various resolutions. If you then run the tiler script then all the 5 min and higher resolutions will be split into tiles so there will then be directories with JP2 tiles inside that folder while the original global grids will be moved up to stain/tiled as safekeeping (we do not use those again or store them on the server).
I just received this error and the process quit. Here is what I'm seeing on my terminal:
srv_downsampler_grid.sh: Must download original source https://planetarymaps.usgs.gov/mosaic/Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.tif
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8100M 100 8100M 0 0 28.7M 0 0:04:41 0:04:41 --:--:-- 28.1M
srv_downsampler_grid.sh: Must convert original tif source to Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc
scripts/srv_downsampler_grid.sh: line 117: [: NaN: integer expression expected
Convert Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_14s_p.grd=ns+s0.5+o0
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 21766 Killed gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}
It looks like the download went fine but it ran into an issue when trying to run grdfilter
. It looks like my home environment still has GMT 6.0.0, so I'm trying it with my PyGMT development environment that has GMT 6.4.0.
Same issue with GMT 6.4:
moon/moon_relief/moon_relief_14s_p.grd exists - skipping
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 22132 Killed gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}
Any idea what the culprit may be? Here is my environment info:
GMT library information:
binary version: 6.4.0
cores: 12
grid layout: rows
library path: /home/will/miniconda3/envs/pygmt-dev/lib/libgmt.so
padding: 2
plugin dir: /home/will/miniconda3/envs/pygmt-dev/lib/gmt/plugins
share dir: /home/will/miniconda3/envs/pygmt-dev/share/gmt
version: 6.4.0
OK, I think you will actually need to run from GMT master - I do not know how to do that within conda since I am from the olden days. But I have made GMT source changes the last 24 hours in responses to issues with the moon processing. So master is a requirement for now.
The problem with big files is that it's not enough to have enough RAM. There needs to exist a contiguous chunk of memory to hold the entire array. Maybe if you reboot and start from a cleanest as possible state it may make a difference ... or not.
True, @joa-quim but also a must to use master since I fixed some issues. They may be macOS-specific though - hard to tell. The crash is in the section where I had to run grdfilter separately on the N and S hemispheres otherwise I got strange horizontal lines that @Esteban82 reported. Doing it this way (until I can find some subtle trouble) works - at least for me - and is only done on the 30s or finer resolutions. This and other issues are most likely some subtle 64-bit counting issue but hard to debug.
Installed gmt from master (showing 6.5.0_e43114b_2023.01.04
) and rebooted; still got the same result. It also used all of my RAM (32 GB) for about a minute prior to crashing.
Down-filter Lunar_LRO_LOLA_Global_LDEM_118m_Mar2014.nc to moon/moon_relief/moon_relief_15s_g.grd=ns+s0.5+o0
scripts/srv_downsampler_grid.sh: line 167: 6018 Killed gmt grdfilter -R-180/180/-90/0 ${SRC_FILE} -Fg${FILTER_WIDTH} -D${FMODE} -I${RES}${UNIT} -r${REG} -G${TMP}/s.grd --PROJ_ELLIPSOID=${DST_SPHERE}
Unfortunately, I don't have any Apple hardware to try this out with. I'm not sure what troubleshooting to try next.
Thanks for trying. What I guess you can try is to run the script with full verbose (bash -xv .....) so that you can see that command with all the actual arguments. Then copy paste and verify it still crashes (it should) but add -Vd so we might learn when it crashes. Then, just mess with -I and go from 15s to 30s to 01m and see if it stops crashing. I would think so as this seems to be related to trouble with 64-bit counting and not so much lack of memory since you have 32 Gb.
THe 14 arc second grid as 4-byte float internally is about 16 Gb, and then it is the output grid at 15s which is almost 14 Gb. So just those two allocations push 30 Gb of memory. For "fun" I am trying this comment (which does not produce the right answer hence the S and N workaround:
gmt grdfilter moon_relief_14s_p.grd -Fg0.18 -I15s -rp -D1 -Gshit.nc=ns -V
and activity monitor says this process is using 45.68 Gb! I am not sure what the extra 15 Gb but maybe a weight grid for something. I will have to debug to understand that 3rd blob.
Third blob is the area weight: Each node on a sphere controls less and less area going towards the poles so it is used in the convolution weighting. Obviously this could be rewritten as just a few columns instead of the entire grid since stuff is constant along rows except at the ends (If not 360).
Note that this grdfilter improvement reduces total memory by about what the input grid needs. So instead of needing 32 Gb for a 16Gb input grid it is now just a hair over 16 Gb. So perhaps @willschlitzer's crashes is related to needing over 40 Gb to do the job.
The grdfilter
update appears to have worked; my computer made it through moon_relief_15s_g
and went on to moon_relief_15s_p
seemingly without issue (although it did get up to 29.1 GB of RAM). I'll plan on letting the script run entirely later this afternoon or overnight.
@PaulWessel The srv_downsampler_grid.sh
script appears to have worked on my Ubuntu machine, currently running srv_tiler.sh
. Assuming it works, what else needs to be done, other than setting the default CPT?
Get, I think that is it. As for the other data CPTs we have a mix of using GMT master FPTs (e.g, geo for DEM) but earth_faa.cpt for free-air anomalies. So for the moon I think we should create a moon_relief.cpt file that has a reasonable range to match the actual data range (Not exact but slightly rounded, .e.g., -9200/10800 for instance, and we add that to the cache system and with recipe saying @moon_relief.cpt we are OK. I did set that # DST_CPT=@moon_relief.cpt before you ran your job I hink, depending on if you updated. Once the tiler finishes (of perhaps it is just after the downsample finished) you should find a file called moon_relief_server.txt in the staging directory. It has all the meta data about each resolugion+registration and should list @moon_relief if so.
So I made moon_relief.cpt simply via gmt makecpt -Cgray -T-9200/10800 > moon_relief.cpt
and it looks like this. OK?
@PaulWessel The
srv_downsampler_grid.sh
script appears to have worked on my Ubuntu machine, currently runningsrv_tiler.sh
. Assuming it works, what else needs to be done, other than setting the default CPT?
The remaining parts are not quite automated and some requires ability to authentically in on the SOEST server. Maybe @maxrjones and @Esteban82 can remind me to what extent they were able to do uploads and work on oceania.soest.hawaii.edu? We had hoped that instead of using a computer in Hawaii, especially with me in Norway, we would have made a deal with UNAVCO or IRIS on hosting our data but they are merging and we are small fries in their workload so I have not been able to make that request yet. Meantime we are using my SOEST server. Because logging into the UH/SOEST network is needed and because they do not really want some "community account" like gmtadmin that we could all share around, I am not sure what the best next step is. You can learn most of what we do from README and MAINTENANCE but I am sure it is not complete.
But at some point we will need to transfer the 6+ Gb of tiles and files to oceania, update the test gmt_data_server.txt file, and then set GMT_DATA_SERVER to test.generic-mapping.tools.org and kick the tires, i.e., make a few maps to see if things actually work of if we forgot something important....
Get, I think that is it. As for the other data CPTs we have a mix of using GMT master FPTs (e.g, geo for DEM) but earth_faa.cpt for free-air anomalies. So for the moon I think we should create a moon_relief.cpt file that has a reasonable range to match the actual data range (Not exact but slightly rounded, .e.g., -9200/10800 for instance, and we add that to the cache system and with recipe saying @moon_relief.cpt we are OK. I did set that # DST_CPT=@moon_relief.cpt before you ran your job I hink, depending on if you updated.
Sounds good about the new CPT. I'm still showing the default CPT as geo
, I'll run git pull
once the tiler script finishes.
Once the tiler finishes (of perhaps it is just after the downsample finished) you should find a file called moon_relief_server.txt in the staging directory. It has all the meta data about each resolugion+registration and should list @moon_relief if so.
It looks like moon_relief_server.txt
was created as soon as the tiler script began running, and metadata is being added as the grids are being created (currently on the 15s grids).
@PaulWessel The
srv_downsampler_grid.sh
script appears to have worked on my Ubuntu machine, currently runningsrv_tiler.sh
. Assuming it works, what else needs to be done, other than setting the default CPT?The remaining parts are not quite automated and some requires ability to authentically in on the SOEST server. Maybe @maxrjones and @Esteban82 can remind me to what extent they were able to do uploads and work on oceania.soest.hawaii.edu? We had hoped that instead of using a computer in Hawaii, especially with me in Norway, we would have made a deal with UNAVCO or IRIS on hosting our data but they are merging and we are small fries in their workload so I have not been able to make that request yet. Meantime we are using my SOEST server. Because logging into the UH/SOEST network is needed and because they do not really want some "community account" like gmtadmin that we could all share around, I am not sure what the best next step is. You can learn most of what we do from README and MAINTENANCE but I am sure it is not complete.
But at some point we will need to transfer the 6+ Gb of tiles and files to oceania, update the test gmt_data_server.txt file, and then set GMT_DATA_SERVER to test.generic-mapping.tools.org and kick the tires, i.e., make a few maps to see if things actually work of if we forgot something important....
I'll work on creating some plots, since it seems like I won't be able to do much with the hosting server. If I move the moon
folder from staging
to the .gmt/server
folder and copy the contents of moon_relief_server.txt
to gmt_data_server.txt
, will I be able to call the grids with the standard remote file syntax (e.g. @moon_relief_30m_g
) when using GMT/PyGMT?
That moon_relief.cpt
is too dark. Maybe set minimum at ~100
Yes that might work, but remember GMT may check the server every day and decide to override your files... I would do this:
That
moon_relief.cpt
is too dark. Maybe set minimum at ~100
Not sure what is best here. That gives 100-255 as the gray range and with any intensity shade we have basically less visible range. Gray is funny in that the hue actually changes with intensity, whereas for colors the hue stays constant and only saturation/value change.
I've been unable to use any of the .grd files when they're saved to my .gmt/server/moon/moon_relief/
folder. I'm getting the error:
grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01d_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01d_g.grd
Is there anything else I need to do to test out using the grids?
Additionally, what's the next step for getting the staged files on the server? I'm assuming they ultimately end up in a moon
folder in /export/gmtserver/gmt/data/server
?
I've been unable to use any of the .grd files when they're saved to my .gmt/server/moon/moon_relief/ folder. I'm getting the error:
We've been testing out the grids using a different directory on the server, not sure about local testing apart from providing the absolute file path to GMT. I'll share instructions in the next couple days about getting the staged files on the server.
Thanks @maxrjones. I think we loaded stuff up on the /gmt/test side and used GMT_DATA_SERVER=test.generic-mapping-tools.org as source and our course it is updating the gmt_data_server.txt with the snippets taht got made during the tiling stage. I can help document that later if needed but I hope you and @Esteban82 might remember.
I can help document that later if needed but I hope you and @Esteban82 might remember.
Later I can share my own notes.
These are my notes. I am not sure how to test the data from the server (last item).
In the terminal type:
ssh esteban82@gmtserver.SOEST.Hawaii.edu
Write password
12345678...
If the password is ok, then a message like this will appear: Last login: Tue Aug 16 16:19:32 2022 from 190.246.68.195
groups
Expected output: gmt
(among others).Enter the server.
Upload data to the TEST folder (for example earth free anomaly)
user=esteban82
local=/home/federico/Github/GenericMappingTools/gmtserver-admin/staging/earth/earth_faa
server_TEST=/export/gmtserver/gmt/test/server/earth
scp -r $local $user@gmtserver.SOEST.Hawaii.edu:$server
scp /home/federico/Software/Github/gmtserver-admin/staging/earth_faa_server.txt esteban82@gmtserver.SOEST.Hawaii.edu:/export/gmtserver/gmt/test
scp /home/federico/Software/Github/gmtserver-admin/information/gmt_data_server.txt esteban82@gmtserver.SOEST.Hawaii.edu:/export/gmtserver/gmt/test
It looks like I have the correct permissions. Should I copy the local staging/moon/
folder (and included files) to the server's test folder?
I would think so.
Also, I don't know if we should a create the moon
folder in test server first (/export/gmtserver/gmt/test/server/moon
)
I believe all files have been copied to the appropriate folders per @Esteban82's guidance. Is there a way to point GMT at the test folder for remote downloads to test out the new grid files?
I think you should this script.
Test the new files (e.g., https://github.com/GenericMappingTools/remote-datasets/blob/main/scripts/remote_map_check.sh)
Source: https://github.com/GenericMappingTools/gmtserver-admin/issues/150
Is there a way to point GMT at the test folder for remote downloads to test out the new grid files?
Try this command:
gmt set GMT_DATA_SERVER=test
And then this to check the server:
gmt --show-dataserver
Desired output: http://test.generic-mapping-tools.org
I enter the test folder (http://test.generic-mapping-tools.org/). I think that the last file should be deleted.
I enter the test folder (http://test.generic-mapping-tools.org/). I think that the last file should be deleted.
@Esteban82 Sounds good, I deleted moon_relief_server.txt
. I was under the impression that it needed to be copied to the server from this line in your notes:
scp /home/federico/Software/Github/gmtserver-admin/staging/earth_faa_server.txt esteban82@gmtserver.SOEST.Hawaii.edu:/export/gmtserver/gmt/test
The GMT C code only looks for gmt_data_server.txt which needs to have all those sections that the tiler builds I think. There is no looking for other planet files directly - it finds that by reading the gmt_data_server.txt file which lists what's on tap.
So, @willschlitzer do you manage to test the moon relief?
I'm showing that I'm using the test server, but I'm getting this when plotting @moon_relief_01d_g.grd
:
grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01d_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01d_g.grd
I got the same error. I fixed it by deleting the file gmt_data_server.txt
from your pc.
Same error after deleting gmt_data_server.txt
. However, looking at the gmt_data_server.txt
that was re-downloaded when I called grdimage
, I'm not seeing any moon relief file information. Are you seeing moon relief data in the version you downloaded?
Maybe you should also delete the file gmt_hash_server.txt
. The files are located in ther server dir in your user dir. You can find this last dir with gmt --show-userdir
.
Are you seeing moon relief data in the version you downloaded?
Yes, I see the info and it works.
Type this to use test folder.
gmt set GMT_DATA_SERVER=test
It appears to have worked on my machine! What's the next step for validating this?
@Esteban82 Have you tried any of the tiled moon resolution grids? I'm having no issues with 01d
, 30m
and 15m
(all gridline registration), but getting an issue with remote files:
grdimage [NOTICE]: Remote data courtesy of GMT data server test [http://test.generic-mapping-tools.org/]
grdimage [NOTICE]: LOLA Moon Relief at 01x01 arc minutes reduced by Gaussian Cartesian filtering (0.6 km fullwidth) [Mazarico et al., 2013].
grdimage [NOTICE]: -> Download grid file [257M]: moon_relief_01m_g/
grdimage [ERROR]: Libcurl Error: HTTP response code said error
grdimage [ERROR]: Probably means @moon_relief_01m_g.grd does not exist on the remote server
grdimage [ERROR]: Unable to obtain remote file @moon_relief_01m_g.grd
Hi Will. No, I haven't try. Now I am on holydays so I won't be abble to test it until March.
@Esteban82 Have you tried any of the tiled moon resolution grids? I'm having no issues with
01d
,30m
and15m
(all gridline registration), but getting an issue with remote files:grdimage [NOTICE]: Remote data courtesy of GMT data server test [http://test.generic-mapping-tools.org/] grdimage [NOTICE]: LOLA Moon Relief at 01x01 arc minutes reduced by Gaussian Cartesian filtering (0.6 km fullwidth) [Mazarico et al., 2013]. grdimage [NOTICE]: -> Download grid file [257M]: moon_relief_01m_g/ grdimage [ERROR]: Libcurl Error: HTTP response code said error grdimage [ERROR]: Probably means @moon_relief_01m_g.grd does not exist on the remote server grdimage [ERROR]: Unable to obtain remote file @moon_relief_01m_g.grd
Hi Will, unfortunately don't have time to check this out myself right now, but here are the steps I would take to debug:
Beyond that, if it turns out to be a GMT bug rather than an issue with the remote files, it'd be necessary to dust off the GMT debugging setup.
I've tried @moon_relief_01d_g
, @moon_relief_01d_p
, @moon_relief_05m_p
, @moon_relief_05m_g
, and @moon_relief_01m_g
. They all work for me.
So, I think we could approve this.
Per a discussion at the forum, I'm hoping to add a lunar topographic relief grid to the GMT remote server.