Update to earth_relief v2.5.5 in main server.

Esteban82 commented 1 year ago

I will update the earth_relief grids (to version v2.5.5) in the main server.

The steps to follow are:

[x] Test the files in test.generic-mapping-tools.org (already there).
[x] The info in gmt_data_server.txt is already updated.
[x] Update the files in /export/gmtserver/gmt/data/server/earth/earth_relief/.
[x] Update the files in /export/gmtserver/gmt/data without tiling (for backwards compatibility with 6.1).
[x] in the remote-datasets repository, update the information for of the grid and in the changelog.
[x] Close this issue.

~~@PaulWessel should I update also the files in the main directory (/export/gmtserver/gmt/data) also?~~

Esteban82 commented 1 year ago

1. Test the files in test.generic-mapping-tools.org

I made this test to avoid having the same issue that with synbath. It's ok.

gmt set GMT_DATA_SERVER test
gmt begin earth_relief png
gmt grdimage @earth_relief_15m
gmt grdimage @earth_relief_01d -Yh
gmt end

earth_relief

Esteban82 commented 1 year ago

I am a bit lost. @PaulWessel could I proceed with this?

PaulWessel commented 1 year ago

Yes you can but may need to be careful and do by hand - I have not written makefiles yet as I have been plugging GMT bugs instead. I think the policy for updates is OK:

When an existing data set is updated (SRTM, Sandwell grav, etc) we can update the server after we have tested that it works.
When a new data set is released (e.g. Pluto) we wait to introduce all of them as a new GMT version gets out (5.6).

IT people at SOEST has not responded so another day goes by, but I want them to rename "test" to "candidate" and then any make or script action from us will only add files or update them in the candidate directory. When we feel a dataset or sets are ready we should do something like

make release moon

and that will

Build a shell script with commands to run on the server.
Copy the shell script to /tmp on the server.
Run that script locally via ssh from our ends.
Add this to a log file

If you want to do some other things then I have not gotten to these yet:

Remaining planets processing (mercury?)
Explore if grdcut could be used to create the image tiles needed etc., e.g., gmt grdcut @earth_day_30s_p -R0/15/15/30 -Gonetile.tif=gd ?
Explore if grdblend can stitch image times together

Esteban82 commented 1 year ago

Ok, I will try to take a look at this:

Remaining planets processing (mercury?) Explore if grdcut could be used to create the image tiles needed etc., e.g., gmt grdcut @earth_day_30s_p -R0/15/15/30 -Gonetile.tif=gd ? Explore if grdblend can stitch image times together

Esteban82 commented 1 year ago

BTW, I am thinking of moving the current earth_relief files (v 2.4) to the test directory (as a backup). I will create the directory earth_relief_v2.4. Unless you have a better suggestion.

Esteban82 commented 1 year ago

@PaulWessel I notice that within earth_relief there are the grids for 03s and 01s (of SRTM). Wouldn't be better if that data is in another directory?

PaulWessel commented 1 year ago

Maybe, but I put them there and they are shared by the other DEMs (gebco, synth, etc.). Sure, I could create a strum directory but remember people running 6.3 also want their workflows to work and unless we duplicate that huge set of files we will break things. So for now, in earth_relief and the code handles how to get there.

Esteban82 commented 1 year ago

Oh, yes, backward compatibilty. So, I will be extra carefull when I update that folder.

Esteban82 commented 1 year ago

I have just updated the files and test them.

earth_relief

gmt set GMT_DATA_SERVER oceania
gmt begin earth_relief png
gmt grdimage @earth_relief_15m
gmt grdimage @earth_relief_01d -Yh
gmt grdimage @earth_relief_03s -Yh -RUY
gmt end

Esteban82 commented 1 year ago

@PaulWessel should I update also the files in the main directory (/export/gmtserver/gmt/data) also? @PaulWessel ??

PaulWessel commented 1 year ago

Hm, I thought you did? You show oceania above and oceania -> data. Hopefully tomorrow, candidate -> candidate and static -> static.

Esteban82 commented 1 year ago

I updated the files in the dir server/earth/earth_relief. But in the directory /export/gmtserver/gmt/data the files for higher resolutions (15s to 05m) are those of the SRTM15+ v2.1. I think I should replace them for the version without tiling.

-bash-4.2$ ls earth_* -l
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_01d.grd -> server/earth/earth_relief/earth_relief_01d_g.grd
-rw-rw-r-- 1 pwessel gmt  249043008 ene 26  2022 earth_relief_01m_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_01m.grd -> earth_relief_01m_g.grd
-rw-rw-r-- 1 pwessel gmt   67688618 ene 26  2022 earth_relief_02m_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_02m.grd -> earth_relief_02m_g.grd
-rw-rw-r-- 1 pwessel gmt   31214311 ene 26  2022 earth_relief_03m_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_03m.grd -> earth_relief_03m_g.grd
-rw-rw-r-- 1 pwessel gmt   18145185 ene 26  2022 earth_relief_04m_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_04m.grd -> earth_relief_04m_g.grd
-rw-rw-r-- 1 pwessel gmt   11887150 ene 26  2022 earth_relief_05m_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_05m.grd -> earth_relief_05m_g.grd
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_06m.grd -> server/earth/earth_relief/earth_relief_06m_g.grd
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_10m.grd -> server/earth/earth_relief/earth_relief_10m_g.grd
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_15m.grd -> server/earth/earth_relief/earth_relief_15m_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_15s.grd -> earth_relief_15s_p.grd
-rw-rw-r-- 1 pwessel gmt 3350851350 ene 26  2022 earth_relief_15s_p.grd
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_20m.grd -> server/earth/earth_relief/earth_relief_20m_g.grd
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_30m.grd -> server/earth/earth_relief/earth_relief_30m_g.grd
-rw-rw-r-- 1 pwessel gmt  905677906 ene 26  2022 earth_relief_30s_g.grd
lrwxrwxrwx 1 pwessel gmt         22 ene 26  2022 earth_relief_30s.grd -> earth_relief_30s_g.grd
lrwxrwxrwx 1 pwessel gmt         48 ene 26  2022 earth_relief_60m.grd -> server/earth/earth_relief/earth_relief_01d_g.grd
-bash-4.2$ pwd
/export/gmtserver/gmt/data
-bash-4.2$ gmt grdinfo earth_relief_05m.grd 
earth_relief_05m.grd: Title: Earth Relief at 05 arc minutes
earth_relief_05m.grd: Command: grdfilter SRTM15+V2.1.nc -Fg9.3 -D1 -I05m -rg -Gearth/earth_relief/earth_relief_05m_g.grd=ns+s0.5+o0 --IO_NC4_DEFLATION_LEVEL=9 --IO_NC4_CHUNK_SIZE=2048 --PROJ_ELLIPSOID=Sphere
earth_relief_05m.grd: Remark: Obtained by Gaussian Cartesian filtering (9.3 km fullwidth) from SRTM15+V2.1.nc [Tozer et al., 2019; http://dx.doi.org/10.1029/2019EA000658]
earth_relief_05m.grd: Gridline node registration used [Geographic grid]
earth_relief_05m.grd: Grid file format: ns = GMT netCDF format (16-bit integer), CF-1.7
earth_relief_05m.grd: x_min: -180 x_max: 180 x_inc: 0.0833333333333 (5 min) name: longitude n_columns: 4321
earth_relief_05m.grd: y_min: -90 y_max: 90 y_inc: 0.0833333333333 (5 min) name: latitude n_rows: 2161
earth_relief_05m.grd: v_min: -10718.5 v_max: 7105.5 name: elevation (m)
earth_relief_05m.grd: scale_factor: 0.5 add_offset: 0 packed z-range: [-21437,14211]
earth_relief_05m.grd: format: netCDF-4 chunk_size: 2048,2048 shuffle: on deflation_level: 9

PaulWessel commented 1 year ago

Those are there for backwards compatibility with 6.1 I think. But no harm in updating them.

anbj commented 1 year ago

Maybe, but I put them there and they are shared by the other DEMs (gebco, synth, etc.). Sure, I could create a strum directory but remember people running 6.3 also want their workflows to work and unless we duplicate that huge set of files we will break things. So for now, in earth_relief and the code handles how to get there.

Can't you move the data, and then make a symlink to have backward compatibility?

Esteban82 commented 1 year ago

The files needed to be updated are (for backward compatibility):

-bash-4.2$ ls -lh *.grd | grep -v ^l
-rw-rw-r-- 1 pwessel gmt 238M ene 26  2022 earth_relief_01m_g.grd
-rw-rw-r-- 1 pwessel gmt  65M ene 26  2022 earth_relief_02m_g.grd
-rw-rw-r-- 1 pwessel gmt  30M ene 26  2022 earth_relief_03m_g.grd
-rw-rw-r-- 1 pwessel gmt  18M ene 26  2022 earth_relief_04m_g.grd
-rw-rw-r-- 1 pwessel gmt  12M ene 26  2022 earth_relief_05m_g.grd
-rw-rw-r-- 1 pwessel gmt 3,2G ene 26  2022 earth_relief_15s_p.grd
-rw-rw-r-- 1 pwessel gmt 864M ene 26  2022 earth_relief_30s_g.grd

My doubt is how could I test it. Should I install GMT 6.1? Is the only way to do it?

Also, I think we could copy those files (without tiling) into within server/earth/earth_relief/ and create some symlinks. So, it would be easier to update. I could try to do it.

PaulWessel commented 1 year ago

I think this was solved earlier. See scripts/srv_setbackwards_links.sh and see if it is still OK.

As for updating from candidate to data, your git pull and try

make server-release

You will be asked if you want to update a specific data set (e.g., _earthsynbath) or hit return for all.Try either (nothing will be running anywhere yet!). That command creates a /tmp/release.sh script and it will be copied to /tmp in the server. The command that would run it is commented out for now.

Please examine the/tmp/release.sh for the two cases and see if you can find any errors. Lot of the commands can be cut/pasted and run without corrupting the oceania server.

Sorry, had appointment earlier and now engaged in making dinner! Let me know if you any bugs in the script and what it creates.

Esteban82 commented 1 year ago

I think this was solved earlier. See scripts/srv_setbackwards_links.sh and see if it is still OK.

Ok, I will take a look. It is for the symlinks only, right? So, I have to manually update those grids in the top dir.

PaulWessel commented 1 year ago

Think that is true, so do the links at the end

Esteban82 commented 1 year ago

I try make server-release make got an issue. It thinks that my name is thor which is not my username for the server (esteban82). I can change it but I wonder if it would be better that the scripts asks for the hostname. Or maybe it is an issue that happened only to me because I don't know very well how to use linux.

make server-release
scripts/server-release.sh
server-release.sh: Enter a specific data set (e.g., mars_relief) or hit return for all data on the candidate: earth-synbath
server-release.sh: Copying earth-synbath from candidate server to oceania
server-release.sh: scp /tmp/release.sh candidate.generic-mapping-tools.org:/tmp
thor@candidate.generic-mapping-tools.org's password:

PaulWessel commented 1 year ago

Interesting, OK we will have to abstract the username then (I am pwessel everywhere so did not think of that). I'll come up with a solution.

PaulWessel commented 1 year ago

OK, try this:

Update from git
export GMT_USER=thor
Run make again. It should now do thor@ etc.

Run it first before you set the parameter and then it should say esteban@

Esteban82 commented 1 year ago

@PaulWessel I think I don't have permission to write release.sh.

After the commit I got this:

server-release.sh: scp /tmp/release.sh esteban82@candidate.generic-mapping-tools.org:/tmp
esteban82@candidate.generic-mapping-tools.org's password: 
scp: dest open "/tmp/release.sh": Permission denied
scp: failed to upload file /tmp/release.sh to /tmp
server-release.sh: ssh esteban82@candidate.generic-mapping-tools.org /tmp/release.sh

And I check in the server and I saw:

-bash-4.2$ pwd
/tmp
-bash-4.2$ ls release.sh -l
-rwx--x--x 1 pwessel gmt 1323 ago 28 05:32 release.sh

PaulWessel commented 1 year ago

Right. The script will destroy itself but of course only if it is run. I have removed it so you can at least try - but then remember to delete it so others can try too!

Esteban82 commented 1 year ago

I think that now ir running well.

-bash-4.2$ cat release.sh 
#!/usr/bin/env bash
# Script made by "make server-release" to be run on the gmtserver
#
# 1. Issue rsync command
rsync -al /export/gmtserver/gmt/candidate/server/all /export/gmtserver/gmt/data/server/all
# 2. Rebuild the gmt_data_server.txt file
# 2a. Make sed script that changes THEDATE to today's dat
date "+%Y-%m-%d" | awk '{printf "s/THEDATE/%s/g\n", 2826791}' > /tmp/sed.txt
# 2b. Find all the dataset server files
find /export/gmtserver/gmt/data/server -name '*_*_server.txt' > /tmp/datasets.lis
# 2c. Count the number of data files or directory entries and start first line of /tmp/gmt_data_server.txt:
cat $(cat /tmp/datasets.lis) | grep -v '^#' | wc -l | awk '{printf "%d\n", }' > /tmp/gmt_data_server.txt
# 2d. Append the header information section after piping via sed to get the date
cat /export/gmtserver/gmt/gmt_data_server_header.txt | sed -f /tmp/sed.txt >> /tmp/gmt_data_server.txt
# 2e. Append all the data set files to the same file
cat $(cat /tmp/datasets.lis) >> /tmp/gmt_data_server.txt
# 2f. Copy the old server file to the backup file
cp -f /export/gmtserver/gmt/data/gmt_data_server.txt /export/gmtserver/gmt/data/gmt_data_server_previous.txt
# 2g. Place the new server file
cp -f /tmp/gmt_data_server.txt /export/gmtserver/gmt/data
# 3. Clean up
rm -f tmp/sed.txt /tmp/gmt_data_server.txt /tmp/release.sh

PaulWessel commented 1 year ago

You could login to the server, edit the release script by commenting out 2g and 3, then run the script and see if it produces a nice gmt_data_server.txt in /tmp (but not copying it anywhere yet)

Esteban82 commented 1 year ago

I just updated the files in the top dir.

Esteban82 commented 1 year ago

I am closing this.

Maybe the info about
export GMT_USER=username for make server-release should be copy somehwere else.

GenericMappingTools / gmtserver-admin

Update to earth_relief v2.5.5 in main server. #214

1. Test the files in test.generic-mapping-tools.org