search

GenericMappingTools / gmtserver-admin

Cache data and script for managing the GMT data server
GNU Lesser General Public License v3.0
7 stars 3 forks source link

Rewrite scripts and makefile to simplify operations #221

Closed PaulWessel closed 10 months ago

PaulWessel commented 10 months ago

I have done lots of work on the management of remote data, with help from @federico. Please update to this branch and type make help in the top directory. You will there see some help on what to do, plus a few high-level targets. These things can still change so best that you examine rather than try the various targets until I give the word. I have run through steps for earth_synbath and venus_relief so I am pretty sure it all works well.

We now have actually four URL addresses for the remote data set work:

oceania.generic-mapping-tools.org [The public one for users] candidate.generic-mapping-tools.org [The next release server for us to get ready and test setup] static.generic-mapping-tools.org [Intended to be run by the est scripts to avoid map failures as data changes] test.generic-mapping-tools.org [The previous server for testing - I think may use it when it doesn't matter if we mess with that directory]

We got a new (well 2012) grid from Sandwell where they had filled in and fixed some glitches in the original grid. It had a funny 2.4 x 1.8 arc minute spacing so I decided to resample it finer to a square 1x1 arc minutes from which we downsample.

If you have any comments on these changes or have suggestions for other targets etc let me know. Oh, someone with rsync experience should double-check scripts/server-release.sh since I am not running that until later.

Esteban82 commented 10 months ago

but data goes as low as 01s

Yes, I am also the 03s. These are the SRTM data that are added in the file at the end. It is easy to edit the script so those records will be at the beggining. BTW, it would be good if order of the resolution is the same than in the remote data set page.

Esteban82 commented 10 months ago

test.generic-mapping-tools.org [The previous server for testing - I think may use it when it doesn't matter if we mess with that directory]

Maybe could be use as a backup for older grid version? At least for some day/weeks.

Esteban82 commented 10 months ago

someone with rsync experience should double-check scripts/server-release.sh since I am not running that until later.

I have a doubt. What would happen with the earth_relief grids that are in the top dir for backward compatibility (link). Will be deleted?

anbj commented 10 months ago

I have a doubt. What would happen with the earth_relief grids that are in the top dir for backward compatibility (link). Will be deleted?

If you use --delete you'll delete files that are not in the src-directory (for 1:1 copy/mirror). So this is a bit dangerous.

PaulWessel commented 10 months ago

I have a doubt. What would happen with the earth_relief grids that are in the top dir for backward compatibility (link). Will be deleted?

If you use --delete you'll delete files that are not in the src-directory (for 1:1 copy/mirror). So this is a bit dangerous.

Yes, true. This is why have not tried to use these commands yet. --delete is now removed.

Esteban82 commented 10 months ago

@PaulWessel I made a commit because I was getting this:

server-release.sh: scp /tmp/release.sh esteban82@candidate.generic-mapping-tools.org:/tmp
thor@candidate.generic-mapping-tools.org's password:
PaulWessel commented 10 months ago

Think we are close to wrapping this up. Perhaps we can approve this, merge, and then deal with further changes separately. This PR is getting a bit large.

PaulWessel commented 10 months ago

Is there any other task for which we do not have a script (or via the makefile)? The cache is updated vi crontab if there are changes, but I did not want to put these huge datasets on some auto-pilot. I think it is good that somebody goes through and make sure each step passes, especially after all my changes. Even if, say, pluto might be on the candidate server, I thin we should start from scratch and do all the steps we would do for a new dataset, up to the point of placing on candidate and rebuild gmt_data_server.txt on the candidate.

Esteban82 commented 10 months ago

Is there any other task for which we do not have a script (or via the makefile)?

Yes, I think we could make a script for the earth_relief grids without tiling for legacy purposes (the files listed here).

For those grids I was thinking on making an specific folder something like earth_relief_legacy. We could make an script that make this (this would be a third script for make earth_relief).

We could place that directory in:

  1. /export/gmtserver/gmt/data/server/earth
  2. /export/gmtserver/gmt/test/server/earth/earth_relief

In the top dir we will have to replace the grids for symlinks but I think that this have to be done only once (so there is no need for a script).

Esteban82 commented 10 months ago

I think it is good that somebody goes through and make sure each step passes, especially after all my changes.

I will try to do it.

joa-quim commented 10 months ago

Don't make copies of those huge datasets. Having earth_relief and earth_gebco is already an exaggeration. And the gmt/data/server/earth has earth_relief and earth_relief2.1, what for? Remember that mirrors copy both of these dirs.

Esteban82 commented 10 months ago

Don't make copies of those huge datasets.

My suggestion is just to update the grids that already exist that are located in the top dir.

BTW, I think that earth_relief2.1 will be deleted.