Update the synbath grid (to version 2) in the main server

[Esteban82]

I will update the synbath grids (to version 2) in the main server.

The steps to follow are:

• [x] Test the files (already in the test server).
• [x] The info in gmt_data_server.txt is already updated.
• [x] Copy and place the updated earth_synbath data from the test server to the main server.
• [x] in the remote-datasets repository, update the information for of the grid and in the changelog.
• [x] Close this issue.

[Esteban82]

I just update the main server

[seisman]

Likely something wrong with the synbath data:

gmt grdcut -R-95/-94/-1.5/-1 @earth_synbath_30s_p -Gtemp.nc
gmt grdinfo temp.nc -C

It gives:

temp.nc -95 -94 -1.5    -1  0   0   0.00833333333333    0.00833333333333    120 60  1   1

as you can see, the min/max values of the dataset are 0.

[seisman]

gmt grdimage @earth_synbath_01d -png map gives:

[PaulWessel]

Can confirm. Checking with @Esteban82

[Esteban82]

I can confirm the issue

[Esteban82]

I try with the files that I upload to the test server and I got the same result. I also check another grids (earth_relief, gebco) and there was no such problem. So I think that I there was a problem when the data was proccessed (i.e. when I run make earth-synbath). I could try to do it again tomorrow to see if I got the same result.

[PaulWessel]

I'll see too

[Esteban82]

Hi @PaulWessel.

I ran the script srv_downsampler_grid.sh again and I get the same wrong grids. I didn't get any error message. I think that the problem is generated with:

gmt grdconvert ${SRC_FILE} ${DST_FILE}=${DST_MODIFY} --IO_NC4_DEFLATION_LEVEL=9

I tried the 15s_p.grd that is created and it already wrong. The original grid it is ok.

I tried with GMT 6.4 on windows. Maybe there is an issue that was already fixed (but in this case I don't know why it didn't happend with the other grids).

Could you try to run it?

[seisman]

@Esteban82 @PaulWessel Any updates on this problematic dataset?

[Esteban82]

Nop. I don't know what the problem is. Instead I could process the previous version. Or try to process the grid whit a 32-bit float format (=nf, so without compression).

[seisman]

Nop. I don't know what the problem is. Instead I could process the previous version.

It would be great to revert back to the previous version if we can't fix the bug in the new version in one or two days. People may be using the synbath dataset and are surprised that it suddenly doesn't work.

[Esteban82]

@PaulWessel what do you think?

[PaulWessel]

Working on it, just finished tiling on my machine. No problems so far. Perhaps memory limit.

[Esteban82]

Great!

[PaulWessel]

Need to go shopping soon so hopefully later today synth will be done. Might give me a chance to write that make/script combo.

[PaulWessel]

git pull the latest and look at the end of the Makefile and the new scripts/place-candidate.sh script. If you have (or just make) staging/eart/earth_synbath then you can (in the top dir) run

make place-synbath

and it should go to SOEST and just list the files there (note I have basically commented out the read commands and instead echo them out until I do some more testing.)

[Esteban82]

git pull the latest and look at the end of the Makefile and the new scripts/place-candidate.sh script.

I don't see any script with that name.

[PaulWessel]

Sorry, in branch placing

[Esteban82]

git pull the latest and look at the end of the Makefile and the new scripts/place-candidate.sh script. If you have (or just make) staging/eart/earth_synbath then you can (in the top dir) run

make place-synbath

and it should go to SOEST and just list the files there (note I have basically commented out the read commands and instead echo them out until I do some more testing.)

I understand that this script is to copy the data from my pc to the server. Is it true?

It is good. But right now the problem I have is that not the synbath grids that process are wrong. Only with that grid it happened to me. So I suspect that something weird is going on.

@PaulWessel Do you try processing the synbath grid?

[PaulWessel]

FYI, almost done with processing and uploading synth. Stay tuned.

[PaulWessel]

The place script will also place the planet_dataset_server.txt section in the planet/dataset directory. My plan is then to rebuild the _gmt_dataserver.txt from all the sections whenever we place, by the script to avoid errors. So for instance, with _earthsynbath I just tested this:

1. scripts/srv_downsampler_grid.sh earth_synbath
2. scripts/srv_tiles.sh earth_synbath
3. make place-sunbath

After these three steps one should be able to use GMT_DATA_SERVER=candidate and test the tiling etc.

One inconvenience right now is we get prompted three times for password.

[PaulWessel]

Does anyone (e.g., @seisman) know why this screws me:

If I run this on the command line:

scp -r staging/venus/venus_relief candidate.generic-mapping-tools.org:/export/gmtserver/gmt/candidate/server/venus

then the directory _venusrelief is placed in the venus directory on the candidate server as intended. However, if I issue that same command from a script run by a makefile (and I run the script with verbose to see the commands) the verbose shell writes

+ scp -r staging/venus/venus_relief candidate.generic-mapping-tools.org:/export/gmtserver/gmt/candidate/server/venus

to show what command is run. But, this places the contents of _venusrelief in the venus directory, i.e., the directory name itself is not created. I know that is possible the difference between copying venu_relief vs venus_relief/ (ends in slash) but I thought that was only needed when you want all the files.

[PaulWessel]

Curiously, if I add / at the end of the ssh command (destination) it works for both the make and the plain CLI.

[PaulWessel]

@Esteban82, since you have placed all the others (and I just updated venus with David's 2012 no gaps grid), the next step is to run a non-existent command that will build _gmt_dataserver.txt for the candidate server from all the individual text bits. The _srvtiler.sh now places the files like _venus_reliefserver.txt inside the _venusrelief tree so it will be easy to update the server file. Could you please

Copy all those individual planet_dataset_server.txt snippets into the corresponding planet_dataset subdirectories on the candidate server and I will work on the script. Once I have this working the steps will have one more entry:

4. make candidate-server

or something like that.

[Esteban82]

Hi @PaulWessel

Glad to help but I think that I don't have permissions to work on Venus dir.

-bash-4.2$ pwd
/export/gmtserver/gmt/candidate/server
-bash-4.2$ ls -l
total 0
drwxrwxr-x 18 pwessel gmt 313 ago 26 01:55 earth
drwxrwxr-x  3 pwessel gmt  25 ago 23 03:03 mars
drwxrwxr-x  3 pwessel gmt  28 ago 23 03:03 mercury
drwxr-xr-x  3 pwessel gmt  25 ago 23 03:02 moon
drwxrwxr-x  3 pwessel gmt  26 ago 23 03:03 pluto
drwx--x--x  3 pwessel gmt  26 ago 26 23:56 venus

[Esteban82]

FYI, almost done with processing and uploading synth. Stay tuned.

I see that now works. @PaulWessel should I copy those files to 'Oceania'?

gmt grdimage @earth_synbath_01d -png map --GMT_DATA_SERVER=candidate

[PaulWessel]

Thanks will fix that shortly. Made many changes to our processing scripts and will report once I have run it through venus again - setting the permission correctly is one of many tasks!

[PaulWessel]

does the candidate dir have earth_synbath_server.txt inside earth_synbath? I am hoping to reduce errors by having you do

make release-earth_synbath

and that would

1. Copy over the files from candidate to oceania (data).
2. Rebuild the gmt_data_server.txt file

Because we have to be ultra careful not making any mistakes on oceania data I would like us to wait until the scripts and makefile is completed.

Meanwhile, refresh the placing branch and you will see changes. Note I have removed the -f options from srv_downsampler_grid.sh and srv_tiler.sh. They now have -n for a run through but do not create files. E.g., after you pull the latest you can go to the gmtserver-admin dir and run

scripts/srv_downsampler_grid.sh earth_synbath -n

and see the various tasks without running them. If you remote -n then it will do the work as before. Then, you can also try

scripts/srv_tiler.sh earth_synbath -n

and get a report there, then remove -n to do the tiling.

[Esteban82]

Ok. I will try to do it tonigth (UTC -3).

[Esteban82]

does the candidate dir have earth_synbath_server.txt inside earth_synbath?

Yes. It is possible to browse in candidate. http://candidate.generic-mapping-tools.org/server/earth/earth_synbath/

I am hoping to reduce errors by having you do

make release-earth_synbath

I try it and get this:

make release-earth_synbath
make: *** No rule to make target 'release-earth_synbath'.  Stop.

Do you mean make server-release ? I saw server_release.sh. It says "DANGEROUS" so I didn't want to run it. I undersand that it will copy all the planets data that are not in the oceania server.

So, I decided to copy the files manually. I try this and got the same previous figure:

gmt grdimage @earth_synbath_01d -png map --GMT_DATA_SERVER=oceania

[PaulWessel]

There is no target release-earth_synbath. I think you mean

make place-earth-synbath

Stay away from anything with server in the name - look at them and see if they make sense but do not run.

[PaulWessel]

See 6. item in makefile help and replace neptune with venus or synbath.

[Esteban82]

I am bit confused.

These things are just to update the data in candidate from data in my pc, right? That is, what you already did. Because I was thinking about how to update "oceania" to replace the erroneous synbath data.

[PaulWessel]

On the road; give me 90 minutes, sorry

[Esteban82]

I think that this was fixed. Do you agree @seisman ?

You could try this: gmt grdimage @earth_synbath_01d -png map --GMT_DATA_SERVER=oceania

[seisman]

I think that this was fixed. Do you agree @seisman ?

You could try this: gmt grdimage @earth_synbath_01d -png map --GMT_DATA_SERVER=oceania

Yes, the PyGMT tests all pass now. Thanks for your fix!