Load Layers: Soil and Landscape Grids of Australia

[Tasilee]

General: http://www.clw.csiro.au/aclep/soilandlandscapegrid/index.html

Soils: http://www.clw.csiro.au/aclep/soilandlandscapegrid/ProductDetails-SoilAttributes.html (12 layers)

Landscape: http://www.clw.csiro.au/aclep/soilandlandscapegrid/ProductDetails-LandscapeAttributes.html (first 13 layers)

These are far more accurate and reliable that the existing equivalents so will REPLACE them.

[M-Nicholls]

there are three version of each of the soils attribute maps - 5th percentile, estimated and 95th percentile, which did we want to load?

[Tasilee]

Good point. There are 6 depths for 10 of the soil layers (0-5cm, 5-15, 15-30, 30-60, 60-100, 100-200). I'd suggest we just need the 0-5cm layers for the ALA but I'll check with a few experts.

[Tasilee]

Kristen Williams is hoping that TERN/AusCover will produce composite layers. Meanwhile, I suggest we go ahead with the 0-5cm ones.

[M-Nicholls]

ok, will do, and for the soil depth map - https://data.csiro.au/dap/landingpage?execution=e5s2, which of the percentiles?

[M-Nicholls]

Those percentiles are within each of the depths as well e.g. http://www.clw.csiro.au/aclep/soilandlandscapegrid/MetaData/ASLG_File_Naming_Conventions.html

[Tasilee]

Hmm, didn't look like the percentiles were showing at http://www.clw.csiro.au/aclep/soilandlandscapegrid/ViewData-QuickView.html but if I read the table on your link right and they are layer options, then all we want is the 0-5cm EV (estimated value).

[M-Nicholls]

Large (>2GB) tif files don't load using current processes - have to be converted to an image pyramid and then requires manual steps to migrate

[ansell]

I am attempting to load one to get a timeframe/resource requirements for loading them all as someone has requested them specifically for a Phylolink demo. Was there any decision about whether to use the 5th or 95th percentile confidence or the estimated value?

Steps so far (will automate at a later date but keeping track of what I have done so far here):

• Request WebDAV access to the files from DAP (takes a few hours to get them onto online storage, so they email you when that is complete)
• Setup davfs2 on data staging machine of choice (after checking the machine can access spatial-test.ala.org.au:22, otherwise put in a ticket to open a firewall port to it, as I needed to)
• Copy /etc/davfs2/davfs2.conf to ~/.davfs2/davfs2.conf and uncomment the "secrets" line
• Add your user to the davfs2 group and relogin to activate the group membership
• Setup a file system mount for the webdav link in email the DAP sends in /etc/fstab

  https://webdav-bm.data.csiro.au/dap_prd_000010889v003/ /mnt/webdav-mounts/tern_soil_grids/dap_prd_000010889v003/ davfs user,noauto,file_mode=600,dir_mode=700 0 1

• Add the URL/username/password from the DAP email to your ~/.davfs2/secrets file
• Rsync (so you can restart if it breaks) the files across. In my case, my local computer doesn't have enough resources to store and investigate the 4GB files easily, so sending them to a server:

  rsync -av /mnt/webdav-mounts/tern_soil_grids/dap_prd_000010889v003/ ans025@ala-cave.it.csiro.au:/data/tern_soil_grids/dap_prd_000010889v003

[ansell]

Then, converted one layer file to EHdr using the following to test:

SSH to ala-cave, then

cd /data/tern_soil_grids/dap_prd_000010889v003/000-005cm/
mkdir NTO_000_005_05_N_P_AU_NAT_C_20140801
gdal_translate -of EHdr NTO_000_005_05_N_P_AU_NAT_C_20140801.tif NTO_000_005_05_N_P_AU_NAT_C_20140801/NTO_000_005_05_N_P_AU_NAT_C_20140801.bil
zip NTO_000_005_05_N_P_AU_NAT_C_20140801.zip NTO_000_005_05_N_P_AU_NAT_C_20140801/*

If it were possible, I would have copied that zip file straight to spatial-test, but instead I needed to copy it through a third host that has access to both.

Then on spatial-test:

cd /data/spatial-data/uploads/
unzip NTO_000_005_05_N_P_AU_NAT_C_20140801.zip
chown -R tomcat7:tomcat7 NTO_000_005_05_N_P_AU_NAT_C_20140801

Then, open:

http://spatial-test.ala.org.au/spatial-service/manageLayers/uploads

Tail the tomcat logs to know when to refresh that page as it failed the first time with HTTP 502, while it was preprocessing the new layer

Once preprocessing done, refresh page and click on create layer next to the desired layer, fill in metadata, and start the layer creation.

Then after seeing no activity on task list or logs, restart tomcat to get it to recognise the new layer. Hypothetically caused by the task scheduler crashing at some point since the last time tomcat was restarted, but no evidence of that other than a hunch and background knowledge knowing that restarting tomcat on spatial-test allows it to progress.

After that completes, create a field from it, and check task list to monitor the 570+ tasks spawned for the creation of the field and its tabulation with other fields. Wait.

[Tasilee]

I think we skip the 5% and 95% layers for now. The concept of these extra layers for each grid layer is that 90% of the time, the value of the grid cell will be between the 5% value and the 95% value.

[ansell]

Thanks @Tasilee , I will focus on the Estimated Value layer in future. However, the first layer loaded was a 5% layer, which looks like it was successfully loaded enough for them to preview it, and knowing that the 95% and EV layers complement it, I kind of want to load them up beside it to give a single demonstration of "Total Nitrogen (0-5cm)" for all three range values:

http://spatial-test.ala.org.au/spatial-service/manageLayers/layer/10904

I am only going to do the 0-5cm for the initial run, as discussed above.

[ansell]

The webdav mount cannot left in place on the data staging machine, it seems, based on it crashing with blocked/hung tasks over the weekend, so it is necessary to unmount it and then comment it out of the /etc/fstab file once the rsync completes. This experience comes from two days of debugging why I couldn't get a shell using ssh, but I could still authenticate to, ala-cave, including waiting for IM&T to restart the machine using the vmware console.

General method to minimise the time connected to webdav is:

mount /data/tern_soil_grids/dap_prd_000011030v001 && rsync -av dap_prd_000011030v001/ dap_prd_000011030v001-rsync && umount /data/tern_soil_grids/dap_prd_000011030v001

[Tasilee]

Email from Kristen WIlliams regarding updated data at 9 second (~250m): https://doi.org/10.4225/08/5b285fd14991f

[Tasilee]

Reviewing: The layers here will replace most of the soil and landscape layers we currently have. This suite of data is definitive.