Load Layer - 2019-2020 fires footprint areas

[Tasilee]

This area has been requested by a number of scientists for the evaluation of the impacts of this fire season. Ideally, tabulating it with the other contextuals would be nice. This would enable evaluations such as the impact of the fires on CAPAD areas.

Investigation and approval

• [ ] The information doesn’t already exist (e.g., IPAs are in CAPAD)
• [ ] Is it an AREA (single or multi-polygon that could be imported/saved) rather than a layer?
• [ ] National coverage, or international coverage where data helps place the Australian region in context
• [ ] Presents information that suggests spatial control of species distributions or would be use-ful in the interpretation or management of species and areas
• [ ] Data is from an authoritative source
• [ ] Spatial resolution of 1km or better preferred
• [ ] Fills a gap in or complements our layer collection (name, space, time, environmental factor)
• [ ] Anticipated broad usage by one or more of the following communities (research, area managers, environmental consultants, citizen scientists, education, public)
• [ ] Has quality metadata
• [ ] Data is in a form that is readily ingested into our systems
• [ ] Is up to date
• [ ] Supersedes an existing layer (deprecation or deletion)
• [ ] Anticipates future requirements
• [ ] Supports nationally significant projects (which themselves would need some evaluation, but GBR management given World Heritage status and threats would pass)
• [ ] Openly available (free and open source data).
• [ ] Externally funded? Probably relates to (14).
• [ ] Obtain approval from ALA Management to load layer based on the investigation

Loading the layer

Once layer loading has been approved, the following are the steps required to load, integrate, and test it.

• [x] Download the original shapefile and metadata file http://www.environment.gov.au/fed/catalog/search/resource/details.page?uuid=%7B9ACDCB09-0364-4FE8-9459-2A56C792C743%7D
• [x] Analyse the shapefile for field value expectations (Single entry in the shapefile as a very complex MULTIPOLYGON with Id=0 as a field attached to it. Loading this failed, so created a new field for the single row with label='0' (text type, rather than numeric), still failed, Adam fixed it manually without saying how)
• [x] Select one or more fields from the shapefile to be included as fields in spatial/biocache
• [x] Upload layer at: https://nectar-spatial-staging.ala.org.au/ws/manageLayers/uploads
• [x] Once the shapefile upload is complete, choose Create layer https://nectar-spatial-staging.ala.org.au/ws/manageLayers/layer/10955
• [x] Enter the layer metadata and choose create layer
• [x] Once the layer is loaded, create one or more fields https://nectar-spatial-staging.ala.org.au/ws/manageLayers/field/cl10955
• [x] Once the fields are created, check that they are visible and correct on nectar-spatial-staging spatial-hub instance by adding the new Layer to a map. Verified:

• [x] If not visible, run https://nectar-spatial-staging.ala.org.au/ws/tasks/create?name=Thumbnails
• [x] If still not visible, restart nectar-spatial-staging
• [x] If still not visible, wait for 24 hours and check again
• [x] If still not visible, give up and submit an issue on github with the details for Adam to look at
• [x] Once the fields are visible, go to https://spatial.ala.org.au/ws/manageLayers/remote and choose Copy layer

• [x] Once the copy is complete, check that the layer copy succeeded. The layer should not be Enabled at this stage, as a final safety if the copy failed to complete successfully.

• [x] Finally, copy the layer to ala-dylan based on running the following on ala-dylan and activate it:

  export LAYER_NAME="niafed_20190701_20200323_v20200324_2" # This is the short name for the layer
  export LAYER_ID="10955" # This is the numeric identifier for the layer
  export FIELD_ID="cl10955" # This is the cl/el identifier for the field
  export API_KEY= #Obtain API key from at https://auth.ala.org.au/apikey/
  /root/copy-layer-from-aws-spatial-prod.sh "${LAYER_NAME}" "${LAYER_ID}" "${FIELD_ID}" "${API_KEY}"

• [x] Enable the layer and the field on https://spatial.ala.org.au/ws/manageLayers/layers
• [x] Check if the layer with LAYER_ID is enabled in https://spatial.ala.org.au/ws/layers
• [x] Check if the field with FIELD_ID is enabled in https://spatial.ala.org.au/ws/fields
• [x] Restart tomcat on ala-dylan.it.csiro.au

  sudo service tomcat7 restart

• [x] Test intersection sampling is working using the following and then inspecting ${FIELD_ID}-intersection-output:

  export INTERSECTING_COORDINATES=-35.5811223,148.9369979;
  export FIELD_ID="cl10955" # This is the cl/el identifier for the field
  /root/check-intersection.py --field-id=${FIELD_ID} --intersecting-coordinates=${INTERSECTING_COORDINATES} --output-dir ~/${FIELD_ID}-intersection-output

Intersection worked, giving the expected value of 0:

root@ala-dylan:~# cat ~/cl10955-intersection-output/sample.csv 
latitude,longitude,cl10955
-35.5811223,148.9369979,0

• [x] Check the fieldsdb returned by ala-dylan (https://sampling.ala.org.au/sampling-service/fieldsdb) and aws-spatial-prod (https://spatial.ala.org.au/ws/fieldsdb)

  Fetch 'https://sampling.ala.org.au/sampling-service/fieldsdb' and double-check that the field now exists and 'indb: true' is on the field that was added
  TODO: automate this using 'jq'.

• [x] Run Complete Process and Sample jenkins job on aws-scjenkins http://aws-scjenkins.ala:9193/job/Complete%20Process%20and%20Sample/ (Or wait for the weekly automated job)
• [x] Run the Complete Reindex on aws-scjenkins http://aws-scjenkins.ala:9193/job/Complete%20Indexing/job/MASTER%20-%20Complete%20Re-index/ (Or wait for the next automated job after the Complete Reprocess)
• [x] Check using biocache.ala.org.au that expected records have the new cl value (individual record displays are sourced from cassandra) https://biocache.ala.org.au/occurrences/search?q=cl10955%3A* and then chose https://biocache.ala.org.au/occurrences/5af76b09-e010-4f19-a17f-64432f5080f1
• [x] Check using biocache.ala.org.au that searches on the new cl value succeed (search results are sourced from solr) https://biocache.ala.org.au/occurrences/search?q=cl10955%3A*
• [x] Check that the new field exists in each of the biocache-service /ws/index/fields calls, noting that calls to /ws/index/fields on different servers are known to be inconsistent. https://github.com/AtlasOfLivingAustralia/biocache-service/issues/263

[ansell]

Scratch that, deleting the layer and trying again.

[Tasilee]

Metadata: I asked Hamish (who was involved in the meetings) to see if he could get some but have not got a response as yet.

[ansell]

Thanks Lee, let me know when the metadata is available to continue the loading process:

https://nectar-spatial-staging.ala.org.au/ws/manageLayers/layer/10953

[ansell]

Metadata provided by Hamish at the same time as he blocked further progress on this on strategic grounds while the spatial layer management problem bubbles through.

When this rises back to the top again, the metadata is available from:

http://www.environment.gov.au/fed/catalog/search/resource/details.page?uuid=%7B9ACDCB09-0364-4FE8-9459-2A56C792C743%7D

[ansell]

Hamish reversed his decision, so this is back on for loading again as a priority.

[ansell]

Deleting the previous staged layer as the website above provides a newer version with a revision date of 2020-03-23 where the previous staged layer had 2020-02-24 as its revision date.

[ansell]

The shapefile contains a single MULTIPOLYGON, with the label 0. Queries on this version will need to select the entire multipolygon as one search filter, but other search filters in biocache can be applied on top of it.

[ansell]

New version of the staged layer is at:

https://nectar-spatial-staging.ala.org.au/ws/manageLayers/layer/10954

Loading the field for it now...

[ansell]

The field creation dialog didn't recognise any field ids from the shapefile (or it is filtering out the Id field that is present), and the field creation failed to do anything (no errors in the web interface or the server logs.............. at all...........).

Will modify the shapefile locally to add a label field to the shapefile for the polygon and try the loading process again.

[ansell]

Adding a label column (text type, instead of integer as the Id column has) is still failing to show up any source columns in the field creation process, which is still failing.

https://nectar-spatial-staging.ala.org.au/ws/manageLayers/layer/10955

That is the end of my knowledge of debugging for the layer-loading system.

[Tasilee]

Thanks @ansell ...I'll call for help @adam-collins

[adam-collins]

https://nectar-spatial-staging.ala.org.au/?layers=cl10955

Loaded on nectar-spatial-staging

[ansell]

@adam-collins What was the key to getting the label field to show up?

[adam-collins]

@ansell This layer load only required more time. When I attempted to create the field, after you created the layer, sufficient time had passed and the label dropdown was available for use.

If I recall correctly, the LayerCreation process must be finished before a field can be created. This should involve not much more than a gdalwarp and loading into Geoserver. Larger and more complex layers will take longer.

[ansell]

@adam-collins Thanks, I will add a new task to the list to check the task queue before clicking "Create Field".

I had previously presumed that when it was possible to "Create Field", that the layer load would be complete at that point.

[ansell]

@adam-collins This load is blocked by the field not being populated in solr, although it is in the solr schema. Our regular Complete Reindex job has been wiping out the /data/solr/biocache/conf directory, as was required by the workaround for the previous issue where this was reported:

https://github.com/AtlasOfLivingAustralia/biocache-store/issues/315

[ansell]

cl10955 isn't showing up for me in https://biocache-ws.ala.org.au/ws/index/fields. There is another Complete Reindex job running now, will see if it appears after that completes.

[Tasilee]

We are still missing the layer in the SP's Tabulation tool, so it looks like the cross-tabulation process has not been run, or stopped, or still running?

[ansell]

@Tasilee Fixing cross-tabulation for layer loading is an open issue. The last advice I had from @adam-collins was to not check the tick box for it during loading.

[Tasilee]

Thanks @ansell. I'll prod @adam-collins to tick and go :)

[ansell]

The issue previously was with very complex layers, and the process not completing months after starting it. This fire footprint layer is possibly the most complex layer we have loaded. It may not just be a case of ticking the box.

[Tasilee]

:) I seem to remember Adam mentioning looking at point intersection processing efficiencies...and this is a candidate to check reality.

This layer is of most benefit with the cross-tabulation intersections done.