Support InaSAFE keywords editing in GeoNode

[gubuntu]

As part of the GeoSAFE improvement effort we will improve the metadata editor in GeoNode since it currently has an issue whereby changing the metadata using the GeoNode metadata editor will remove InaSAFE keywords from the metadata document. Thus our intent is to improve the metadata editor to support non-destructive metadata editing (i.e. leave InaSAFE keywords intact)

We also propose to implement an extension to the existing GeoNode metadata editor that will support the creation of InaSAFE specific keywords within the existing ISO19115 document for a layer. This will largely emulate the InaSAFE desktop keywords editor, but be available within the web environment.

• [ ] ensure GeoNode metadata workflow is non-destructive
  • [x] no matter how the original metadata is captured it should be treated the same and remain lossless (either with data upload, separate upload or UI capture)
  • [ ] existing metadata should be editable via the UI in a completely lossless process.
• [ ] UI metadata editor must handle specific non-standard extensions non-destructively (InaSAFE and ThinkHazard)

[ismailsunni]

Should we wait until geosafe is migrated to use InaSAFE 4.x ? There is a big difference between keywords in version 3.5 (current geosafe) and version 4.x.

cc @timlinux

[gubuntu]

@ismailsunni I agree the InaSAFE keyword aspect must wait until inaSAFE 4 is in GeoSAFE but you can still do generic work that will make metadata editing non-destructive

[ismailsunni]

I try to investigate the problem here, and I got surprising(?) these results:

1. The InaSAFE metadata is NOT destrucitble with update geonode metadata or upload new metadata file. a. Update metadata through wizard / advanced metadata editor, will only change the metadata in the geonode model. The metadata that's used by geosafe is stored as real file (an .xml file). So, the changes won't have any effect in the InaSAFE metadata. b. Update metadata through upload metadata option will give you an error or geonode will create new layer object (not updating the current metadata). See #251

2. Actually, geonode have supplemental infomation field in its metadata. But, when we upload a layer, geonode harvest the xml file then generate the fields' content. Unfortunatelly, the owslib that they use to get the values from the xml file use gmd:supplementalInformation/gco:CharacterString to get supplemental information. Mean while, for InaSAFE we have gmd:supplementalInformation/inasafe_keywords. So, it will be always ignored.

[Gustry]

Thanks for this review. What about the checkbox Preserve Metadata XML. What is happening in the background ?

[lucernae]

Hi @ismailsunni I'm aware of this problem when working on oondra. This is just like what you said. So, for @gubuntu and myself, this is probably not surprising. In fact, this is exactly the problem.

What we did for Oondra because of this is we escape inasafe_keywords as gco:CharacterString because it was the only way possible at that time. The other option is for inasafe_keywords to have proper ISO metadata, which is not possible at that time.

In my opinion, the reason why Geonode metadata and InaSAFE metadata doesn't play nice together at the moment is because we put all InaSAFE related metadata in supplementalInformation as an incorrect format. If you look at the schema: http://www.isotc211.org/2005/gmd/identification.xsd and see that supplementalInformation indeed have type gco:CharacterString. So, in my opinion, InaSAFE should follow this in the future and put inasafe keywords there as CharacterString type, which means, the whole inasafe_keywords xml tree needs to be escaped (if we want to put it in supplementalInformation). But honestly, I'm no metadata expert, so you should probably ask experts help on this decision. I'm just saying that we should not change the way Geonode get supplementalInformation (using gmd:supplementalInformation/gco:CharacterString) because I think it is already correct.

I will just list some alternatives from the top of my head here, just put some additional pros and cons if you want:

1. Treat InaSAFE metadata as a different document:

Implementation

GeoSAFE will be responsible for extracting inasafe_keywords, then upload it to geonode as a different file or maybe 'document', and linked with the original layer upload. But because of this, no changes in Geonode part.

Pros

• Will easily be integrated for InaSAFE 3.5 and 4, because it works the same way by extracting the keywords.
• No additional changes in Geonode metadata UI. If we need to, we can override metadata UI template from GeoSAFE to edit InaSAFE keywords like in the desktop wizard.

Cons

• New layer upload will need to separate inasafe keywords, and we need to somehow hook into layer upload for GeoSAFE to extract the keywords. I don't know how to implement this in django. We will probably need to add new signal in Geonode for GeoSAFE to hook into?

2. Insist that InaSAFE keywords be put on supplementalInformation

Implementation

Escape inasafe keywords xml tree structure into a CharacterString. InaSAFE keywords is still on the same xml metadata.

Pros

• Metadata upload doesn't change. It will be saved in db, I suppose.
• No need to use Geonode signal. GeoSAFE can override metadata UI template views if it needs to.

Cons

• Not sure if this is the right thing? It seems like a hack
• Will need to review how inasafe keywords will be saved in InaSAFE metada file. Or maybe put it on another field? Which does the correct thing? Need to consult to metadata expert anyway.
• Will have to update both InaSAFE 3.5 and 4. On oondra, we just populate supplementalInformation directly in Geonode metadata UI, with escaped inasafe keywords.

I hope this brings more context.

[ismailsunni]

Thanks @lucernae for your comment. Perhaps @timlinux or @gubuntu want to read it also.

For the 1st alternative, how will InaSAFE find the InaSAFE metadata keywords if it's in the separated file? Currently, InaSAFE will only check from the same filename as the layer file (only different extension).

For the 2nd alternative, will it work on InaSAFE? If I read correctly, you want to treat the InaSAFE keywords as string. Then InaSAFE will need to change the way how it reads the keywords.

I was thinking that we can store the inasafe related keywords in the geosafe db, but it seems it's difficult to make sure that the metadata file in the layer contains the InaSAFE keywords.

Sorry, my finger slipped, accidentally close the ticket before.

[timlinux]

In my discussion with @ismailsunni this morning I proposed another (simpler?) option:

• implement a save signal so that when metadata is saved in GeoNode, we can call a custom function
• in the custom function, obtain the key fields we care about (title, description, category etc.) that are about to be written to the layer model
• write these updated properties to the on-disk copy of InaSAFE keywords metadata file

Of course this will have limitations for remote datasets but will be a step forward in making the file based layers work well....

[ismailsunni]

Thanks for this review. What about the checkbox Preserve Metadata XML. What is happening in the background ?

@Gustry it will the metadata non-editable since it wants to preserve the metadata file

[ismailsunni]

I have made basic implementation like @timlinux's comment above. Should I also move forward to make InaSAFE metadata editor?

[timlinux]

@ismailsunni yes - we need to provide a similar environment to the desktop metadata editor.

[lucernae]

@ismailsunni can you give me an update on this?

[ismailsunni]

Current situation:

• There are two metadatas (from geonode and from xml file).
• The geonode one is editable. But the update is not applied to the xml file. So, InaSAFE will still work although we edit the geonode one. But, we can not edit the xml file content
• Per discussion with @timlinux , it's bettter to upgrade GeoSAFE to use InaSAFE v4 first to avoid working twice on the metadata editor

Update: point 2 above should be we can update the geonode one, and some properties that related to InaSAFE are updated to the xml file one. E.g. title.

[lucernae]

It seems possible to work on point 2 first. What I will make will be to replace the xml file but retain the inasafe-keywords for now. But this needs to be implemented in GeoSAFE.

[gubuntu]

Do we have any ability yet to edit metadata in QGIS client via GeoNode API? cc @gustry

[gubuntu]

We can do this properly once GeoSAFE is upgraded to InaSAFE v4 but in the meantime if we have to get something working with 3.5 I propose a post-metadata-save hook that writes out the metadata file to disk alongisde the hazard or exposure layer so that InaSAFE uses it.

• user edits InaSAFE 3.5 metadata XML in Supplemental Information field of a layer and Saves
  • the geonode save operation will escape the InaSAFE keyword XML, which will have to be unescaped for writing to disk.
• Updated metadata is available for GeosAFE.

[gubuntu]

@lucernae if this is address by https://github.com/kartoza/geosafe/pull/227 please merge and deploy that asap.

[lucernae]

That PR at the moment address these things:

• Make sure Geonode's metadata editor works and doesn't destroy InaSAFE metadata
• Show current InaSAFE specific metadata if any.

[gubuntu]

@lucernae does https://github.com/kartoza/geosafe/pull/378 fix this?