QEP 50: Enhanced Layer Metadata Management

[tomkralidis]

QGIS Enhancement 50: Support for Enhanced Layer Metadata Management

Date 2015/03/26 Author Tom Kralidis Contact tomkralidis at gmail dot com Last Edited 2015/03/26 Status Draft Version QGIS 2.10

Summary

This QEP aims to provide enhanced metadata management within the Layer Properties dialog, providing the following benefits:

• fulsome metadata representation of QGIS layer metadata at the core (no duplications or QGIS plugins)
• streamlining downstream plugins to operate against QGIS layer metadata at its core to provide value-added properties/metadata profiles, validation, etc.
• easier publishing (via MetaSearch)
• enhanced discovery (better metadata is more easily discovered)

  Overview

QIGS by default provides a Layer Properties Metadata tab which allows a user to enter various values (and select from various select boxes) under various sections like Description, Attribution, MetadataUrl, LegendUrl. The Properties section is autogenerated from the layer and is ready only. These values are stored in the QGIS project file within the <maplayer> element.

Currently, there are no mechanisms to:

• save values in Metadata tab as a standalone standards-based (ISO, Dublin Core) metadata document (XML)
• interact with other QGIS plugins such as MetaSearch or Metatools

This leaves the QGIS layer metadata management story fragmented and spread across many internal mechanisms.

It is proposed to improve layer metadata management at the core of QGIS by enhancing the Layer Properties Metadata tab with a core QGIS metadata model, thus allowing for better metadata export, publishing, and use by downstream plugins seamlessly, efficiently, and in an authoritative manner.

This QEP will support an overall workflow per below:

manage Layer Metadata (QGIS core) -> publish/update/delete Layer Metadata (MetaSearch) to CSW

Proposed [Technical] Solution | Change

The Layer Properties Metadata tab will be reviewed to assess the following change improvements:

• design (tabs vs. tre)
• elements review

  Implementation Details

The UI file will be reviewed and modified.

Changes to the metadata model will be applied to the QGIS project file.

Affected Files ...............

• https://github.com/qgis/QGIS/blob/master/src/ui/qgsvectorlayerpropertiesbase.ui
• https://github.com/qgis/QGIS/blob/master/src/ui/qgsrasterlayerpropertiesbase.ui
• https://github.com/qgis/QGIS/blob/master/src/app/qgsvectorlayerproperties.cpp
• https://github.com/qgis/QGIS/blob/master/src/app/qgsrasterlayerproperties.cpp
• https://github.com/qgis/QGIS/blob/master/src/mapserver/qgswfsprojectparser.cpp
• https://github.com/qgis/QGIS/blob/master/src/core/qgsmaplayer.cpp
• https://github.com/qgis/QGIS/blob/master/python/core/qgsmaplayer.sip
• https://github.com/qgis/QGIS/blob/master/src/core/qgsmaplayer.h

  Test Coverage

(required for technical solutions/changes if applicable)

Performance Implications

No performance implications are expected at this time given the lightweight nature of this QEP to the codebase.

Further Considerations/Improvements

This QEP could be extended by adding deeper layer metadata support by providing direct ISO or Dublin Core support/validation.

Given the below workflow:

manage Layer Metadata (QGIS core) -> publish/update/delete Layer Metadata (MetaSearch) to CSW

QGIS MapServer could be enhanced to link directly to the metadata url generated by publishing to a CSW.

As well, a QGIS-pycsw module could provide a CSW service much like QGIS MapServer, in which case layer publishing to MapServer could also include metadata publishing inline.

Restrictions

Backwards Compatibility

Given that the layer metadata model will be enhanced and not changed, there will be little impact on backwards compatibility. There will be a UI design/change which will not break functionality.

Documentation

Documentation will be enhanced accordingly in http://docs.qgis.org/2.2/en/docs/user_manual/working_with_vector/vector_properties.html#metadata-menu

Issue Tracking ID(s)

(required)

References

(optional)

Miscellaneous

(optional)

Voting History

(required)

[pcav]

I fully support this.

[NathanW2]

@tomkralidis is there anyone that you know who maintains the metadata parts of QGIS?

[tomkralidis]

Not offhand in core. Perhaps @timlinux would have a better idea.

[NathanW2]

+1 from me on this.

This leaves the QGIS layer metadata management story fragmented and spread across many internal mechanisms.

This is a good objective for stuff like this. Nothing worse then having to use little bit of each thing to get the job done.

[timlinux]

Not offhand in core. Perhaps @timlinux would have a better idea.

Yes to confirm, there is not anyone specifically maintaining metadata.

[timlinux]

I have a couple of concerns about metadata lifecycles. Firstly I should mention that I am supportive of this proposal - @tomkralidis and I have chatted about it offline from this proposal too.

My main concern is about how we will manage layer metadata so that it

• Persists between project instances that use the same layer: an update to layer metadata in one instance should ideally propagate to other instances.
• Caters for remote datasets (e.g. postgis connection) where the data does not physically reside on disk: I propose we could use a similar mechanism to that currently used for storing styling info in the database.
• Updates itself as the dataset changes: I would like to have a mechanism in place so that basic info such as extents, layer preview etc. can get automatically written to the metadata as a layer is changed.

[ninsbl]

From experience, Metadata is often neglected because documenting data is often cumbersome and standards are complex and thus editing metadata can be time consuming and complicted. Therefore, this QEP is very interesting and I think that a good user experience is essential in order to make people document their data. In order to facilitate the users experience, as much metadata as possible should be added automatically, either from templates or other available information.

• Similarly to most other editors (GRASS GIS [1], GeoNode[2]), add basic info from the data source itself (CRS, Extent, data ranges, attributes…).
• User specific templates / default values would be useful for efficient metadata editing
• It would be nice if users could configure, or add configured information from file or db sources, about users, project etc. For example, TAGS related to point of contact could be populated, by choosing a name from a (centrally managed) list (e.g. from Catalogue accounts, other contact details tables, or Groupware). Here the solution in QSphere [3] might give some inspiration.
• Furthermore, for editing purposes, it would be advantageous if users could configure sets of keywords that can/should be used, thus to allow to choose from a list of controlled vocabulary rather than being totally free of adding whatever content, for specific metadata entries (to increase standardisation). The list of keywords should be fully customisable (also here the solution in QSphere [3] might give some inspiration).

For the second and third bullet point, QGIS would need to allow storing (and importing) configurations (e.g. in a sqlite db? Json settings file? ecc.) for e.g. contact or keyword sources, CSW endpoints used for publishing (#33), probably together with authentication information (if not LDAP or the like is used).

Even if such functionality can be found in specific metadata tool (GeoNetwork, GeoNode, ...), the big advantage of having this in QGIS is the much closer integration with user works and that metadata is generated where the data is produced too. And of cours, not all users of QGIS can afford to run server oriented metadata solutions like the named above. And they have the need to document their data too!

[1] https://grasswiki.osgeo.org/wiki/ISO/INSPIRE_Metadata_Support [2] https://github.com/GeoNode/geonode/issues/2408 [3] https://qgis.projets.developpement-durable.gouv.fr/projects/qsphere

[tomkralidis]

@timlinux / @ninsbl thanks for the comments. My initial thought is for QGIS to manage an SQLite3 instance of the metadata with a link to filepath/connection as a means to keep linkages intact across projects.

I do agree that the huge win here is metadata management agnostic of CSW, and baked right into QGIS.

[timlinux]

Please note there is ongoing discussion on this topic in a dedicated gitter channel here:

https://gitter.im/qgis/metadata

And a draft specification is being worked on here:

https://docs.google.com/document/d/1twE8J345Sz1rk1z34672eqlEiL2mJT4bZbuF5oRTCoM/edit?usp=sharing

Please feel free to add comments to that doc.

[timlinux]

Please note that this QEP is largely superseded by QEP #91 - please continue any discussion there.