GNIP: GeoNode Metadata Editor Improvements

[afabiani]

Overview

GeoNode provides a metadata editor which offers a basic set of fields in a quite flat form; the current form layout makes it relatively difficult for the user to find out which fields are mandatory, and how the different information are related one to the other. The editing form should be made more user-friendly, for instance by driving the user to edit the mandatory metadata, and helping him to easily spot the missing parts.

Proposed by

Emanuele Tajariol (GeoSolutions) Alessio Fabiani (GeoSolutions)

Assigned to release

None yet.

Motivation

GeoNode is a tool whose main purpose is sharing data. Metadata are functional in this context in order for the system to be able to keep a minimum of data information and to search data according to some standard filters. This means the GeoNode is not dealing with metadata in their full complexity, as other projects do, but restricts its range of metadata functionalities to a set of elements loosely related to the mandatory information that ISO19115(1) requires.

Metadata are fundamental in cataloging data, since the more information are associated to a certain data, the easier it will be to search for and reuse it. On the other side, all this information need to be inserted and edited somehow. The edit process is usually long and time consuming, and often error prone if not properly driven.

In the open source software arena there are projects which are explicitly targeted at managing metadata, such as GeoNetwork Opensource(2) . GeoNetwork approach to data and metadata is just the opposite with respect to GeoNode: GeoNetwork concentrates on metadata, while leaving the data dissemination as an ancillary service. This complete dedication to the metadata aspects makes GeoNetwork well fit in any system which requires an all-around tool for dealing with metadata (metadata editing, standard query protocols and so on), so that GeoNode itself contemplates GeoNetwork as one of the possibile catalog backends. These considerations lead us to take GeoNetwork as one of the state-of-the-art models to which a metadata tool should compare to. Anyway the functionalities in GeoNetwork are way too advanced for the purposes metadata are used in GeoNode, so we may borrow just some ideas from the editor UI in GeoNetwork.

GeoNetwork offers validation procedures to check, using different integrated tools, if the metadata under editing is valid according to the ISO schema and other constraints. Those validation procedures are quite complex and do not need to be replicated inside the GeoNode editor, anyway the user should always be able to check if the entered metadata are valid, with clear indication of the fields that needs to be corrected.

Proposal

Improved edit form

The current edit form is quite simple, and only includes a set of information that are a subset of the ISO19115. Furthermore the editing fields are not grouped by topic, and the user is not well informed about which fields are mandatory and which are not.

Mandatory fields and validation

ISO19115 dictates some constraints in the various fields of a metadata document. Since one of the most important point in running a metadata catalogs is the possibility of exchanging data, adhering the most to standard formats is one way to facilitate the integration among catalogs. Most of the constraints are usually about the existence of some specific information; some other times, the constraints involve the values of more than one field, e.g. a specific field should be assigned a determined value according to the value of another field (this kind of constraints are often called cross-field checks).

In order to follow the metadata constraints, the metadata manager should be guided in compiling the various metadata fields. The editor UI should clearly show which metadata fields are mandatory, and the invalid fields should be clearly marked, in real time where possible, or at user request and when saving (for cross-fields checks). Since these checks imply the implementation of some validation logic, it can be optionally also used to disallow the user to submit a resource metadata that is not valid or complete. Such option should be enabled by the administrator in the system configuration.

Field grouping

Generally a metadata record is not a set of independent information about data. This information are often quite structured (for instance in ISO19115 the metadata are a set of information in a strict hierarchycal structure), and the various fields are inter related.

Showing to the user this inter-relation may facilitate the understanding of the semantic of the information, and the editing of the related data. All the information fields should be split in different groups in the user interface, grouping together the related metadata fields.

When the editing interface appears, only the mandatory fields should be visualized and ready for editing; the available but optional fields should be hidden at first, leaving to the user who really needs it the option to visualize and edit them. This can be easily accomplished using the widget provided by the framework.

Keywords

Keywords are free text tags which are used to categorize metadata. Given the free text nature of this data, errors such like misspellings or other minor editing issues (e.g.: using singular and plural words in different metadata) may decrease the usefulness of this tool. In order to solve this issue, often one or more thesauri are given, that is a fixed set of terms that are selected from a list. Offering a list of fixed keywords, contained in a thesaurus configured by the administrator, will facilitate the user in categorizing the data being documented. Thesauri also solve the localization issue in keyword search: being a list of fixed values, the stored value may be presented in the current user language, provided that a translation is set in the thesaurus.

The use of a thesaurus will be configured at system level by the administrator. Thesauri will be provided in a standard format(3) . If the system is configured to use a thesaurus, the keyword input field will not be a textual field, but a multi select list, where the items will be displayed in the current language selected in the browser. In the same way, in the main search page the keyword selection list will display the keywords in the searcher language.

Dashboard

Administrators should be able to have an overview on all the inserted metadata, and quickly spot any invalid metadata stored in the catalog. GeoNode already provides dashboards for

• layers
• maps
• documents

In order not to have yet another resource list, those three existing dashboards will be enhanced with

• information about the metadata status (valid or invalid),
• filtering by the status
• in-line editing capabilities
• batch editing

The metadata status will be shown in a new column in the dashboard. The status is computed when the metadata is stored, and it’s added as a new information related to the metadata itself. This field will be a new searchable field, along with owner, category, licence and so on.

From the dashboard, the administrator will be able to:

• open the resource detail page to edit the metadata (this can be only done for maps at the moment)
• edit some of the most common metadata fields without the need to open the detail editor page;
• perform a batch editing

  Batch editing

Batch editing will allow the administrator to set the same values for the edited fields for all the selected resources.

While in the dashboard, the admin will select one or more elements; the drop down action will have the new “edit metadata” entry. Once selected, the admin will be presented with an editing UI quite similar to the one for a single entry; Differences in the UI will be mainly on the fields that need to be unique (e.g. the title), which won’t be editable. When the editing will be finished (i.e. after the admin presses the “save” button on the UI), the fields that will have been edited will be updated in all the selected elements, while the unedited fields will be left untouched.

Uploading metadata

There may be cases when a given published data could already have an existing metadata in ISO19139 format, or it is desired for a metadata to be edited in an external tool which allow more information to be added to the metadata document. When such a metadata exists, it should be possible to use it to describe the related data.

GeoNode handles the single metadata fields in an internal format; the fields values are then merged into a fixed template whenever a new ISO metadata is needed, that is on creation or after that metadata have been edited. This means that the metadata XML document is recreated from scratch every time it is stored.

The requirement is to allow the user to upload its own metadata, making possible to edit in GeoNode the set of fields that the GeoNode metadata editor supports, while, at the same time, leaving unaltered all the remaining information in the metadata document. In this scenario the fixed template should not be used, but the new edited values should be injected directly in the original XML document. In order to obtain this behavior, each editable field should be bound to its xpath(4), which shall be used for both reading the values from the XML metadata into the internal model, and to store the edited values into the original uploaded XML document.

Mockups

Metadata editor

As proposed in the previous sections, the metadata editor will be modified in order to have

• field grouped by sections
• clear indication of which fields are mandatory
• handling of keywords by thesauri

The main editing page will be similar to the picture in Fig. 1: Fig. 1

The first section (“Data Info”) contains more or less the fields enclosed in the _MDDataIdentification element of ISO19115.

The second section (“Metadata Info”) contains the other elements contained in _MDMetadata.

The elements in “Responsible parties” are elements that are either referred to data or to all the metadata; by having all the responsible parties one next to the other will help the user to understand the differences among them, which can be more difficult to understand when having these references scattered through the whole page.

The info in “Resource administration” and “Attributes” sections will not be used in the _ISO19115 _ metadata, but since the geonode user is used to find and edit these information in this page, we’ll leave them here.

The first 3 sections contains all of the _ISO19115 _ _mandatory _information, so that as soon as the editor page is presented, the user will have a clear idea about the information he has to insert in order to come out with a valid metadata record associated to the resource.

Since the GeoNode user is usually allowed to edit also some more optional fields than the mandatory ones, we want to keep such fields in the editing interface, but we don’t want them to clutter the UI.

In each section that allows more optional fields, there will be a button “other information” that will expand the section panel and show such fields (Fig. 2).

Fig. 2

Metadata editor: “data info” section

Fig. 3

Most of the fields in this section (Fig. 3) are already present in the current GeoNode metadata editor. What’s new here is the possibility to enter keywords related to one or more specific thesauri.

A dropdown selection list will be presented to the user with all the available thesaurus terms.

The administrator will be able to tell to the system whether a keyword from a given thesaurus should be mandatory or not. In such case, the thesaurus dropdown will appear in the base editing page; otherwise, the keyword selection from such thesaurus will only appear when the section is expanded, showing the “Other data information”.

Metadata editor: “data info” extended section

The picture in Fig. 4 shows the expanded “Data Info” section. Fig. 4

Most of the fields are already presented in the current GeoNode version. We also have the optional thesauri keywords, as explained in the previous paragraph.

Metadata editor: “data info” section

Fig. 5

The fields in these section (Fig. 5) are exactly the same as the ones in the current GeoNode metadata editor.

Should we need to be strictly conformant to ISO19115 specification, the “Other restrictions” field should be editable only when the “Restriction” dropdown selects the “Other restrictions” entry.

Metadata editor: “Metadata info” extended section

The extended section only adds the “dropdown” selector for the optional Spatial Representation Info (Fig. 6):

Fig. 6

Metadata editor: “Responsible parties” section

Fig. 7

The “metadata author” field in this section (Fig. 7) is exactly the same as the one in the current GeoNode metadata editor; the author will be chosen among one use in GeoNode DB, defaulting to the current user which is editing the metadata.

The “Metadata Point of Contact” and the “Data owner” may be users not registered in the GeoNode instance, so we may want to enter the various fields one by one. ISO19115 allows for lots of info, but, in order to keep things simple, we’ll only request the position name (e.g. “GIS office”), the Organization name and one mail address.

In case more parties or role are needed, they may be added in the extended part of this section.

New Metadata Editor Page Proposal

Accordingly to what stated on the points above and the need to improve usability, we are going to propose the following solution

Overview

The metadata editor page will be splitted into tabs as shown in Fig. 8

Fig. 8

The first tab contains the mandatory entries for ISO19115, trying to keep as lowest as possible the number of fields the user must fill.

The second tab allows the users to insert optionally and advanced information, not mandatory to be compliant with ISO but still useful to improve search and browse capabilities of the catalog.

The third tab allows the users to update information about the ownership of the Layer, making available ways to define and edit responsible parties, ownership and permissions of the Layer itself.

Mandatory Fields

The Fig. 9 gives an idea on the content of the Metadata mandatory fields.

Fig. 9

The fields are divided into two sections: "Data Info" and "Metadata Info" as explained on the previous sections of this document.

The Keywords are automatically inserted into the thesaurus when the user saves the metadata. The thesaurus can be always managed via the Admin interface.

The Topic Categories presented by default are the one mandatory for ISO, but they are still customizable from the GeoNode Admin GUI. It will be possible add more categories and associate icons in order to provide a visual feedback to the user instead of a boring long list of categories.

Advanced Fields

The Fig. 10 shows how the advanced fields are presented on a separate tab, which the user can optionally fill if interested on specifying very detailed information about the Layer.

Fig. 10

Ownership

The Fig. 11 shows a draft mockup for the fields related to the ownership of the Layer and Metadata.

Fig. 11

It is worth to point out that "Responsible Parties" section is dynamic. It will be possible to add and remove many different type of responsible parties.

The user can also save/load information to a shared address-book in order to avoid writing them every time.

Useful links

• (1) http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=53798 ISO19115 is an “International Standard [that] provides information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of digital geographic data." (as reported in the ISO19115 document itself).
• (2) http://geonetwork-opensource.org/
• (3) http://www.w3.org/TR/2009/REC-skos-reference-20090818/
• (4) http://www.w3.org/TR/xpath20/ Xpath is an expression language that allows, among other things, to define the position of a given element within an XML document.

[capooti]

Thanks, very nice GNIP, looking forward to see it implemented. Regarding keywords I love the idea of the localized thesaurus. I would make this configurable: in some case may be desirable to just use keywords as free typing tags. Regarding the implementation I would stick to django-taggit and provide an external application to provide the extended feature that let the user to provide the translations.

[tomkralidis]

@afabiani thanks for this very valuable GNIP. I think this will help move GeoNode's metadata story along quite nicely. A few questions:

• is any XSLT being proposed here? Or does lxml continue to underpin the XML functionality?
• pycsw is the default CSW catalogue of GeoNode. We need to ensure the pycsw bindings remain/are updated, given pycsw how tightly binds to the GeoNode Django model
• it would be valuable to have a section on backward compatibility
• is the approach here to add deeper support for ISO 19115? Are there any considerations for extensibility (such as INSPIRE, WMO Core Metadata Profile, etc.)?

[etj]

Hi @tomkralidis:

• XSLT: no foreseen XSLT involved: as reported in the proposal, each field in the editor should have an associated XPath needed to extract / replace the fields value (in case the oswlib does not provide a direct mapping), which will be handled programmatically; so the idea is to stick with the existing xml libs.
• pycsw: the metadata update procedure would modify the metadata document, and replace it inside pycsw as well. In other words pycsw will still be the central component.
• Backward compatibility: no compatibility issues are foreseen. Some existing metadata may be marked as "invalid" if any mandatory field is missing, but this info will only be presented to the editors as a way to easily spot missing data and will not prevent the resources to be used as ususal.
• Extensibility: the proposal does not take in consideration any ISO19115 application profile. It's only aimed at improving the metadata adherence to ISO19115/19139 schema, and at simplifying the editing tasks.

[stefanocudini]

+1

[dvictori]

I'm a simple user but this proposal is very interesting. We've been using GeoNode for metadata / data catalog and one thing we found missing was the ability to add more Points of Contact and define the different roles. This GNIP addresses those issues. Great!

[cgiovando]

Some comments, partly based on these more recent mockups, but largely apply to the above too:

• build with usability as main driver - chetelodicoaffà? :wink: - for people not be discouraged by inefficient data input methods. This is in line with the idea of creating reference dictionaries for content to come up automatically when users start typing or just to be able to pick from a list of values. Limit free text entries and long lists to scroll through to the minimum;
• automatic saving of forms, so data is not lost if user does not click save every time;
• batch editing for all users, both admins and data contributors, or at least provide ways to store common information to all layers by each user in his/her profile section (e.g. contact information elements);
• add contextual tips and suggestions throughout forms to clearly explain what the user is expected to fill in and how;
• make use of "smart" (not annoying!) automatic reminders that ping users by email or through on-site notification when something is missing or needs to be updated;
• reward users who fill in metadata;
• have solid schema mappings for automatically importing metadata from uploaded user files, already mentioned for ISO19139, but make sure whatever ESRI uses by default for shapefiles is also supported;
• have site-wide configurable policies (by admin) to require various levels of data completeness ...finding a balance between discouraging data contributions (because of metadata input burden) and having enforced sets of metadata;
• for "date" in the mandatory section, will it allow to enter multiple "date types"? If so, make it explicit when clicking the dropdown menu. Also it may be better to put "type" before (above) the actual date selector;
• for "topic categories" unless they look very intuitive (or people are already familiar with them), I think a dropdown menu with text entries may be faster for selection;
• for "regions" dynamically filter out anything outside of layer's bbox, so that the list to choose from gets shorter and users don't have to scroll all the way down every time to select e.g. South America/Venezuela if the list always starts with Africa;
• very often the "metadata info" section will be the same for layers from the same author, so again, make sure there are all kind of options for batch editing, or at least carry over from the previous entry, so that it gets pre-filled, and the user then is just asked to confirm that everything it that section is correct;
• somewhere in the workflow would be good to have the user confirm (by clicking an "I agree" button") that he/she has the rights to publish that data and assumes the responsibility for metadata entered (...thinking especially for the license choice);
• you may already be planning to, but would be good that all metadata entering be built on an API, so that external applications can automatically update elements. Let's say a department uses a custom app to edit road data which is then automatically pushed/synced to a Geonode, would be good for the metadata to work the same way so that authors don't have to manually login to update e.g. last update date;

In addition to usability and interface design, I think it's as much important to find strategies to engage people with a task that's traditionally boring a repetitive. Gamification, rewards, automatic reminders, visual completeness meters are some ideas. It must be a balance between annoying, and effective :)

[d3netxer]

Could someone please share with me ISO19115?

[jj0hns0n]

@afabiani do you think this will be ready for the first 2.5 (July 15th)

[etj]

@jj0hns0n I'm afraid it won't, anyway many parts on the backend will be ready.

[jj0hns0n]

@etj @afabiani can you give us an update on this?

[afabiani]

Yep, we are working on this, but I'm afraid we won't be able to finish for the current release.

[sintsh]

In Geonode we need to use external Geonetwrok for metadata handling. To do this we changed the local settting pycsw to external GeoNetwork. After that what we think is that we can get the metadata of Geonetwork into Geonode but we cannot get it for discovery purpose. To do this discovery service from Geonode how can we do this? Anyone can I get Help? Thank you in advance