Make metadata wizard more flexible for custom metadata

[t-book]

Hi @mattiagiupponi

I saw you worked on the Extrametadata implementaion . Hence I hope it is okay to ping you directly.

I'm just starting implementing cutom metadata for a client, where it would be nice using the existing new ExtraMetadata Model. Further to keep it generic so that the community can benefit from it.

My Idea would be as following:

1. Make the current dataset_metadata more flexible
2. contrib-apps or geonode project decorate the view for passing additional data and customize panels.html

1. Make dataset_metadata more flexible

geonode/layer/views.py

def dataset_metadata(
        request,
        layername,
        template='datasets/dataset_metadata.html',
        ajax=True,
        panel_template='layouts/panels.html', <-- allow panels.html to be extended
        custom_metadata=None): <- pass additional context
      ...

    return render(request, template, context={
        ...
        "panel_template": panel_template,
        "custom_metadata": custom_metadata,
        ...

datasets/dataset_metadata.html

        <div id="mdeditor_form" class="col-md-12 form-controls">
          {% form dataset_form using panel_template %} <-- use the panel from settings.py || default
          {# dataset_form|as_bootstrap #}
        </div>

2. Example of custom metadata implementation

Doing so one would not need to touch the core anymore but could decorate the method as needed from geonode_project ot a contrib app.

As a poc:

my_app/my_view.py

from django.conf import settings
from django.http import HttpResponse
from geonode.base.models import ExtraMetadata
from geonode.layers.views import dataset_metadata, _resolve_dataset, _PERMISSION_MSG_METADATA
import json

def custom_metadata_decorator(view_func):
    def _wrapped_view(request, *args, **kwargs):
        layer = _resolve_dataset(
            request,
            kwargs.get('layername'),
            'base.change_resourcebase_metadata',
            _PERMISSION_MSG_METADATA
        )
        if request.method == 'POST':
            # Call the original view function
            response = view_func(request, *args, **kwargs)
            if response.status_code == 200:
                # Handle additional form input this should be made dynamic from settings ?
                color = request.POST.get('resource-color')
                data = {'color': color}
                json_data = json.dumps(data)
            try:
                extra_meta = ExtraMetadata.objects.get(resource=layer)
                extra_meta.metadata = json_data
                extra_meta.save()
            except ExtraMetadata.DoesNotExist:
                extra_meta = ExtraMetadata.objects.create(resource=layer, metadata=json_data)
                layer.metadata.add(extra_meta)
            return response
        else:
            # On GET, add the extra metadata to the context 
            custom_metadata = {}
            try:
                resource = layer.metadata.first()
                custom_metadata = json.loads(resource.metadata)
            except Exception as e:
                ...

            default_panel_template = 'layouts/panels.html'
            panel_template = getattr(settings, 'PANEL_TEMPLATE', default_panel_template)
            response = view_func(request, 
                                *args, 
                                panel_template=panel_template, 
                                custom_metadata=custom_metadata, 
                                **kwargs)
            return response

    return _wrapped_view

dataset_metadata = custom_metadata_decorator(dataset_metadata)

my_app/templates/custom_panels.html

{% extends "layouts/panels.html" %}

{% block dataset_abstract %}
{{ block.super }}<br>

<div id="req_item">
    <span><label for="id_resource-title">Color</label>
    <input 
      type="text" 
      name="resource-color" 
      value="{{ custom_metadata.color }}" 
      maxlength="255" 
      placeholder="Color" 
      required="" 
      id="id_resource-color">
</div>

{% endblock %}

{% block layer_extra_metadata %}
{% endblock %}

---

My question are,

• do you have a better approach in mind instead of a decorating the existing view (also in regard of mapstore2 and accessing the data)? (I would not patch the ResouceBase Model anymore but rely on the json field)
• further I wonder why a ManyToMany relation is used. In which case would a dataset have several ExtraMetadatas attached?

best regards

Toni

[t-book]

fyi @gannebamm @giohappy

[mattiagiupponi]

Hi @t-book Overall looks like a good idea so we can stop changing the core project and start to demand changes to the projects or external app changes.

To reply to your questions:

do you have a better approach in mind instead of decorating the existing view (also regarding mapstore2 and accessing the data)? (I would not patch the ResouceBase Model anymore but rely on the json field)

Not yet, but I like your solution. Mapstore will rely on the API, and as soon as is preserved via the core model should be fine.

further, I wonder why a ManyToMany relation is used. In which case would a dataset have several ExtraMetadatas attached?

The base idea of this extra metadata field was planned for GeoNode 3.3.x by adding this metadata field was possible to add facets dynamically. But on 4.x/4.1.x/master this is no longer needed. I guess we have space to work on this. The reasons for the M2M relation were:

• from UI/API (actually) is possible to add a list of additional metadata, but internally are saved as multiple instances. Let's take the following example: [{"a": 1}, {"b": 2}] The above json is saved as two different rows in the database as you can see from this part of the resource_manager. https://github.com/GeoNode/geonode/blob/2ce23ff2f65d24f00a71da3d24cac6d056057be7/geonode/resource/utils.py#L277-L285
• Makes the API easier to read/maintain/use. Let's assume that you want to save 1000 extra metadata to your resource, but (for any reason) the metadata at position 681 is no longer needed. With the actual API is possible to send the delete statement to the extra metadata with the ID 681 directly. Having a unique json can complicate this operation since you should read the whole document and re-save it. FYI the APIs are defined here So in your example, you will have two entries, one provided by the extra metadata (if present) and the second from your color metadata.

Additional notes

I see from your example that the validation of the extra metadata is skipped. Via UI and API, there is a function called validate_extra_metadata which takes care of validating the input with a pre-defined schema available in the settings. This schema helps to avoid the extra metadata being a battlefield. If you are going to override the function you can skip it, but I strongly suggest using it if possible. The schema is completely customizable by resource type, there is no need to use the default value.

[t-book]

Thanks a log @mattiagiupponi Now the m2m makes sense! Also the validation is a good point. From what I see we would need to change the serializers a bit as well?

https://github.com/GeoNode/geonode/blob/2ce23ff2f65d24f00a71da3d24cac6d056057be7/geonode/base/api/serializers.py#L515 Should become

metadata = DynamicRelationField(ExtraMetadataSerializer, embed=True, many=True, deferred=False)

and metadatashould then be added to the dataset serialializer, so that the metadata is provided for mapstore via the api, right?

https://github.com/GeoNode/geonode/blob/8ccf5fb1ecb9e23b9cea63414d51dc7f391498b9/geonode/layers/api/serializers.py#L137-L141

By the way, do you know where in mapstores codebase the keys are defined that are rendered in the detail info window from the api response? Could not find any probs or config.

[mattiagiupponi]

and metadatashould then be added to the dataset serialializer, so that the metadata is provided for mapstore via the api, right?

Adding the metadata to the dataset API should not be needed since it inherit from the ResourceBase where the metadata is already exposed.

Making the embed =True is not strictly needed since (in my opinion) is better to keep them hidden by default because if you have 1000 metadata, loading them by default along with the resource information could be problematic. If you want to see them via API is enough to call it by adding the ?include[]=metadata to the query string, for example:

http://localhost:8000/api/v2/resources?include[]=metadata

PS: AFAIR for now mapstore doesn't take care of the extra metadata

By the way, do you know where in mapstores codebase the keys are defined that are rendered in the detail info window from the api response? Could not find any probs or config.

I'm not a big expert on mapstore clients, but I guess are defined here: https://github.com/GeoNode/geonode-mapstore-client/blob/d0d4d0a18f1e253aef30b5010c1b083c490851c9/geonode_mapstore_client/static/mapstore/configs/localConfig.json#L665 @giohappy can be more helpful in this case

[t-book]

Thanks @mattiagiupponi

PS: AFAIR for now mapstore doesn't take care of the extra metadata

I would also say this is not implemented yet.

Anyways, I would move on with your comments on the backend part and then create a PR for the dataset_metadata view. (I first need to fully understand everything the view is responsible for. Looks it does a lot of things ...)

[giohappy]

@t-book The recent improvements to the info panel are headed to have a much more flexible panel. The goals are:

• the ability to define by configuration which fields to show and with which widget
• configure additional tabs with custom fields
• give room to projects to add tabs (along the info panel) and show whatever they want
• and now I would add being able to format extra_metadata

We don't have any plans for this development, but @allyoucanmap can tell you where in MapStore this things are managed.

[t-book]

This sounds more than promissing @giohappy ! @allyoucanmap could you share where the current details information logic is defined in mapstore?

[mwallschlaeger]

Hey @t-book,

I am really looking forward for this changes. I just wonder how this extra metadata get passed to the metadata xml. Have you looked at this part already. Will it be available like {{ layer.extra_metadata.color }}

[afabiani]

@t-book take a look at this example

https://training.geonode.geosolutionsgroup.com/master/GN4/055_project_customize_model.html#update-the-detail-panel-info-box

[allyoucanmap]

This sounds more than promissing @giohappy ! @allyoucanmap could you share where the current details information logic is defined in mapstore?

@t-book they are described in localConfig.json and each resource page could have its own config:

• catalogue page in the detailsTabs prop of the ResourcesGrid plugin
• map viewer in the tabs props of the DetailViewer plugin
• dataset viewer in the tabs props of the DetailViewer plugin
• document viewer in the tabs props of the DetailViewer plugin
• geostory viewer in the tabs props of the DetailViewer plugin
• dashboard viewer in the tabs props of the DetailViewer plugin

The logic for the code is inside the DetailsInfo component and it currently supports these types: link, query, date, html and text.

We started also to fix a bit the doc here splitting it by version:

• https://geonode.org/geonode-mapstore-client/master/
• https://geonode.org/geonode-mapstore-client/4.1.x/

We added some info related to the DatailViewer and ResourcesGrid plugins with js doc:

• DetailViewer Plugin doc
• ResourcesGrid Plugin doc

Please if you think the doc could be improved based on your experience and you can contribute by adding more related to this topic inside the js doc or adding a custom documentation page it could be really helpful, thanks

[allyoucanmap]

@t-book take a look at this example

https://training.geonode.geosolutionsgroup.com/master/GN4/055_project_customize_model.html#update-the-detail-panel-info-box

@afabiani we changed this behaviour in latest version

[t-book]

Thanks a lot @afabiani and @allyoucanmap. I will see if I can show the metadata as explained by Stefano.

---

@mwallschlaeger To be honest the XML output is currently not asked by the client. But I would try to have a look at it as well. With the current implementation the Extrametadata will be an m2m relation that could be added by GUI and the API. This means one will not always now the keys to use like {{ layer.extra_metadata.color }}.

From what I see is that we could add the extrametadata by extending csw_gen_xml:

    def csw_gen_xml(self, layer, template):

        # Load extrametadata
        # Todo 1: only add if SETTINGS Option is set
        # Todo 2: make sure keys are valid to be used as XML keys
        extra_metadata = [json.loads(extra.metadata) for extra in layer.metadata.all()]

        id_pname = 'dc:identifier'
        if self.type == 'deegree':
            id_pname = 'apiso:Identifier'
        site_url = settings.SITEURL.rstrip('/') if settings.SITEURL.startswith('http') else settings.SITEURL
        tpl = get_template(template)
        ctx = {'layer': layer,
               'SITEURL': site_url,
               'id_pname': id_pname,
               'LICENSES_METADATA': getattr(settings,
                                            'LICENSES',
                                            dict()).get('METADATA',
                                                        'never'),
                 # pass extra data as context
                'extra_metadata': extra_metadata}
        md_doc = tpl.render(context=ctx)
        return md_doc

and then one could output all Extrametadata in full_metadata.xml

  {% for extra in extra_metadata %}
      {% for key, value in extra.items %}
      <gmd:{{ key }}>{{ value }}</gmd:{{ key }}>
      {% endfor %}
  {% endfor %}

Surely this needs more work, and maybe some guidance. (I must admit I'm not really into pycsw it's tests and harvesters.) Also the html representation should be updated.

---

@mattiagiupponi

I worked on the backend, my current plan is as following. As shown above I would provide a PR that makes the geonode more flexible regarding extra metadata.

Next to the PR I'm creating a basic contrib app that enrich the GUI and hopefully also the XML and html output as discussed with @mwallschlaeger . For sure the app will not cover all needs but can hopefully act as a starting point for others.

My view decorator acts like a FormFactory that reads a json file with Form definitions:

{
    "fields": [
        {
            "type": "NumberField",
            "name": "age",
            "label": "Age",
            "required": false,
            "widget": "TextInput"
        },
        {
            "type": "CharField",
            "name": "planet",
            "label": "Planet",
            "required": false,
            "widget": "TextInput"
        },

This fields are translated to Django forms and Widgets and the DB json Data is used as values:

(The fields are shown in an extra step by panels.html, but could also extend existing steps)

In case a field has not been defined in the json definition but has been added via REST, the FormFactory respects this field, but as it does not know the correct type, an Input field will be used (could be improved later to read a type key). Also plans exist to make the FormFactory more pluggable so that others could add forms as plugins ...

My questions are:

• What should/could happen if somehow two entries with the same json key exist (added twice by the API)? I would say, the GUI should use the one with the higher ID and replace the second on next save?

[mattiagiupponi]

• What should/could happen if somehow two entries with the same json key exist (added twice by the API)? I would say, the GUI should use the one with the higher ID and replace the second on the next save.

I would see this from a bigger point of view. The same metadata can be possibly applied to different resources so we have also to distinguish which metadata belongs to which resource.

Plus by how the endpoint is made, json with the same key is fully accepted as soon as they pass the validation phase, the same key can have different values. We can evaluate adding a new control where if the metadata for that specific resource is already preset (same key and same values) raises an error so the user knows that the metadata is already available, but with a lot of metadata can be a slowing process

Assuming to take the newest ID may be too much heuristic assumption.

By see your work, I'll go to show all of them without any fear :)