Move metadata to custom entity

[dafeder]

Metadata is currently stored in a single content type, data, and differentiated with a text field, "type". This was a convenient way to get the system up and running but presents some problems:

1. Metadata shows up in the Drupal content administration screen, including child data objects like "distributions" and "publishers".
2. All nodes receive a route in Drupal, so to keep all the component pieces of a single dataset published or unpublished according to their parent's status will require some kind of syncing function, which does not yet exist
3. We don't currently have a straightforward way to manage the core schemas if you want to break from what we ship with
4. On a purely conceptual level, we have essentially created a concept of "bundles" within data entities but implemented them in a nonstandard way, making the system less intuitive for Drupal developers

Proposal is that we refactor the Metastore module to implement a new entity.

Metadata storage architecture

• All pieces of metadata will live in a new entity with the machine name of metadata and the label "Metadata Item" (should it be "Metastore Item"?)
  • All Metadata Items will have at minimum a json_data field, along with basic properties like created, modified, owner and uuid
  • Note that there will be no title property; the getLabel() method will check the bundle configuration, and if no criteria is found, look for a "name", "title" or "label" property in the schema. As a last resort it will return a string "[schema name] [uuid]"
• The Metadata entity will have multiple bundles, with the machine name of schema and the label "Metadata Schema". These schema definitions will include:
  • A JSON Schema document. We should discuss if this should be an arbitrary path/URL, or a selection of schemas registered elsewhere, or even a longtext field stored directly in the bundle configuration.
  • An optional UI Schema based on the UI Schema for react-jsonschema-form
  • Properties to mark a schema as a predefined core DKAN object type (more below)
  • The schema field to use for the getLabel() method

While this will result in a truly flexible metadata architecture, we have some core concepts in the DKAN codebase that need to be represented somewhere. Schemas will need to have the capability to be classified as one of the following types:

• Dataset: This means the schema lives at the top level of the catalog. Any entities from a dataset bundle will recieve a route and generally be treated as a first-class citizen. DKAN will ship with a bundle called "dataset" that serves this purpose, but this can be replaced with any other bundle. The system must have at least one dataset bundle to function. We may support one and only one bundle upon initial refactoring, but plan to eventually support multiple dataset bundles on a single system. Datasets can be created manually *
• Vocabulary A vocabulary schema is used to provide multiple-choice options for an enum field in another schema. Items can be created manually *
• Resource A resource is a container for a specific link or file. This may become obsolete in the future, but currently a lot of DKAN workflow presumes that any pointers to specific data resources are wrapped in their own child object referenced from the parent dataset.

Schemas not identified as one of these core types are simply available for storage with no special behaviors associated with them.

* Schemas that are designated for "manual creation" can be created by a user action -- clicking "create new [bundle name]" or posting to an endpoint. Schemas not available for manual creation are created by the DKAN referencing system and may only be created or modified through their parent entities.

Metadata admin UI

This proposal assumes that the change to Drupal Form API described in #3166 have been implemented.

The metadata edit form will need to distinguish between child entities that have a cardinality of 1 (such as publisher) and child entities that allow for an unlimited array of objects (primarily in the case of distribution). The parent schema and ui schema should be able provide enough context to determine this without adding additional properties to the schema bundle definition.

In the case of any schema field that allows for unlimited complex child objects, we can decide between inserting sub-forms with AJAX, or adding an "Add [schema bundle name]" button to datasets for a multistep/"wizard" experience.

Upgrade path

This would of course represent at least a new minor version and would require an easy upgrade path to migrate "Data" nodes into Metadata items with the proper schema bundles.

[fmizzell]

@dafeder Regarding issue 1, this is not a bug, it is the reason for the referencing system. If one does not expect metadata pieces to need to be managed as individual things (for example updating a publisher's info), then it should not be chosen as a referenceable object and it should never show up as its own entity. So effectively, the use case of having referenced entities that need to be hidden should not exist.

[fmizzell]

@dafeder How does the new approach of using a custom entity solves issue 2?

[kimwdavidson]

This proposal has the thumbs up to move forward, but needs to be ticketed

[dafeder]

Work being organized in https://github.com/GetDKAN/dkan/projects/4