SDMX 3.0: implement "feature 001 Improve the exchange of reference metadata" for SDMX-JSON messages

In order to support the new requirement, it is proposed to:

structure message changes:

Reference Metadata against Structure Metadata

The Provision Agreement now only links to a Dataflow, and a new Object called Metadata Provision is introduced which enables linking to Metadataflows.
Restrict the MSD to a single Report structure, by removing the Report structure container. The Metadata Attributes are now a direct child of the MSD and not children of Report Structure containers.
The Metadata Target is moved it to the Metadataflow/Metadata Provision and is no longer a part of the MSD.
The Metadata Target is simplified, only allowing a references to identifiable structure(s), which may include wildcarded references. The Metadata Provision can also include zero to many Metadata Targets in order to further restrict the targets of the Metadataflow.
The reported MetadataSet includes an absolute reference to the Metadataflow and/or Metadata Provision which it is reported against, and one or more target structures, which must comply with the restrictions imposed by the Metadataflow and/or the Metadata Provision.
The role of the content constraint for a metadata collection is not to restrict targets, but restrict content for each Metadata Attribute.

Reference Metadata against Datasets

The ability for the DSD to optionally describe the relationship between the MSD Metadata Attribute and the Dimensions against which metadata can be reported. Where the absence of this relationship definition is interpreted as the Metadata Attribute being able to be reported against any combination of Dimensions.
The AttributeRelationship provides the ability to make Dimension references optional by using an object with properties "id" and "optional" instead of a simple string with the "id":
```
"attributeRelationship": {
"dimensions": [
    { "id": "REF_AREA", "optional": true}, "INDICATOR"
]
},
```
XHTML is introduced as an allowable data type. The already used XHTMLType allows for mixed content of text and XHTML tags.
Non-enumerated text Attributes (HTML/String) also support multilingual text.

--> Implemented as translation from xml structure schemas by J

data message changes:

Reference Metadata against Structure Metadata

Removing the Report structure reference and the AttributeSet container from the MetadataSet. The Metadata Attributes are now a direct MetadataSet

The reported MetadataSet includes an absolute reference to the Metadataflow and/or Metadata Provision which it is reported against (a simple structure reference!), and one or more target structures (now also simple structure references!), which must comply with the restrictions imposed by the Metadataflow and/or the Metadata Provision.


"metadataSets": [
{
"action": "Information",
"id": "ALB",
"providerID": "ALB",
"version": "1.0.0",
"metadataflowRef": "urn:sdmx:org.sdmx.infomodel.metadatastructure.Metadataflow=OECD:ALB_DQAF(1.0.0)",
"name": "Metadata set",
"names": {
    "en": "Metadata set",
    "fr": "Set de métadonnées"
},
"target": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=OECD:GDP(1.0.0)",
"attributes": [{
    "id": "ATTRIBUTE_1_1",
    "value": "CODED_TEXT",
    "attributes": [{
        "id": "ATTRIBUTE_1_2",
        "structuredText": "<p>An XHTML text</p>",
        "structuredTexts": {
            "en": "<p>An XHTML text</p>"
        },
        "text": "A text",
        "texts": {
            "en": "A text"
        }
    }]
}]
}]

"metadataSets": [ { "action": "Information", "id": "ALB", "providerID": "ALB", "version": "1.0.0", "metadataProvisionRef": "urn:sdmx:org.sdmx.infomodel.metadataProvision=OECD:ALB_DQAF(1.0.0)", "name": "Metadata set", "names": { "en": "Metadata set", "fr": "Set de métadonnées" }, "target": "urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=OECD:GDP(1.0.0)", "attributes": [{ "id": "ATTRIBUTE_1_1", "value": "CODED_TEXT", "attributes": [{ "id": "ATTRIBUTE_1_2", "structuredText": "

An XHTML text

", "structuredTexts": { "en": "

An XHTML text

" }, "text": "A text", "texts": { "en": "A text" } }] }] }]

- The current [GenericMetadata message](https://github.com/sdmx-twg/sdmx-json/tree/develop/metadata-message) is updated for SDMX3.0.

#### Reference Metadata against Datasets
- Values of **MSD Metadata Attributes** referenced by DSD are allowed to be transmitted with or without the data (and so allow including only relevant dimensions)
- **For data messages we extend this possibility to transmit any attribute values with or without the data (thus separately).** 
```json
"attributes": {
    "group": [          # a new property for attributes attached at partial keys (in addition to dataset, series and observation)
        {
            "id": "GROUP_ATTR",
            "name": "Group attribute",
            "names": { "en": "Group attribute"},
            "values": [ # values to be used if attribute is enumerated
                {
                    "id": "AAA",   
                    "name": "Aaa",
                    "names": {"en": "Aaa", "fr": "Aaa"}
                }
            ]
        }
    ]
}

"dataSets": [
    {
        "action": "Information",
        "groupAttributes": {
            ":0::": [0] # a subset of dimensions is sufficient for the new `groupAttributes` object containing the attributes attached at partial keys, however separators ':' are required between all dimensions included in the message, indexes are used when attributes are enumerated.
        }
    }
]

XHTML is introduced as an allowable data type.

Non-enumerated text attributes (HTML/String) also support multilingual text.


"attributes": {
"group": [
    {
        "id": "LOCALISED_HTML_ATTR",
        "name": "Attribute with multi-lingual HTML",
        "names": { "en": "Attribute with multi-lingual HTML"},
        "format": { # to be used for any format specifications
            "textFormat": "xhtml",  # textFormat to be used when not enumerations
            "isMultilingual": true  
        }       # values to be used if enumerated
    }
]
}

"dataSets": [ { "action": "Information", "groupAttributes": { ":0::": [{"en": "

Lorem Ipsum

", "fr": "

Lorem Ipsum

"}] # Values are written when attributes are not enumerated. This applies to all attributes including those presented at dataset, series or observation level. } } ]


- Any attribute type also supports multiple values (from "feature 003 Microdata exchange").
```json
"attributes": {
    "group": [
        {
            "id": "MULTI_VALUED_ATTR",
            "name": "Attribute with multiple values",
            "names": { "en": "Attribute with multiple values"},
            "format": { # to be used for any format specifications
                "textFormat": "string", # textFormat to be used when not enumerations
                "isMultilingual": false,
                "maxOccurrences": 2  
            }       # values to be used if enumerated
        }
    ]
}

"dataSets": [
    {
        "action": "Information",
        "groupAttributes": {
            ":0::": [["text value 1", "text value 2"]]  # Attribute values are written directly into the dataSets when not enumerated. Multi-valued attributes are presented as an array.
        }
    }
]

"attributes": {
    "group": [
        {
            "id": "MULTI_VALUED_ATTR",
            "name": "Attribute with multiple multi-lingual values",
            "names": { "en": "Attribute with multiple multi-lingual values"},
            "format": { # to be used for any format specifications
                "textFormat": "string", # textFormat to be used when not enumerations
                "isMultilingual": true,
                "maxOccurrences": 2  
            }       # values to be used if enumerated
        }
    ]
}

"dataSets": [
    {
        "action": "Information",
        "groupAttributes": {
            ":0::": [[{"en": "Value1", "fr": "Valeur1"}, {"en": "Value2", "fr": "Valeur2"}]]    # Multi-valued attributes are presented as an array. Inside each array field (value) is an object allowing for multi-lingual value versions.
        }
    }
]

sdmx-twg / sdmx-json