Open amercader opened 8 years ago
I am using ckanext-scheming
to define schemas for special dataset types.
Link to example schema: https://github.com/ckan/ckanext-scheming/blob/master/ckanext/scheming/camel_photos.json
I would like to have the metadata of these data types automatically represented in DCAT via this extension. I'm pretty sure that is the scope described by this issue, but just clarifying.
Currently, when I append .jsonld or .rdf to the dataset it displays only the normal metadata and ignores the other fields. Do those field need to have a namespace defined for them somehow?
Is there a recommended way to achieve this functionality?
I think this is what I've been looking for (and hoping someone else would have done). When I first saw that this extension was compliant with dcat-ap, I assumed that it meant that it would automagically create the relevant fields (with the required
label for properties that are mandatory in the AP) - but it seems like I was a bit optimistic!
Does one already exist? If not, it looks like I'll need to create one - if it's helpful to add to a repo of common schemas (that may or may not already exist) I'd be happy to add it when I've done it
@wood-chris there is a scheming mapping for DCAT-AP Switzerland (which is a slightly adapted version of DCAT-AP EU) here: https://github.com/opendata-swiss/ckanext-switzerland/blob/master/ckanext/switzerland/dcat-ap-switzerland_scheming.json
Dumping some thoughts here on scheming support.
At the time the processors (parsers/serializers) that map between CKAN and DCAT were written, usage of ckanext-scheming was still starting to become widespread, so custom DCAT fields that didn't link directly to standard CKAN fields were stored as extras (see all the ones marked extra:
here). So the DCAT version_notes
field would be stored as:
{
"name": "test_dataset_dcat",
"title": "Test dataset DCAT",
// ....
"extras": [
{"key": "version_notes", "value": "Some version notes"}
// ....
]
}
The pattern nowadays is to create custom fields in a scheming schema, that internally handles the conversion to / from extras:
{
"name": "test_dataset_dcat",
"title": "Test dataset DCAT",
"version_notes": "Some version notes"
// ....
}
That's the goal of creating a DCAT scheming schema, that all properties are custom fields of the CKAN Dataset, aligning with the dcat:Dataset ones.
The difficulty here is how to offer support for existing sites using previous versions of the extension.
version_notes
) will check first for a root level field in the dataset_dict and if it's not there, look for an extra with that key (here)extras
, which will make the resulting dataset dict incompatible with a scheming schema that defines the field as a dataset field. When creating or updating, we will get the following error: File "/home/adria/dev/pyenvs/ckan-py3.9/ckan/ckan/logic/action/create.py", line 185, in package_create
raise ValidationError(errors)
ckan.logic.ValidationError: None - {'extras': [{'key': ['There is a schema field with the same name']}]}
The second case is relevant to CKAN sites that import DCAT RDF representations from other systems (I'd imagine that in most cases through the DCAT harvester) and create CKAN datasets from them.
My current thinking on how to approach this is:
[1] At first I thought about being clever and inspecting the dataset schema (if scheming was being used) to see if there were DCAT fields defined, and not store the values in extras if so but that seemed brittle, and sites could have different schemas in used, potentially for different DCAT versions even. I think it's better to be explicit and make site maintainers
Sorry if this is a bit convoluted, here's a TLDR:
In the next major version of ckanext-dcat I want to change the RDF DCAT Parsers so they store custom DCAT fields as first level CKAN dataset fields instead of dataset extras, but keep the old behviour via config option for backwards compatibility
Chatting with @wardi about this, actually using scheming_dataset_schema_show
to check if there is a schema that contains DCAT fields could be a really good approach, as besides detecting if values need to be stored at the root level, we could use it to mark DCAT fields with certain keys in the schema (like dcat_validators
, etc). These are available to the output of scheming_dataset_schema_show
, snippets, validators etc.
To the point of knowing what schema to use a good approach might be:
dataset
schemaPR with summary of work done so far here: https://github.com/ckan/ckanext-dcat/pull/281
We could provide one or two predefined schemas covering all DCAT and DCAT-AP fields ready to be used or customized by instance maintainers.
This will also help in handling multilingual metadata (#55)
We could have:
What we need: