Open augusto-herrmann opened 4 years ago
@augusto-herrmann sure - do you want to prep one? We generally would like a "json-ld" context for data packages https://github.com/frictionlessdata/specs/issues/218
@rufuspollock I did search the repository for JSON-LD, but somehow I missed those previous threads. Sorry. :flushed:
Anyway, for this issue, what I had in mind is smaller in scope: just mapping the data package metadata to DCAT2 classes and properties. DCAT2, as a W3C recommendation, is very recent – less than a month old. JSON-LD would be just a tool for achieving that goal.
Why is that important? Because it would make easier for data catalogs (such as CKAN, with the DCAT plugin) to automate importing data packages as datasets.
I am also well aware of the previous failed attempts trying to reconcile data packages and the W3C's CSVW and I'm not trying to delve into that at the moment. That and fully converting the data to linked data is a much harder problem that should be considered out of scope to what I'm proposing here.
I can try to sketch something, but I wanted to first make sure that it made sense, so as not to be a wasted effort.
@augusto-herrmann 👍 on doing the mapping - go for it.
@augusto-herrmann any update here 😄 ?
I don't have a lot of time to devote to this right now, but here are some steps I thought about how to undertake this:
datapackage.json
file, make it also valid JSON-LD that generates triples in DCAT2 format. That way, a single JSON file could comply with both standards.@augusto-herrmann we now have a spreadsheet via frictionlessdata/forum#11 and i've added a sheet for DCAT 2 https://docs.google.com/spreadsheets/d/1XdqGTFni5Jfs8AMbcbfsP7m11h9mOHS0eDtUZtqGVSg/edit#gid=729988073 - would you be up for adding the list of DCAT 2 attributes there?
Sure. I'm working on it.
@augusto-herrmann how is this going?
I remember doing some of that work last year, but I need to verify at which point I left it (I do not remember anymore) and resume it when I have some time.
Here is how much of it is done.
Classes:
Though I expect that most of the latter ones either don't map to Frictionless or are less relevant.
Hi,
Thanks for all of the existing work on mappings, I'd like to help progress this and map out what's left. @augusto-herrmann are you continuing to work on this? Initially I plan to focus solely on mapping the Table Schema as that format fits the majority of the datasets we [in the project described below] will be working with.
I appreciate some of what's covered in this post might be slightly out of scope for this issue, let me know if it would be more useful to create a new one.
I'm working on a project where the metadata of the datasets will be expressed in RDF, we're looking at using the Frictionless Data specifications and wider tools as a way to reduce the friction for users [of the climate data hub we're working on] to generate the RDF metadata. Specifically, one user story would be using data package creator to generate a datapackage.json, we'll create a Python library that can map from the datapackage.json to an RDF representation - though the use of a JSON-LD context could enable the datapackage.json to be understood as RDF (removing the need for a secondary mapping step).
Reasons we ideally want to have an RDF metadata representation:
Key components needed to enable this:
I thought it might be handy to summarise some of the existing discussions relating to using RDF for Frictionless Data packages
Pint
which would enable automated unit handling when carrying out analysis, example shown herePint
and QUDT, e.g. Pint
has units such as dtype="pint[lbf ft]" whereas QUDT has the name FT-LB_F for the same unitrdfType
in favour of generalising the model
to enable expression of semantic links to RDF conceptsrdfType
usefulWow, @AyrtonB, this seems like an amazing project! 😀 Also you've made an awesome summary there of the discussions surrounding Frictionless and RDF. Good job!
A month ago I progressed a little bit more on the mapping, but I was taking this slowly, because I'm always busy with lots of other things. If you've got a dedicated project to take this forward, it makes total sense that you'd take it over from here. The checklist above pretty much marks the point where I left off. So you could continue the mapping by reviewing the parts I've already done and continue what is still left to do.
Hi @AyrtonB! This looks awesome! I think you've done a great job summarizing the work that needs to be done and the current situation. I'm on the Frictionless team (with @roll) and would be happy to support you if you have questions or need help. Communicating on github works very well for us, but if you want to have a call to chat let me know :-)
Thanks. That's handy to know @augusto-herrmann, sounds good I'll work from that.
@lwinfree that would be really helpful, thank you. I think for this specific issue the next steps are pretty clear and I'll try to keep the convo in this thread atm. For the project I'm working on we're also looking at creating a custom schema for 'data dictionaries' that can act as a central link between different datasets - e.g. where for the original datasets you only have to specify one foreignKey that maps to the primaryKey in a data dictionary, removing duplication and ensuring only one central dataset needs to be updated when a new dataset is mapped in. It would be really useful to have a call around the implementation of this use-case if possible :)
Sounds great @AyrtonB!
I've made some progress towards creating a parser for generation an RDF representation of the table schema datapackage.json. Currently I have a Python script that includes a tableSchema
object, this generates a graph representation of the table schema using rdflib
when passed the url for a datapackage. I've documented the code generation and application in this notebook. An example RDF representation can be found here, the original input was this Open Power Systems Data datapackage.json.
This is very much a pilot but has been useful in terms of working out which terms in different ontologies could be useful, as well as identifying how best to approach the parsing with Python. Next I plan to create an ontology for the frictionless data spec that includes a tableSchema object (building on existing specs), I'll then use the concepts described in it and refactor the Python code.
@context
would enable the datapackage.json
files to be converted into JSON-LDdatapackage.json
which would likely not be ideal for the FD specificationdatapackage.json
to an RDF representationjinja2
templates for raw RDF.xmlrdflib
, then exporting the final graph to an RDF.xml fileCurrently I have a Python script that includes a
tableSchema
object, this generates a graph representation of the table schema usingrdflib
when passed the url for a datapackage. I've documented the code generation and application in this notebook.
That is a cool experiment! Have you finished already the mapping I started on the Google Spreadsheet linked above? Because I think we should only start developing practical implementations once the mapping of concepts between Tabular Data Packages and DCAT2 is completely finished. Otherwise we may end up doing unnecessary repetition.
- The only way to do that seems to involve changing the structure of the
datapackage.json
which would likely not be ideal for the FD specification
Why would creating a JSON-LD @context
for frictionless require changing the structure of datapackage.json
? Could you please elaborate that?
- How best to handle blank nodes? e.g. for each resource/dataset
Perhaps data packages and resources should not be blank nodes, but URI identified instead. There could be a determined way to create those URIs, appending hashes at the end, something like #datapackage
for packages/datasets and #resource-name
for resources.
- Do we need both dataset and distribution? - Currently all attributes are tied to the distribution and dataset is only used as a link to the catalog
DCAT2 makes a distinction between dataset and distribution in the way that the former is more abstract and the latter is a concrete representation or serialization of the same data. While this makes philosophical sense, from a pragmatic point of view there is no use doing that separation. In Frictionless, the different representations of the same data are just different resources in the same data package.
It is not an exact match, but maybe the dcat:Dataset
should be mapped to f11d:Package
and dcat:Distribution
to f11d:Resource
.
- What previous issues existed with CSVW? - Looks ideal for describing fields but previous comments have alluded to incompatibility
I do not remember exactly, but I think the main problem is that CSVW makes using RDF mandatory and many in the community would rather make it optional.
- Should the table-schema data-structure be described in OWL?
I think the initial scope of this issue was just mapping data packages in a general sense, which may not even be tabular. Table schema could be a possible next step after this is done.
- Does coverage need to be comprehensive or would it work to start from a subset?
I think this mapping should enable conversion of metadata from the basic Data Package -> DCAT2 and DCAT2 -> Data Package with as little information loss as possible. Except, for the moment, for profiles, like table schema, of course.
- What's are the semantic differences between: name, title, description, and longDescription
The differences between name
, title
and description
are all laid out in the specification. I checked but could not find a longDescription
attribute, neither in the Data Package specification, nor in DCAT2. Where did you find it?
Hi @AyrtonB and @augusto-herrmann! I'm working on cleaning up issues in this repo and wanted to touch base with you two about this. Thanks for all the work you've done so far! Are you still interested in working on this? There is no pressure from me, and no time crunch, I'm just inquiring about the status :-) Hope you are both doing OK!
Hi @lwinfree, thanks for bringing this up. Yes, I am still interested in this issue, but don't have much time do dedicate to solve it – last I checked, some effort was required to finish the mapping between the f11d specs and DCAT2 classes and properties in RDF. I'm curious to find out how far @AyrtonB has been able to progress with it, or in more detail the questions I asked above, if possible.
Hi people,
I have implemented an initial version of DCAT mapper (highly based on ckanext-dcat
) for the new dplib-py
library (lightweight data package models). If you are interested in contributing, please take a look at https://github.com/frictionlessdata/dplib-py/blob/main/dplib/plugins/dcat/models/package.py (and resource.py
) -- it uses a very simple Pydantic-based model framework so it's quite easy to do metadata mappings in fully-typed manner
PS.
Note that unlike ckan
and others dcat
mapper has two layers mapping rfd -> dcat model -> dp model
. Actually, I think it will be great if it's possible to extract something like dcatlib
and re-use in both dplib-py
and ckanext-dcat
The W3C has just published as a recommendation the Data Catalog Vocabulary – DCAT, version 2. Should there be a mapping from the Data Package specification to DCAT2?
If so, I think it would be possible to implement with a JSON-LD context.
WDYT, @lwinfree, @rufuspollock, @roll ?