lilybradley
commented
10 years ago Allow an array for different schema urls, if arrays are allowed. The benefit of allowing an array, is that we add additional schema fields.
Closed philipashlock closed 9 years ago
lilybradley
commented
10 years ago Allow an array for different schema urls, if arrays are allowed. The benefit of allowing an array, is that we add additional schema fields.
smrgeoinfo
commented
10 years ago so the catalog might include entries that use different schema? Does each record identify which schema it uses?
gbinal
commented
10 years ago If versioning transitioned from have the first record reflect the data.json file itself to a set of file metadata at the top level of the json file, there's a question of what the format should look like and any other optional metadata, such as:
We'd also need to document pretty well what this looks like and how agencies would migrate to it.
philipashlock
commented
10 years ago One proposal is to use the same property names used for this purpose in JSON Schema ($schema and $ref) and include all the dataset listings as part of an array of objects in an overarching catalog object like the Catalog domain in DCAT (#350)
In this context $schema would refer to the canonical URL for the current version of the POD schema, but $ref would be a URL for an actual JSON schema file defining the schema. By default $ref could refer to the canonical JSON schema file hosted on the POD website, but if an agency wanted to extend the schema with some of their own data model they could refer to their own JSON schema file here. The $schema URL would always be required and would always refer to the URL identifying the base schema and its version.
Here's an excerpt of this in a data.json file, but you can see a full example in this gist
{
"$schema": "http://project-open-data.github.io/v1.1/schema",
"$ref": "http://project-open-data.github.io/v1.1/schema/catalog.json",
"dataset": [...]
}@jpmckinney any feedback on this as opposed to JSON-LD? https://github.com/project-open-data/project-open-data.github.io/issues/23#issuecomment-17978611
At the very least it seems like we've been making progress on DCAT compatibility so that it should a little easier to provide a JSON-LD serialized variant
jpmckinney
commented
10 years ago You can do both. Popolo is an RDF vocabulary, and it has both JSON Schema and JSON-LD contexts available: for example, Person schema and Person context.
I haven't see $ref as a top-level field. Where is that defined? I've only seen it used to refer to sub-schema, e.g. in Popolo, the field for the sources of information about a Person refers to the Link schema.
philipashlock
commented
10 years ago @jpmckinney Perhaps this proposal isn't using $schema and $ref in exactly the same way they're used in JSON Schema. It might make more sense to use id instead of $ref but one thing that's nice about using "$" is it can help ensure these properties are output before dataset in instances where JSON is generated such that properties are ordered alphabetically in the output. This becomes more relevant when you want to inspect this at the beginning of a 100MB JSON file.
I saw there were some somewhat related questions (1, 2) about this on the JSON Schema mailing list but I wasn't sure if there was more recent thinking or if hyper-schema addresses this in a way that would be relevant, so I just brought it up again on the JSON Schema mailing list.
jpmckinney
commented
10 years ago On second thought, I think the issue is that JSON Schema doesn't specify any way for an instance to refer to its schema, and that $schema and $ref are reserved for use in JSON Schema files. Perhaps invent a new _schema property to point to the JSON Schema?
philipashlock
commented
10 years ago @jpmckinney looks like you can do it with HTTP headers using rel="describedBy" (see the draft) so perhaps we use describedBy in place of $ref and use the Dublin Core term conformsTo in place of $schema
jpmckinney
commented
10 years ago Sounds good, though I'm not clear on the difference between $ref and $schema (or describedBy and conformsTo).
philipashlock
commented
10 years ago One is meant to be a URI that identifies the standard (conformsTo) and the other (describedBy) is specifically meant to be a URL that points to a JSON Schema. We would require conformsTo while describedBy would be optional, though we would recommend that describedBy point to the canonical JSON Schema file on the Project Open Data Site. However if a publisher extended the canonical schema they could point to their own JSON Schema defining that.
Here's an updated example:
{
"conformsTo": "http://projectopendata.placeholder.gov/v1.1/schema",
"describedBy": "http://projectopendata.placeholder.gov/v1.1/schema/catalog.json",
"dataset": [...]
}To respond to @BernHyland's recommendation to use JSON-LD to accommodate this, which I too had questioned, which @jpmckinney responded saying both approaches were possible, and whereas there has already been long discussions about whether/how to incorporate linked data, and partial examples of JSON-LD, I wanted to at least see what a minimal JSON-LD representation of the current proposal would be to better inform that discussion. Here is a first attempt at trying to JSON-LD-ify (to the simplest extent possible) the JSON example of the current schema. This may very well be an imperfect serialization so I'm putting it here both to get feedback on what a minimally viable JSON-LD serialization would be and to get feedback on how this compares to a JSON serialization of the schema without JSON-LD requirements.
I don't want this particular issue to turn into a broader debate about linked data as #21 did, but we can create a new issue for that as needed. I'd rather this focus on how JSON-LD can help address the specific need raised in this issue. Also note that data.gov, which aggregates this Project Open Data metadata from across the Federal government, already offers linked data in the form of the RDF XML serialization of DCAT provided by CKAN as well as the Schema.org microdata/RDFa mapping of DCAT.
cc: @gkellogg @amercader @philarcher1
{
"@context": {
"dcat": "http://www.w3.org/ns/dcat",
"org": "http://www.w3.org/ns/org",
"vcard": "http://www.w3.org/2006/vcard/ns"
},
"@id": "http://www.agency.gov/data.json",
"@type": "dcat:Catalog",
"conformsTo": "https://project-open-data.cio.gov/v1.1/schema",
"describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json",
"dataset": [
{
"@type": "dcat:Dataset",
"accessLevel": "public",
"accrualPeriodicity": "R/P1Y",
"bureauCode": [
"018:10"
],
"conformsTo": "http://www.agency.gov/widget-taxonomy/",
"contactPoint": {
"@type": "vcard:Contact",
"fn": "Jane Doe",
"hasEmail": "mailto:jane.doe@agency.gov"
},
"describedBy": "http://www.agency.gov/datasets/widgets-dictionary.html",
"dataQuality": true,
"description": "This dataset provides national statistics on the production of widgets",
"distribution": [
{
"@type": "dcat:Distribution",
"description": "Widgets data as a CSV file",
"downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets.csv",
"format": "CSV",
"mediaType": "text/csv",
"title": "widgets.csv"
},
{
"@type": "dcat:Distribution",
"description": "Widgets data as a zipped CSV file with attached data dictionary",
"downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets-all.zip",
"format": "Zipped CSV",
"mediaType": "application/zip",
"title": "widgets-all.zip"
},
{
"@type": "dcat:Distribution",
"conformsTo": "http://www.agency.gov/widget-data-standard/",
"describedBy": "http://www.agency.gov/widgets/schema.json",
"describedByType": "application/schema+json",
"description": "Widget data as a JSON feed",
"downloadURL": "http://www.agency.gov/feeds/widgets-all.json",
"format": "JSON",
"mediaType": "application/json",
"title": "widgets-all.json"
},
{
"@type": "dcat:Distribution",
"accessURL": "https://data.agency.gov/api/widgets-statistics/",
"description": "A fully queryable REST API with JSON and XML output",
"format": "API",
"title": "Widgets REST API"
}
],
"identifier": "widgets-0001",
"issued": "2011-11-22",
"keyword": [
"widget",
"manufacturing",
"factory"
],
"landingPage": "http://agency.gov/widgets/data",
"language": [
"en-US"
],
"license": "http://creativecommons.org/publicdomain/zero/1.0/",
"modified": "2011-11-19T12:00:00Z",
"primaryITInvestmentUII": "021-006227212",
"programCode": [
"018:001"
],
"publisher": {
"@type": "org:Organization",
"name": "Widget Services",
"subOrganizationOf": {
"@type": "org:Organization",
"name": "Office of Citizen Services and Innovative Technologies",
"subOrganizationOf": {
"@type": "org:Organization",
"name": "General Services Administration",
"subOrganizationOf": {
"@type": "org:Organization",
"name": "U.S. Government"
}
}
}
},
"references": [
"http://agency.gov/docs/widgets-1.html",
"http://agency.gov/docs/widgets-2.html"
],
"rights": "This dataset has been given an international public domain dedication for worldwide reuse",
"spatial": "United States",
"systemOfRecords": "http://www.agency.gov/widgets/sorn/",
"temporal": "2009-09-01T12:00:00Z/2010-05-31T12:00:00Z",
"theme": [
"manufacturing"
],
"title": "U.S. Widget Manufacturing Statistics"
}
]
}
philarcher
commented
9 years ago Thank you for the name check, Philip.
As it happens I'll see Gregg Kellogg this week (there's a big W3C get together in Santa Clara) so I'll talk this through with him. Initial reaction - this looks very promising. Really glad to see convergence and use of the Web as a data platform, not just a transport mechanism.
Phil.
On 24/10/2014 22:16, Philip Ashlock wrote:
To respond to @BernHyland's recommendation to use JSON-LD to accommodate this, which I too had questioned, which @jpmckinney responded saying both approaches were possible, and whereas there has already been long discussions about whether/how to incorporate linked data, and partial examples of JSON-LD, I wanted to at least see what a minimal JSON-LD representation of the current proposal would be to better inform that discussion. Here is a first attempt at trying to JSON-LD-ify (to the simplest extent possible) th e gist example of the current schema. This may very well be an imperfect serialization so I'm putting it here both to get feedback on what a minimally viable JSON-LD serialization would be and to get feedback on how this compares to the schema without JSON-LD requirements.
I don't want this particular issue to turn into a broader debate about linked data as #21 did, but we can create a new issue for that as needed. I'd rather this focus on how JSON-LD can help address the specific need raised in this issue. Also note that data.gov, which aggregates this Project Open Data metadata from across the Federal government, already offers linked data in the form of the RDF XML serialization of DCAT provided by CKAN as well as the Schema.org microdata/RDFa mapping of DCAT.
cc: @gkellogg @amercader @philarcher1
{ "@context": { "dcat": "http://www.w3.org/ns/dcat", "org": "http://www.w3.org/ns/org", "vcard": "http://www.w3.org/2006/vcard/ns" }, "@id": "http://www.agency.gov/data.json", "@type": "dcat:Catalog", "conformsTo": "https://project-open-data.cio.gov/v1.1/schema", "describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json", "dataset": [ { "@type": "dcat:Dataset", "accessLevel": "public", "accrualPeriodicity": "R/P1Y", "bureauCode": [ "018:10" ], "conformsTo": "http://www.agency.gov/widget-taxonomy/", "contactPoint": { "@type": "vcard:Contact", "fn": "Jane Doe", "hasEmail": "mailto:jane.doe@agency.gov" }, "describedBy": "http://www.agency.gov/datasets/widgets-dictionary.html", "dataQuality": true, "description": "This dataset provides national statistics on the production of widgets", "distribution": [ { "@type": "dcat:Distribution", "description": "Widgets data as a CSV file", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets.csv", "format": "CSV", "mediaType": "text/csv", "title": "widgets.csv" }, { "@type": "dcat:Distribution", "description": "Widgets data as a zipped CSV file with attached data dictionary", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets-all.zip", "format": "Zipped CSV", "mediaType": "application/zip", "title": "widgets-all.zip" }, { "@type": "dcat:Distribution", "conformsTo": "http://www.agency.gov/widget-data-standard/", "describedBy": "http://www.agency.gov/widgets/schema.json", "describedByType": "application/schema+json", "description": "Widget data as a JSON feed", "downloadURL": "http://www.agency.gov/feeds/widgets-all.json", "format": "JSON", "mediaType": "application/json", "title": "widgets-all.json" }, { "@type": "dcat:Distribution", "accessURL": "https://data.agency.gov/api/widgets-statistics/", "description": "A fully queryable REST API with JSON and XML output", "format": "API", "title": "Widgets REST API" } ], "identifier": "widgets-0001", "issued": "2011-11-22", "keyword": [ "widget", "manufacturing", "factory" ], "landingPage": "http://agency.gov/widgets/data", "language": [ "en-US" ], "license": "http://creativecommons.org/publicdomain/zero/1.0/", "modified": "2011-11-19T12:00:00Z", "primaryITInvestmentUII": "021-006227212", "programCode": [ "018:001" ], "publisher": { "@type": "org:Organization", "name": "Widget Services", "subOrganizationOf": { "@type": "org:Organization", "name": "Office of Citizen Services and Innovative Technologies", "subOrganizationOf": { "@type": "org:Organization", "name": "General Services Administration", "subOrganizationOf": { "@type": "org:Organization", "name": "U.S. Government" } } } }, "references": [ "http://agency.gov/docs/widgets-1.html", "http://agency.gov/docs/widgets-2.html" ], "rights": "This dataset has been given an international public domain dedication for worldwide reuse", "spatial": "United States", "systemOfRecords": "http://www.agency.gov/widgets/sorn/", "temporal": "2009-09-01T12:00:00Z/2010-05-31T12:00:00Z", "theme": [ "manufacturing" ], "title": "U.S. Widget Manufacturing Statistics" } ] }
Reply to this email directly or view it on GitHub: https://github.com/project-open-data/project-open-data.github.io/issues/309#issuecomment-60450525
Phil Archer W3C Data Activity Lead http://www.w3.org/2013/data/
http://philarcher.org +44 (0)7887 767755 @philarcher1
amercader
commented
9 years ago @philipashlock Thanks for the pointer to this.
I'm not familiar with the full context of this discussion (I'll try and check the links) but to me the JSON-LD approach if what is needed is bridging the convenience of a plain JSON based metadata shcema and the benefits of Linked Data.
In the context of CKAN, we've been recently working on expanding support for importing RDF serializations based on DCAT, but we've focused on more traditional formats like RDF/XML and Turtle. I'm really interested to see how well we can support JSON-LD as well, both for ingesting it and producing it so this is hugely helpful. I'm by no means a JSON-LD expert but at a first glance I think that the @context would need to be expanded to describe all fields (or all fields that want to be understood in a LD context). Contexts can be defined in many different ways, eg you may need a @context for each entity type (Catalog, Dataset, Distribution, Organization, etc). Good news is that these don't need to be embedded on the metadata, but can be linked to a permanent location, eg
{
"@context" : "http://data.gov/contexts/catalog.jsonld"
"@type": "Catalog",
"title": "Some Catalog",
"datasets": [
"@context" : "http://data.gov/contexts/dataset.jsonld",
"@type": "Dataset",
//...
]
//...
}http://data.gov/contexts/catalog.jsonld being:
{
"Catalog": "http://www.w3.org/ns/dcat#Catalog",
"dataset": "http://www.w3.org/ns/dcat#dataset",
"title": "http://purl.org/dc/terms/title",
"description": "http://purl.org/dc/terms/description",
//...
}I'll try to come up with a full example soon, as this is something I wanted to add as part of the work in progress on ckanext-dcat
Also note that data.gov, which aggregates this Project Open Data metadata from across the Federal government, already offers linked data in the form of the RDF XML serialization of DCAT provided by CKAN as well as the Schema.org microdata/RDFa mapping of DCAT
Here is some further CKAN specific discussion about how these could be improved.
https://github.com/ckan/ckan/issues/1890 https://github.com/ckan/ckan/pull/348
I'll feed back any news on the DCAT/RDF front in CKAN if relevant to this thread.
waldoj
commented
9 years ago @philipashlock, I'm confused by one aspect of this—it seems like the labels are reversed from the logical choices. It seems like describedBy would be the description of the schema, and conformsTo would the actual schema that it conforms to. But if I follow, conformsTo is the description of the schema, and describedBy is the actual schema. Do I have that right?
philipashlock
commented
9 years ago @waldoj perhaps my choices are based more on precedents for using these terms than what you would expect by seeing them here for the first time, but I think there's also a subtle distinction between what we mean when we refer to the Project Open Data schema and the schema as represented with JSON Schema.
First on precedents for the terms.
The precedent for describedBy as used with JSON Schema:
When using the "Link" header, the relation type used MUST be "describedBy", as defined by RFC 5988, section 5.3 [RFC5988]. The target URI of the "Link" header MUST be a valid JSON Schema.
The precedent for conformsTo as defined in Dublin Core:
An established standard to which the described resource conforms.
As far as the distinction between the Project Open Data schema and the schema as represented with JSON Schema:
The schema is something that could be described with other formats other than JSON Schema or the main website where it's documented, so instead of referring to it by a particular serialization, we're using the main URL as a unique identifier for it. The JSON Schema file isn't "the schema" or "the spec" it's just one way of describing it in a machine readable way. In other words, conformsTo is a meant to reference a more abstract concept whereas describedBy is meant to refer to a specific serialization. This is consistent with how these terms are used throughout the schema where there might be a standard a dataset conforms to, but no machine readable file to test that - or vice versa.
waldoj
commented
9 years ago Huh, I see the sense in picking up those those existing practices, but the result sure is confusing to me! (FWIW, that JSON Schema draft proposal expired over a year ago, and appears to have been abandoned by its creators.) Well, conformsTo and describedBy it is. Thanks so much for spelling things out.
philipashlock
commented
9 years ago @waldoj FWIW, describedby it's not specific to JSON Schema, it's also used for POWDER, the ADMS profile of DCAT and JSON-LD
The use of this term might be less confusing where it's used in the dataset context where it replaces the previous term dataDictionary - in that context it could very well be human readable documentation that describes the dataset.
Think of conformsTo as referencing a standard something conforms to and describedBy as referencing documentation or a schema that describes it regardless of whether or not it's a standard.
philipashlock
commented
9 years ago @waldoj in case it helps, you got it right to begin with, so sorry I didn't make that more clear.
It seems like
describedBywould be the description of the schema, andconformsTowould the actual schema that it conforms to
That's correct and that's how they're being used.
waldoj
commented
9 years ago Now I'm confused again. :) So conformsTo is machine-readable, and describedBy is human-readable? Or is it the other way around?
philipashlock
commented
9 years ago The fields are actually used in three classes: the Catalog, the Dataset, and the Distribution.
In all cases, they have this general meaning:
conformsTo references a unique identifier of a data standard the resource conforms todescribedBy references a description of the resourceIn the Dataset and Distribution class, describedBy can also be paired with describedByType if it's machine readable rather than human readable, but in the Catalog class (what you originally asked about) it's expected to always be machine readable and point to a JSON Schema file.
I think part of the confusion is that we're referring to this standard with the word "schema" and one of the ways we're describing it is with something called JSON Schema. It might be more clear if we swapped out the word "schema" in your original statement with "standard" like so:
describedBywould be the description of the standard, andconformsTowould the actual standard that it conforms to
In the Catalog class, describedBy is always machine readable, but it never matters if conformsTo is since that's only used as an identifier for a standard.
waldoj
commented
9 years ago Ah-ha! Yes, now everything has snapped into place. So they do mean the thing that I initially thought that they should mean, and all is right in the world. :) You're right, my use of the word "schema" was confusing everything. Thanks for your patience, @philipashlock. :)
gkellogg
commented
9 years ago @philarcher1 and I worked on the JSON-LD representation described at the top of the issue a little bit at TPAC. Here's a link to the JSON-LD playground that starts to improve the context and show how it actually expands out: http://tinyurl.com/q6q3zjo. That's probably the best place to experiment with JSON-LD markup and transformations.
Copied here below:
{
"@context": {
"dcat": "http://www.w3.org/ns/dcat#",
"org": "http://www.w3.org/ns/org#",
"vcard": "http://www.w3.org/2006/vcard/ns#",
"@vocab": "http://www.w3.org/ns/dcat#",
"dc": "http://purl.org/dc/terms/",
"describedby": {
"@id": "http://powder/describedby",
"@type": "@id"
},
"description": "dc:description"
},
"@id": "http://www.agency.gov/data.json",
"@type": "dcat:Catalog",
"conformsTo": "https://project-open-data.cio.gov/v1.1/schema",
"describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json",
"dataset": [
{
"@type": "dcat:Dataset",
"accessLevel": "public",
"accrualPeriodicity": "R/P1Y",
"bureauCode": [
"018:10"
],
"conformsTo": "http://www.agency.gov/widget-taxonomy/",
"contactPoint": {
"@type": "vcard:Contact",
"fn": "Jane Doe",
"hasEmail": "mailto:jane.doe@agency.gov"
},
"describedBy": "http://www.agency.gov/datasets/widgets-dictionary.html",
"dataQuality": true,
"description": "This dataset provides national statistics on the production of widgets",
"distribution": [
{
"@type": "dcat:Distribution",
"description": "Widgets data as a CSV file",
"downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets.csv",
"format": "CSV",
"mediaType": "text/csv",
"title": "widgets.csv"
},
{
"@type": "dcat:Distribution",
"description": "Widgets data as a zipped CSV file with attached data dictionary",
"downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets-all.zip",
"format": "Zipped CSV",
"mediaType": "application/zip",
"title": "widgets-all.zip"
},
{
"@type": "dcat:Distribution",
"conformsTo": "http://www.agency.gov/widget-data-standard/",
"describedBy": "http://www.agency.gov/widgets/schema.json",
"describedByType": "application/schema+json",
"description": "Widget data as a JSON feed",
"downloadURL": "http://www.agency.gov/feeds/widgets-all.json",
"format": "JSON",
"mediaType": "application/json",
"title": "widgets-all.json"
},
{
"@type": "dcat:Distribution",
"accessURL": "https://data.agency.gov/api/widgets-statistics/",
"description": "A fully queryable REST API with JSON and XML output",
"format": "API",
"title": "Widgets REST API"
}
],
"identifier": "widgets-0001",
"issued": "2011-11-22",
"keyword": [
"widget",
"manufacturing",
"factory"
],
"landingPage": "http://agency.gov/widgets/data",
"language": [
"en-US"
],
"license": "http://creativecommons.org/publicdomain/zero/1.0/",
"modified": "2011-11-19T12:00:00Z",
"primaryITInvestmentUII": "021-006227212",
"programCode": [
"018:001"
],
"publisher": {
"@type": "org:Organization",
"name": "Widget Services",
"subOrganizationOf": {
"@type": "org:Organization",
"name": "Office of Citizen Services and Innovative Technologies",
"subOrganizationOf": {
"@type": "org:Organization",
"name": "General Services Administration",
"subOrganizationOf": {
"@type": "org:Organization",
"name": "U.S. Government"
}
}
}
},
"references": [
"http://agency.gov/docs/widgets-1.html",
"http://agency.gov/docs/widgets-2.html"
],
"rights": "This dataset has been given an international public domain dedication for worldwide reuse",
"spatial": "United States",
"systemOfRecords": "http://www.agency.gov/widgets/sorn/",
"temporal": "2009-09-01T12:00:00Z/2010-05-31T12:00:00Z",
"theme": [
"manufacturing"
],
"title": "U.S. Widget Manufacturing Statistics"
}
]
}@gkellogg @philarcher1 I also started working on expanding the JSON-LD example, any feedback is welcome:
Playground link: http://goo.gl/SrDyyH
gkellogg
commented
9 years ago Just a couple of notes:
dc then you can define "description": "dc:description".Dataset). This pattern is best used when inserting data taken from a different source, or where it substantially affects the representation of the JSON-LD. Note that embedded contexts won't survive round-tripping.xsd:dateTime) that these prefixes are defined.subOrganizationOf). One way to check for this is to set "@vocab": "_:" in the context, which will cause all undefined terms to show up as blank node identifiers rather than simply be removed when expanding or re-compacting.amercader
commented
9 years ago @gkellogg thanks a lot for this, it is incredible feedback. I've created a new issue in our repo to stop hijacking this thread, I'll follow up with the changes there: ckan/ckanext-dcat#20
BernHyland
commented
9 years ago Thanks Gregg (and Phil), that is really helpful guidance and provides the needed guidance on JSON-LD. Looks like the W3C TPAC was a highly productive & busy!!
Cheers,
Bernadette Hyland
On Oct 29, 2014, at 4:20 PM, Gregg Kellogg notifications@github.com wrote:
@philarcher1 and I worked on the JSON-LD representation described at the top of the issue a little bit at TPAC. Here's a link to the JSON-LD playground that starts to improve the context and show how it actually expands out: http://tinyurl.com/q6q3zjo. That's probably the best place to experiment with JSON-LD markup and transformations.
Copied here below:
{ "@context": { "dcat": "http://www.w3.org/ns/dcat#", "org": "http://www.w3.org/ns/org#", "vcard": "http://www.w3.org/2006/vcard/ns#", "@vocab": "http://www.w3.org/ns/dcat#", "dc": "http://purl.org/dc/terms/", "describedby": { "@id": "http://powder/describedby", "@type": "@id" }, "description": "dc:description" }, "@id": "http://www.agency.gov/data.json", "@type": "dcat:Catalog", "conformsTo": "https://project-open-data.cio.gov/v1.1/schema", "describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json", "dataset": [ { "@type": "dcat:Dataset", "accessLevel": "public", "accrualPeriodicity": "R/P1Y", "bureauCode": [ "018:10" ], "conformsTo": "http://www.agency.gov/widget-taxonomy/", "contactPoint": { "@type": "vcard:Contact", "fn": "Jane Doe", "hasEmail": "mailto:jane.doe@agency.gov" }, "describedBy": "http://www.agency.gov/datasets/widgets-dictionary.html", "dataQuality": true, "description": "This dataset provides national statistics on the production of widgets", "distribution": [ { "@type": "dcat:Distribution", "description": "Widgets data as a CSV file", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets.csv", "format": "CSV", "mediaType": "text/csv", "title": "widgets.csv" }, { "@type": "dcat:Distribution", "description": "Widgets data as a zipped CSV file with attached data dictionary", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets-all.zip", "format": "Zipped CSV", "mediaType": "application/zip", "title": "widgets-all.zip" }, { "@type": "dcat:Distribution", "conformsTo": "http://www.agency.gov/widget-data-standard/", "describedBy": "http://www.agency.gov/widgets/schema.json", "describedByType": "application/schema+json", "description": "Widget data as a JSON feed", "downloadURL": "http://www.agency.gov/feeds/widgets-all.json", "format": "JSON", "mediaType": "application/json", "title": "widgets-all.json" }, { "@type": "dcat:Distribution", "accessURL": "https://data.agency.gov/api/widgets-statistics/", "description": "A fully queryable REST API with JSON and XML output", "format": "API", "title": "Widgets REST API" } ], "identifier": "widgets-0001", "issued": "2011-11-22", "keyword": [ "widget", "manufacturing", "factory" ], "landingPage": "http://agency.gov/widgets/data", "language": [ "en-US" ], "license": "http://creativecommons.org/publicdomain/zero/1.0/", "modified": "2011-11-19T12:00:00Z", "primaryITInvestmentUII": "021-006227212", "programCode": [ "018:001" ], "publisher": { "@type": "org:Organization", "name": "Widget Services", "subOrganizationOf": { "@type": "org:Organization", "name": "Office of Citizen Services and Innovative Technologies", "subOrganizationOf": { "@type": "org:Organization", "name": "General Services Administration", "subOrganizationOf": { "@type": "org:Organization", "name": "U.S. Government" } } } }, "references": [ "http://agency.gov/docs/widgets-1.html", "http://agency.gov/docs/widgets-2.html" ], "rights": "This dataset has been given an international public domain dedication for worldwide reuse", "spatial": "United States", "systemOfRecords": "http://www.agency.gov/widgets/sorn/", "temporal": "2009-09-01T12:00:00Z/2010-05-31T12:00:00Z", "theme": [ "manufacturing" ], "title": "U.S. Widget Manufacturing Statistics" } ] } — Reply to this email directly or view it on GitHub.
philarcher
commented
9 years ago Following on from what Gregg and I sat and did quickly the other day, I have spent more time on this. I won't call it the end result, but the current state of play can be seen (again in the JSON-LD playground) at
Some comments:
First of all, again, let me stress how terrific it is to see this. The beauty of JSON-LD is that it gives you a layer of abstraction that allows person A to use their chosen vocabulary that, as if by magic, means that person B can see it in their preferred schema too.
The first thing I've done is to split out the various namespaces that DCAT uses. A lot of it is good ol' Dublin Core, but there's the POWDER describedby (mapped here as describedBy, complete with camel case B), a couple of FOAF terms, VCard, ORG and SKOS. There are a bunch of terms that, AFAICT, are unique to POD so I put those in the POD namespace. The default vocabulary in the context is DCAT.
xsd:dateTime is rarely needed in open data. It's useful for things like weather reports that are generated multiple times per day but if the periodicity is less than once a day (and for most open data sets it is), then just xsd:date is fine (i.e. 2014-11-03, not 2014-11-03T19:52:00Z which is what xsd:dateTime requires me to write for 'now.') Specifying xsd:date throws up an error where a dateTime value is given but there would be an error in other cases if we specified xsd:date. I humbly suggest that from the T onwards is deleted in those cases.
I don't know about access levels, bureau and program codes, and data quality values. describedByType and primaryITInvestmentUII?? Is there a controlled vocabulary for these? They would normally be represented as SKOS Concept schemes (something I can add if needed).
I haven't touched the data itself, just the context section. I notice there are a lot of blank nodes here - something a future iteration should fix.
Finally, this is all prime input to current work at W3C on Data on the Web Best Practices. That WG is chartered to extend DCAT as needed, in particular to cover data quality and granularity, as well as data usage, which may be a description of an application, usage of data in a research effort or any other use.
I'll follow up separately, Phil, on setting up a call.
HTH for now - happy to continue to work on this if helpful.
Phil.
For ease of reference, the context and data are copied below:
{ "@context": { "dcat": "http://www.w3.org/ns/dcat#", "org": "http://www.w3.org/ns/org#", "vcard": "http://www.w3.org/2006/vcard/ns#", "foaf": "http://xmlns.com/foaf/0.1/", "@vocab": "http://www.w3.org/ns/dcat#", "dc": "http://purl.org/dc/terms/", "pod": "https://project-open-data.cio.gov/v1.1/schema#", "skos": "http://www.w3.org/2004/02/skos/core#", "describedBy": { "@id": "http://www.w3.org/2007/05/powder#describedby", "@type": "@id" }, "downloadURL": { "@id": "dcat:downloadURL", "@type": "@id" }, "accessURL": { "@id": "dcat:acessURL", "@type": "@id" }, "title": "dc:title", "description": "dc:description", "issued": { "@id": "dc:issued", "@type": "http://www.w3.org/2001/XMLSchema#date" }, "modified": { "@id": "dc:modified", "@type": "http://www.w3.org/2001/XMLSchema#date" }, "language": "dc:language", "license": "dc:license", "rights": "dc:rights", "spatial": "dc:spatial", "conformsTo": { "@id": "dc:conformsTo", "@type": "@id" }, "publisher": "dc:publisher", "identifier": "dc:identifier", "temporal": "dc:temporal", "format": "dc:format", "accrualPeriodicity": "dc:accrualPeriodicity", "homepage": "foaf:homepage", "accessLevel": "pod:accessLevel", "bureauCode": "pod:bureauCode", "dataQuality": "pod:dataQuality", "describedByType": "pod:describedByType", "primaryITInvestmentUII": "pod:primaryITInvestmentUII", "programCode": "pod:programCode", "fn": "vcard:fn", "hasEmail": "vcard:email", "name": "skos:prefLabel", "subOrganizationOf": "org:subOrganizationOf" }, "@id": "http://www.agency.gov/data.json", "@type": "dcat:Catalog", "conformsTo": "https://project-open-data.cio.gov/v1.1/schema", "describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json", "dataset": [ { "@type": "dcat:Dataset", "accessLevel": "public", "accrualPeriodicity": "R/P1Y", "bureauCode": [ "018:10" ], "conformsTo": "http://www.agency.gov/widget-taxonomy/", "contactPoint": { "@type": "vcard:Contact", "fn": "Jane Doe", "hasEmail": "mailto:jane.doe@agency.gov" }, "describedBy": "http://www.agency.gov/datasets/widgets-dictionary.html", "dataQuality": true, "description": "This dataset provides national statistics on the production of widgets", "distribution": [ { "@type": "dcat:Distribution", "description": "Widgets data as a CSV file", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets.csv", "format": "CSV", "mediaType": "text/csv", "title": "widgets.csv" }, { "@type": "dcat:Distribution", "description": "Widgets data as a zipped CSV file with attached data dictionary", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets-all.zip", "format": "Zipped CSV", "mediaType": "application/zip", "title": "widgets-all.zip" }, { "@type": "dcat:Distribution", "conformsTo": "http://www.agency.gov/widget-data-standard/", "describedBy": "http://www.agency.gov/widgets/schema.json", "describedByType": "application/schema+json", "description": "Widget data as a JSON feed", "downloadURL": "http://www.agency.gov/feeds/widgets-all.json", "format": "JSON", "mediaType": "application/json", "title": "widgets-all.json" }, { "@type": "dcat:Distribution", "accessURL": "https://data.agency.gov/api/widgets-statistics/", "description": "A fully queryable REST API with JSON and XML output", "format": "API", "title": "Widgets REST API" } ], "identifier": "widgets-0001", "issued": "2011-11-22", "keyword": [ "widget", "manufacturing", "factory" ], "landingPage": "http://agency.gov/widgets/data", "language": [ "en-US" ], "license": "http://creativecommons.org/publicdomain/zero/1.0/", "modified": "2011-11-19T12:00:00Z", "primaryITInvestmentUII": "021-006227212", "programCode": [ "018:001" ], "publisher": { "@type": "org:Organization", "name": "Widget Services", "subOrganizationOf": { "@type": "org:Organization", "name": "Office of Citizen Services and Innovative Technologies", "subOrganizationOf": { "@type": "org:Organization", "name": "General Services Administration", "subOrganizationOf": { "@type": "org:Organization", "name": "U.S. Government" } } } }, "references": [ "http://agency.gov/docs/widgets-1.html", "http://agency.gov/docs/widgets-2.html" ], "rights": "This dataset has been given an international public domain dedication for worldwide reuse", "spatial": "United States", "systemOfRecords": "http://www.agency.gov/widgets/sorn/", "temporal": "2009-09-01T12:00:00Z/2010-05-31T12:00:00Z", "theme": [ "manufacturing" ], "title": "U.S. Widget Manufacturing Statistics" } ] }
On 31/10/2014 16:44, Bernadette Hyland wrote:
Thanks Gregg (and Phil), that is really helpful guidance and provides the needed guidance on JSON-LD. Looks like the W3C TPAC was a highly productive & busy!!
Cheers,
Bernadette Hyland
On Oct 29, 2014, at 4:20 PM, Gregg Kellogg notifications@github.com wrote:
@philarcher1 and I worked on the JSON-LD representation described at the top of the issue a little bit at TPAC. Here's a link to the JSON-LD playground that starts to improve the context and show how it actually expands out: http://tinyurl.com/q6q3zjo. That's probably the best place to experiment with JSON-LD markup and transformations.
Copied here below:
{ "@context": { "dcat": "http://www.w3.org/ns/dcat#", "org": "http://www.w3.org/ns/org#", "vcard": "http://www.w3.org/2006/vcard/ns#", "@vocab": "http://www.w3.org/ns/dcat#", "dc": "http://purl.org/dc/terms/", "describedby": { "@id": "http://powder/describedby", "@type": "@id" }, "description": "dc:description" }, "@id": "http://www.agency.gov/data.json", "@type": "dcat:Catalog", "conformsTo": "https://project-open-data.cio.gov/v1.1/schema", "describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json", "dataset": [ { "@type": "dcat:Dataset", "accessLevel": "public", "accrualPeriodicity": "R/P1Y", "bureauCode": [ "018:10" ], "conformsTo": "http://www.agency.gov/widget-taxonomy/", "contactPoint": { "@type": "vcard:Contact", "fn": "Jane Doe", "hasEmail": "mailto:jane.doe@agency.gov" }, "describedBy": "http://www.agency.gov/datasets/widgets-dictionary.html", "dataQuality": true, "description": "This dataset provides national statistics on the production of widgets", "distribution": [ { "@type": "dcat:Distribution", "description": "Widgets data as a CSV file", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets.csv", "format": "CSV", "mediaType": "text/csv", "title": "widgets.csv" }, { "@type": "dcat:Distribution", "description": "Widgets data as a zipped CSV file with attached data dictionary", "downloadURL": "https://data.agency.gov/datasets/widgets-statistics/widgets-all.zip", "format": "Zipped CSV", "mediaType": "application/zip", "title": "widgets-all.zip" }, { "@type": "dcat:Distribution", "conformsTo": "http://www.agency.gov/widget-data-standard/", "describedBy": "http://www.agency.gov/widgets/schema.json", "describedByType": "application/schema+json", "description": "Widget data as a JSON feed", "downloadURL": "http://www.agency.gov/feeds/widgets-all.json", "format": "JSON", "mediaType": "application/json", "title": "widgets-all.json" }, { "@type": "dcat:Distribution", "accessURL": "https://data.agency.gov/api/widgets-statistics/", "description": "A fully queryable REST API with JSON and XML output", "format": "API", "title": "Widgets REST API" } ], "identifier": "widgets-0001", "issued": "2011-11-22", "keyword": [ "widget", "manufacturing", "factory" ], "landingPage": "http://agency.gov/widgets/data", "language": [ "en-US" ], "license": "http://creativecommons.org/publicdomain/zero/1.0/", "modified": "2011-11-19T12:00:00Z", "primaryITInvestmentUII": "021-006227212", "programCode": [ "018:001" ], "publisher": { "@type": "org:Organization", "name": "Widget Services", "subOrganizationOf": { "@type": "org:Organization", "name": "Office of Citizen Services and Innovative Technologies", "subOrganizationOf": { "@type": "org:Organization", "name": "General Services Administration", "subOrganizationOf": { "@type": "org:Organization", "name": "U.S. Government" } } } }, "references": [ "http://agency.gov/docs/widgets-1.html", "http://agency.gov/docs/widgets-2.html" ], "rights": "This dataset has been given an international public domain dedication for worldwide reuse", "spatial": "United States", "systemOfRecords": "http://www.agency.gov/widgets/sorn/", "temporal": "2009-09-01T12:00:00Z/2010-05-31T12:00:00Z", "theme": [ "manufacturing" ], "title": "U.S. Widget Manufacturing Statistics" } ] } — Reply to this email directly or view it on GitHub.
Reply to this email directly or view it on GitHub: https://github.com/project-open-data/project-open-data.github.io/issues/309#issuecomment-61290105
Phil Archer W3C Data Activity Lead http://www.w3.org/2013/data/
http://philarcher.org +44 (0)7887 767755 @philarcher1
philipashlock
commented
9 years ago Thanks you so much @philarcher1, @gkellogg, @amercader
I would then suggest we collapse this @context down to a URL hosted on the main Project Open Data website. So instead of displaying this inline, we could just do:
{
"@context": "https://project-open-data.cio.gov/v1.1/schema/data.jsonld",
"@id": "http://www.agency.gov/data.json",
"@type": "dcat:Catalog",
"conformsTo": "https://project-open-data.cio.gov/v1.1/schema",
"describedBy": "https://project-open-data.cio.gov/v1.1/schema/catalog.json",
"dataset": [
{...}
]
}Would that be ok?
As you've noted, accessLevel, bureauCode, dataQuality, primaryITInvestmentUII, and programCode are all specific to Project Open Data so I don't know if there's a better way of defining those than what you've already done. I'm not familiar with how this would be done using SKOS Concept schemes, so I'd definitely need more background there. I suspect there's a better way to handle describedByType and I was digging into that on https://github.com/project-open-data/project-open-data.github.io/issues/382#issuecomment-60682436 if you have anything to add.
Also, just to check, is it ok to have multiple @id's on an object, or a term with a definition that expands out as "@type": "@id" when you've already used @id? For example, you've described conformsTo and describedBy as both expanding out as an @id which seems right, but there's already "@id": "http://www.agency.gov/data.json"
BernHyland
commented
9 years ago @philipashlock Thank you for really digging into this issue and issue #362. Having @gkellogg and @philarcher1 commenting on the thread has helped to highlight this importance of shared namespace and provision of necessary code samples. Thanks guys.
@philipashlock - What you've proposed above in terms of collapsing @context may not be complete. I believe the example @philarcher1 gave (21 hours ago) that provides the canonical namespace URL may be required however, please double check the W3C spec (Recommendation) specifically in relation to @context, see http://www.w3.org/TR/json-ld/#the-context. Ultimately, @gkellogg is best positioned as one of the main W3C document editors to provide a final thumbs up or additional details to get this code sample / documentation right. What you outlined about the other terms that you're adding to the Project Open Data Metadata Schema v1.1 look fine AFAIK. { "@context": { "dcat": "http://www.w3.org/ns/dcat#", "org": "http://www.w3.org/ns/org#", "vcard": "http://www.w3.org/2006/vcard/ns#", "@vocab": "http://www.w3.org/ns/dcat#", "dc": "http://purl.org/dc/terms/", "describedby": { "@id": "http://powder/describedby", "@type": "@id" }, "description": "dc:description" } ...
Thanks @gkellogg & @philarcher1 for your guidance. @philipashlock, thanks very much for taking onboard this feedback & working hard to get it right. Major kudos to the small but mighty team at GSA who are getting a lot done to advance the US Gov't open data effort.
NB: to those monitoring this thread -- Why does this matter? IMHO, it's critical to get this Project Open Data guidance specified relevant to W3C standards for data publishing is because many US Gov't Agencies / Departments will be held accountable for publishing using the proposed Metadata Schema v1.1 that @philipashlock & team are spearheading. If they don't publish in conformance to this schema the penalties may include reduced funding. So this is not an intellectual exercise, it is very real & important for the ongoing future of US Government Open Data.
philarcher
commented
9 years ago Thanks Bernadette,
Sorry I've been swamped today with one thing or another and so haven't replied to @philipashlock's reply.
What I did yesterday will, I'm sure, need improvement, and Bernadette is right - Gregg has more detailed knowledge than I do of JSON-LD and I probably have more knowledge than him of DCAT, ORG etc. I'm not familiar with the POD schema and plans which I (or anyone else doing this) need to make a proper job of it.
But the basics are clear: a canonical @context file that everyone can use in their JSON-LD descriptions of datasets, whether in a central repository or distributed across multiple repositories, is the best way to ensure interop.
Take Spain as an example. Spain comprises many proud regions (Basque country, Andalucia, Catalonia etc.) and solutions from Madrid do not always go down well. BUT... by not only recommending DCAT but a specific profile of DCAT (produced by Makx Dekkers on behalf of the European Commission), the central Spanish data portal is able to offer datasets from all over the country.
So we're on the right road with excellent prospects but please don't take what I wrote yesterday as gospel.
Phil.
On 04/11/2014 16:57, Bernadette Hyland wrote:
@philipashlock Thank you for really digging into this issue and issue #362. Having @gkellogg and @philarcher1 commenting on the thread has helped to highlight this importance of shared namespace and provision of necessary code samples. Thanks guys.
@philipashlock - What you've proposed above in terms of collapsing @context may not be complete. I believe the example @philarcher1 gave (21 hours ago) that provides the canonical namespace URL may be required however, please double check the W3C spec (Recommendation) specifically in relation to @context, see http://www.w3.org/TR/json-ld/#the-context. Ultimately, @gkellogg is best positioned as one of the main W3C document editors to provide a final thumbs up or additional details to get this code sample / documentation right. What you outlined about the other terms that you're adding to the Project Open Data Metadata Schema v1.1 look fine AFAIK. { "@context": { "dcat": "http://www.w3.org/ns/dcat#", "org": "http://www.w3.org/ns/org#", "vcard": "http://www.w3.org/2006/vcard/ns#", "@vocab": "http://www.w3.org/ns/dcat#", "dc": "http://purl.org/dc/terms/", "describedby": { "@id": "http://powder/describedby", "@type": "@id" }, "description": "dc:description" } ...
Thanks @gkellogg & @philarcher1 for your guidance. @philipashlock, thanks very much for taking onboard this feedback & working hard to get it right. Major kudos to the small but mighty team at GSA who are getting a lot done to advance the US Gov't open data effort.
NB: to those monitoring this thread -- Why does this matter? IMHO, it's critical to get this Project Open Data guidance specified relevant to W3C standards for data publishing is because many US Gov't Agencies / Departments will be held accountable for publishing using the proposed Metadata Schema v1.1 that @philipashlock & team are spearheading. If they don't publish in conformance to this schema the penalties may include reduced funding. So this is not an intellectual exercise, it is very real & important for the ongoing future of US Government Open Data.
Reply to this email directly or view it on GitHub: https://github.com/project-open-data/project-open-data.github.io/issues/309#issuecomment-61673694
Phil Archer W3C Data Activity Lead http://www.w3.org/2013/data/
http://philarcher.org +44 (0)7887 767755 @philarcher1
philipashlock
commented
9 years ago Thanks again for all the feedback. At first I thought it would make sense to discuss JSON-LD with this issue as an alternative to the initial proposal, but I think there's value in doing both so I'd like to move the JSON-LD discussion over to #388 and continue to move forward with the existing proposal to use conformsTo and describedBy as the Project Open Data specific way of addressing this. This doesn't mean we shouldn't do JSON-LD as well, but the initial proposal with conformsTo and describedBy seems like the simplest and most direct first step.
gbinal
commented
9 years ago Makes sense. Totally agree with integrating this but don't see it as a hard requirement for #357.
philipashlock
commented
9 years ago Just as an update here, we've stuck with conformsTo as a requirement and describedBy as optional, but we've also incorporated all the JSON-LD keywords into the v1.1 update. I've indicated the contents of @context are still a draft and we can continue to work on refining it as part of #388. If you have additional comments to add about JSON-LD support, please add them to #388 instead.
gbinal
commented
9 years ago Thank you for driving the conversation around this issue and helping to assemble the v1.1 metadata update.
There appears to be strong consensus around this issue, which has been accepted in the v1.1 update and merged into Project Open Data. Project Open Data is a living project though. Please continue any conversations around how the schema can be improved with new issues and pull requests!
It's important for government staff as well as the public to continue to collaborate to make the Open Data Policy ever better. Though the v1.1 update is a substantial update, future iterations do not have to be, so whatever your ideas - big or small - please continue to work with this community to improve how government manages and opens its data.
Currently we just have this section on http://project-open-data.github.io/catalog/
Also see #79