robons
commented
3 years ago So I did some testing on this and I'll describe my findings below:
Attempting to use the @base to set the document's Base URL
Given the following CSV:
Period
year/2020
month/2020-01And the following metadata JSON document:
{
"@context": ["http://www.w3.org/ns/csvw", {"@base": "https://example.org/"}],
"@id": "#dataset",
"tables": [
{
"url": "problem.csv",
"tableSchema": {
"columns": [
{
"titles": "Period",
"name": "period",
"propertyUrl": "#dimension/period",
"valueUrl": "http://reference.data.gov.uk/id/{+period}"
}
],
"aboutUrl": "http://some/observation/{+period}"
}
}
]
}I get the following output from csv2rdf:
csv2rdf -u problem.csv-metadata.json -m annotated
13:23:14.328 [main] ERROR csv2rdf.main - #error {
:cause clj-http: status 404
:data {:request-time 993, :repeatable? false, :protocol-version {:name HTTP, :major 1, :minor 1}, :streaming? true, :chunked? false, :type :clj-http.client/unexceptional-status, :reason-phrase Not Found, :headers {Cache-Control max-age=604800, Content-Type text/html; charset=UTF-8, Date Tue, 13 Jul 2021 13:23:14 GMT, Expires Tue, 20 Jul 2021 13:23:14 GMT, Server EOS (vny/044F), Vary Accept-Encoding, Connection close}, :orig-content-encoding nil, :status 404, :length -1, :body <!doctype html>
<html>
<head>
<title>Example Domain</title>
... html continues - removed by robons for brevity ...
, :trace-redirects []}
:via
[{:type clojure.lang.ExceptionInfo
:message clj-http: status 404
:data {:request-time 993, :repeatable? false, :protocol-version {:name HTTP, :major 1, :minor 1}, :streaming? true, :chunked? false, :type :clj-http.client/unexceptional-status, :reason-phrase Not Found, :headers {Cache-Control max-age=604800, Content-Type text/html; charset=UTF-8, Date Tue, 13 Jul 2021 13:23:14 GMT, Expires Tue, 20 Jul 2021 13:23:14 GMT, Server EOS (vny/044F), Vary Accept-Encoding, Connection close}, :orig-content-encoding nil, :status 404, :length -1, :body <!doctype html>
<html>
<head>
<title>Example Domain</title>
... html continues - removed by robons for brevity ...
, :trace-redirects []}
:at [slingshot.support$stack_trace invoke support.clj 201]}]
:trace
[[slingshot.support$stack_trace invoke support.clj 201]
[clj_http.client$exceptions_response invokeStatic client.clj 239]
... again, edited for brevity ...
[csv2rdf.main main nil -1]]}csv2rdf is evidently trying to resolve the CSV file at http://example.org/problem.csv which it naturally fails to do.
What the CSV-W Spec Says
The CSV-W spec confims that this is the expected behaviour
A table description might contain:
Example 5
"url": "example-2014-01-03.csv"
in which case the url property on the table would have a single value, a link to example-2014-01-03.csv, resolved against the base URL of the metadata document in which this was located. For example if the metadata document contained:
Example 6
"@context": [ "http://www.w3.org/ns/csvw", { "@base": "http://example.org/" }]
this is equivalent to specifying:
Example 7
The CSV-W spec also confirms that the base URL is not used as a base for relative URIs defined in json-ld metadata:
Note that the @base property of the @context object provides the base URL used for URLs within the metadata document, not the URLs that appear as data within the group of tables or table it describes. URI template properties are not resolved against this base URL: they are resolved against the URL of the table.
i.e. if we define the following CSV-W metadata json file:
{
"@context": "http://www.w3.org/ns/csvw",
"@id": "#dataset",
"tables": [
{
"url": "this-should-be-where-uris-are-relative-to.csv",
"tableSchema": {
"columns": [
{
"titles": "Period",
"name": "period",
"propertyUrl": "#dimension/period",
"valueUrl": "http://reference.data.gov.uk/id/{+period}"
}
],
"aboutUrl": "http://some/observation/{+period}"
}
}
]
}Then we get the following RDF output from csv2rdf:
<http://some/observation/year/2020> <file:/workspace/this-should-be-where-uris-are-relative-to.csv#dimension/period>
<http://reference.data.gov.uk/id/year/2020> .
<http://some/observation/month/2020-01> <file:/workspace/this-should-be-where-uris-are-relative-to.csv#dimension/period>
<http://reference.data.gov.uk/id/month/2020-01> .Conclusions
- The
@baseproperty which can be defined in the CSV-W is not a property which allows us to set the base URL we want json-ld metadata to be defined relative to. - In order to allow us to configure the URI, we must either:
- Ask Swirrl to add a command-line argument to the
csv2rdfapplication to allow us to specify what the json-ld metadata's base URL should be. - Add a build step following the
csv2rdfprocess which finds-and-replaces all occurances offile:///workspace/...with an alternative base URL.
- Ask Swirrl to add a command-line argument to the
@rossbowen @ajtucker what are your thoughts on how we want to proceed here?
rossbowen
ajtucker
canwaf
First, check if we can use base-uri property in CSV-W to generate URIs for relative URIs, if not... If we can, proceed by creating a Jenkins pipeline issue to allow for the configuration of the base-uri property. If we can't, chuck this over the fence for csv2rdf as a feature request.
Basic test case is whether does csv2rdf break with the base-uri with a URL in it, if it tries to find a remote CSV then it breaks. If it uses the local CSV then it works.
Investigate whether csv2rdf and csvlint follow the csvw spec.