iherman
commented
9 years ago Gregg,
I see that Jeni has merged already the separate branch on importing metadata:
http://w3c.github.io/csvw/metadata/index.html#importing-metadata
It is a relatively complex algorithm, there are some open issues, but we should check whether it answers your problems. I think it should....
Ivan
Ivan Herman Tel:+31 641044153 http://www.ivan-herman.net
(Written on mobile, sorry for brevity and misspellings...)
On 03 Jan 2015, at 22:19, Gregg Kellogg notifications@github.com wrote:
In working on my implementation, I've faced come seeming contradictions in assertions made in Model, Metadata, RDF and JSON documents.
TL-DR: consider changing metadata processing order to place embedded metadata at a lower precidence than other metadata.
In the Model document Locating Metadata section, the processing order of metadata gives the following precidence:
User-supplied metadata, Embedded metadata, Metadata from a Link header, File-specific metadata, and Directory-specific metadata Setting asside the format of any user-supplied metadata, I take this to mean the following:
Open the target file (either CSV or metadata JSON) If the result of opening the file resulted in a Link header with rel=describedby, process the resulting file as a metadata document Otherwise, if "file-metadata.json" exists, where "file" is the opened file without an extension, attempt to open that and process it as a Metadata document Otherwise, if "metadata.json" relative to the original file, attempt to open that and process it as a Metadata document. Otherwise, create new empty Table metadata. Attempt to extract metadata from the file, resulting in something like the following:
Given:
countryCode,latitude,longitude,name AD,42.546245,1.601554,Andorra AE,23.424076,53.847818,"United Arab Emirates" AF,33.93911,67.709953,Afghanistan Processing the file using steps 1-5 from above, presuming that no other metadata is found results in defaults for Dialect information. This would produce metadata similar to the following:
{ "@context": "http://www.w3.org/ns/csvw", "@id": "https://example.org/countries.csv", "schema": { "columns": [{ "name": "countryCode", "title": "countryCode", "predicateUrl": "countryCode", }, { "name": "latitude", "title": "latitude", "predicateUrl": "latitude", }, { "name": "longitude", "title": "longitude", "predicateUrl": "longitude", }, { "name": "name", "title": "name", "predicateUrl": "name", }], "urlTemplate": "_:" } } This is then merged in with the (empty) metadata to create the metadata used to process the document. (Note this being the case, the purpose of the Core Tablular Data processing model is not clear).
If there were directory-specific metadata (such as in Example 22 in Foreign Key Reference Between Resources, the description of merging in csv2rdf seem to be at odds with my notion of what it means to merge documents together.
I think processing of metadata should probably be left to the metadata and model documents, so that it's not repeated needlessly. If we establish rules for extracting metadata from a CSV, along with clear precidence rules for Inherited Properties, there should be no need to repeat this in the csv2rdf and csv2json documents.
In any event, the merge semantics described here seem to be at odds with the merge order described above. For example, merging the embedded metadata into the directory metadata would replace the "name" and "urlTemplate" values with the extracted values. I realize that the wording says not to overwrite those values (or to order "title" content), but that seems like a reverse merge to me.
Perhaps the order should make the embedded metadata at a lower precidence, so that we start with embedded metadata, and merge in metadata found in steps 2-4 above before merging in user-provided metadata. The primary consideration would be for merging in the "title" property, which would need to ensure that merged-in "title" has a non-empty intersection with any existing value, and that the results of the merge are ordered with the original value first, followed by other values (although, this is significantly complicted by the use of language-maps, which are inherently unordered).
Processing is still complicated, as Dialect values probably need to be applied before the input is processed which implies two passes of the input document: first to extract embedded metadata, and then to process the file including all embedded, found, and user-specified metadata.
I think that these change, combined with consolodating metadata processing in metadata & model documents, would make things much simpler. (And, if the Core Tabular Data could use the same rules, but just change the default for headerRowCount to 0, it could be simplified further still).
— Reply to this email directly or view it on GitHub.
gkellogg
JeniT
In working on my implementation, I've faced come seeming contradictions in assertions made in Model, Metadata, RDF and JSON documents.
TL-DR: consider changing metadata processing order to place embedded metadata at a lower precidence than other metadata.
In the Model document Locating Metadata section, the processing order of metadata gives the following precidence:
Setting asside the format of any user-supplied metadata, I take this to mean the following:
rel=describedby, process the resulting file as a metadata documentAttempt to extract metadata from the file, resulting in something like the following:
Given:
Processing the file using steps 1-5 from above, presuming that no other metadata is found results in defaults for Dialect information. This would produce metadata similar to the following:
This is then merged in with the (empty) metadata to create the metadata used to process the document. (Note this being the case, the purpose of the Core Tablular Data processing model is not clear).
If there were directory-specific metadata (such as in Example 22 in Foreign Key Reference Between Resources, the description of merging in csv2rdf seem to be at odds with my notion of what it means to merge documents together.
I think processing of metadata should probably be left to the metadata and model documents, so that it's not repeated needlessly. If we establish rules for extracting metadata from a CSV, along with clear precidence rules for Inherited Properties, there should be no need to repeat this in the csv2rdf and csv2json documents.
In any event, the merge semantics described here seem to be at odds with the merge order described above. For example, merging the embedded metadata into the directory metadata would replace the "name" and "urlTemplate" values with the extracted values. I realize that the wording says not to overwrite those values (or to order "title" content), but that seems like a reverse merge to me.
Perhaps the order should make the embedded metadata at a lower precidence, so that we start with embedded metadata, and merge in metadata found in steps 2-4 above before merging in user-provided metadata. The primary consideration would be for merging in the "title" property, which would need to ensure that merged-in "title" has a non-empty intersection with any existing value, and that the results of the merge are ordered with the original value first, followed by other values (although, this is significantly complicted by the use of language-maps, which are inherently unordered).
Processing is still complicated, as Dialect values probably need to be applied before the input is processed which implies two passes of the input document: first to extract embedded metadata, and then to process the file including all embedded, found, and user-specified metadata.
I think that these change, combined with consolodating metadata processing in metadata & model documents, would make things much simpler. (And, if the Core Tabular Data could use the same rules, but just change the default for
headerRowCountto 0, it could be simplified further still).