b5
commented
6 years ago Ok, a few changes, general plan to make this work. I think we should flip this around & make it possible to add dataset.json references to dt archives. This will provide the dt archive with the added benefit of POD / DCAT / qri / JSONLD ready metadata, and will make it possible to register a DT archive with qri. As a side benefit qri can be used to edit the metadata of an archive, and the cdxj index will become a sql-queryable database, making it possible to
The benefit to qri is this archive can be used as a starting point for extracting further datasets. As an example a user could extract data from an excel file within the archive, referencing the archive as the initial dataset from which the information was extracted. This provides archival provenance as the foundation to build a dataset upon. Users will be able to return to pages in the archive to get the context in which the original data was presented as a reference for writing metadata or understanding changes over time.
This ups the need for json selector syntax support within qri's dataset_sql, it would allow users to query data stored within the cdxj json field, which would be hella dope.
Steps to complete an initial implementation:
- [x] Add Json datatype to
dataset/datatypepackage - [x] Add cdxj Reader/Writer wrappers to
dataset/dsio - [x] Round out cdxj test coverage to something north of 60%
- [x] Add Support for
DataFormatCDXJformat to dataset package - [x] Add cdxj index & dataset.json to
dt archivecommand - [ ] update
dataset_sqlwith tests for a cdxj formated dataset, including json output - [x] add in a cdxj validator to
dataset/validate - [ ] confirm frontend doesn't explode with cdxj-formatted data
over at @datatogether we're writing a number of archival-quality tools, I think we should modify the dataset spec to include all archival details that the WARC spec assumes should be covered.
Once we've implemented #86, we should look to make those url Ingest's contain archival backing information.