Closed mbjones closed 2 years ago
I see two use cases that would need to be supported - 1. building a package from scratch and 2. downloading a package from DataONE to create a DataPackage in R. This may involve updates to addMember
, getMember
, insertRelationships
- for use case 1.
For use case 2, the rdataone
package methods uploadDataPackage
and getDataPackage
would need updating.
Are there other use cases that we would support?
One way to do this in datapack
is to create a user friendly method that wraps the following
dataPackage <- insertRelationship(dp, filePatj,
"http://www.w3.org/ns/prov#atLocation",
someObjectID)
Call the method setFilePath
or something something similar. Tools that use this package (such as rdataone
) can then call this method in some preferred way.
If this library is meant to be fairly general, any tooling that uses it can easily create its own method that does the work. For example, we could instead just modify rdataone.addMember
to take a file path, which would in turn call datapack.DataPackage.insertRelationship
.
At the end of the day, datapack.insertRelationship
is going to have to be called somewhere.
Any @mbjones or @gothub - any opinions on whether or not I add a method to the datapack
DataPackage class: addFilePath(objectId, path)
, or instead modify addMember
in rdataone
to take an optional file path and call datapack.DataPackage.insertRelationship
from there?
@ThomasThelen I think that modifying a DataPackage is in the scope of datapack
, and if you want a utility method for setting prov:atLocation
, then it would be good to be placed in datapack
so that anyone creating a package can call it. datapack::describeWorkflow
is s similar utility method.
@mbjones I'm currently writing the R implementation and I'd like to outline a few workflows to make sure what I'm doing makes sense and to gather feedback on how this is exposed.
DataObject
to a DataPackage
This is one of the most basic workflows. Consider a user that wants to get a bagit representation of some files on their disk. They decide to use datapack
to...
The code to do this without any sort of file path adding would look something like...
library(datapack)
dp <- new("DataPackage")
# Add the script to the DataPackage
progFile <- system.file("./extdata/pkg-example/logit-regression-example.R", package="datapack")
progObj <- new("DataObject", format="application/R", filename=progFile)
dp <- addMember(dp, progObj)
serializeToBagIt(dp)
The goal is to introduce a non-invasive way of letting the user describe the path of logit-regression-example.R
. In this case, the progFile
variable has a nicely formed path: It doesn't include a letter drive, the user's home directory, or other non-relevant paths (ie not Desktop/cat-memes/extdata/pkg-example/....
).
It would be great to just grab the path in progFile
, and add it as the prov:atLocation field. The case of nicely formed paths however, is not always the case and it should be assumed that people are going to be using full paths that contain drive letters, home folders, etc. Problems also arise when thinking about how to automatically parse the path.
We can certainly automatically clean C:/tthelen/top-secret-experiments/github/datapack/myData.csv
to tthelen/top-secret-experiments/github/datapack/myData.csv
however, there's no way to tell that the desired path is datapack/myData.csv
.
Another approach that is most likely less error prone is asking the user to add relative path when creating the DataObject.
new("DataObject", format="application/R", filename=progFile, filepath=./extdata/pkg-example/logit-regression-example.R)
This use case follows the same general pattern as the first one. In both cases the user will be interacting with datapack to create a DataObject
and then adds it to a DataPackage
.
I've created a draft pull request, linked above, with the changes. @mbjones, @gothub, @jeanetteclark if you could take a look at that and let me know if there are any additional changes that that need to be made I'd appreciate it!
Ready for review.
This is an enhancement request to support file hierarchies in the R
datapack
implementation compatible with other implementations of the DataONE data packaging specification, and bring it into conformance with the RDA recommendation as described in https://github.com/DataONEorg/api-documentation/issues/5Currently
DataPackage
consists of a flat collection ofDataObject
instances. A proposal has been made to include hierarchies of objects within folders to more easily allow DataPackages to mirror file system hierarchies. WithinDataPackage
, this requires that we record the location of objects within the nested folder hierarchy, relative to the root of the data package. Here is an example folder hierarchy that a researcher might want to encode in their data package:This hierarchy would be encoded by recording the relative path of each object in the hierarchy in a
prov:atLocation
property in the OAI-OREResourceMap
for the package. Thus, we need to add methods for adding and removing these prov:atLocation properties from theResourceMap
. They would be serialized in the resource map as follows:<prov:atLocation>outputs/output1.csv</prov:atLocation>
It's important to note that the path is relative to the data package root, and does not include
file://
nor thedata
directory that is part of the bag.In addition, we need to serialize and de-serialize these object hierarchies when creating importing and exporting versions of packages, currently in BagIt format. A BagIt bag that contains a DataPackage would serialize the objects of the package within the
data
directory of the bag, and would treat thedata
directory as the root of the file hierarchy for the package. The package hierarchy shown above would be placed within a BagIt bag as follows:See the proposal on modifying the DataONE package specifiction for more details and discussion.