Create a new DataObject that references an existing object in DataONE

[gothub]

It should be possible to easily create a DataObject for an object that has already been uploaded to DataONE. It is currently possible for different DataONE packages to include the same object, but a mechanism to create a DataObject, DataPackage that could contain a mix of new local objects and existing DataONE objects is not implemented.

This issue has been raised in other context, but the root of the solution should probably begin with an update to DataObject. Potential solutions:

• add a constructor argument that specifies the existing DataONE identifier for an object
• on upload (in rdataone) uploadDataPackage would detect that the object already exists, by checksum for example, and assign the proper DataONE identifier
  • this would be complicated by multiple identifiers for an object with the same checksum

Other issues that are related to this:

• https://github.com/DataONEorg/rdataone/issues/251
• https://github.com/ropensci/datapack/issues/111

[cboettig]

Thanks @gothub , this would be awesome. In general, I'd love to see a native constructor for DataObject, all the documentation shows the use of new("DataObject", ...) and I believe that's not considered best practice (and makes it harder to import the method by namespace). This also 'hides' the S4 internals from the user. (likewise it would be good if there were accessor methods to avoid users needing @, but I think you largely do that already!)

I'd love to have the function datapack::DataObject(...) as the preferred mechanism. Then the syntax for constructing new objects from local files would be unchanged.

Good question about what the syntax for connecting to an existing DataObject should be. The idea of confirming if the object exists by checksum would be brilliant, but I think not practical (DataONE simply doesn't enforce unique checksums, only unique ids). Other options:

• User just gives the existing DataONE id in the id argument. This means the constructor has to make an API call to DataONE to decide if the id exists though, which isn't ideal.
• Maybe if user gives an id but omits the filename argument, the function would assume we were referring to an existing object? Would this create problems down the line, i.e. if the id didn't exist?
• Overload the filename argument, such that specifying a local file means "upload new object", while specifying a DataONE id means "use existing object" ? From the user's viewpoint this is maybe most intuitive, but I always hesitate to overload an argument like that...

[gothub]

@cboettig @mbjones

For creating DataObjects that refer to a DataONE pid, I propose:

do <- new("DataObject", id="urn:uuid:4dc4a896-31c2-4185-b1d7-ebb37f3f9cd6", reference=TRUE);

The additional parameter reference=TRUE is needed because it is an error to create a DataObject without specifying data to include in it. This param makes it clear that this empty object is a reference. This also sets a slot for this object so that operations on this DataObject can be done intelligently downstream, for example when a DataPackage that contains this object is uploaded to DataONE. If you can recommend a better name for this param, please do.

The 'dataone' package is being updated to indicate which package members are currently in DataONE, for example:

 dp
Members:

filename                    format                      mediaType  size     identifier                                    modified local in DataONE 
strix-pacific-northwest.xml eml://ecoinf...rg/eml-2.1.1 NA         22840    urn:uuid:206ece99-1a14-48a5-9def-4947c999e7ad y        y     n     
Strix-occidentalis-obs.csv  text/csv                    NA         1227     urn:uuid:279ca903-5942-4f9e-9c2b-19acc76f2b3d n        y     y     
OwlNightj.csv               text/csv                    NA         165963   urn:uuid:4dc4a896-31c2-4185-b1d7-ebb37f3f9cd6 n        n     y     
filterObs.R                 application/R               tex...rsrc 554      urn:uuid:75be9163-f05b-42a5-91c8-a95eb9a6fbec n        n     y   

[mbjones]

@gothub @cboettig This use case seems almost identical to the use case that we already support with dataURL as the input parameter for the constructor. From the code documentation:

dataURL A character string containing a URL to remote data (a repository) that this DataObject represents.

This table from the code docs shows where the object expects its data to live based on which parameters are provided to the constructor:

dataUrl	filename	dataobj	comment
Y	N	N	used for lazy loaded DataObjects, 'dataUrl' is the data source
N	Y	Y	'dataobj' is the data source, 'filename' is sysmeta.filename (download filename)
N	Y	N	'filename' is the data source, 'filename' is sysmeta.filename
N	N	Y	Invalid, if 'dataobj' is specified, 'filename' must also be specified.

Can you elaborate on what we need that differs from what is already provided by lazy loading data from a dataURL? One thing I can see is I think we are lacking good user-facing documentation of the use of this feature.

[gothub]

@mbjones @cboettig yes, good point. I considered using id with the setting from the first row in the table above, however, I see these drawbacks:

• the user has to understand the requirement and determine the correct CN resolve, 'object' or MN 'object' URL
• the user doesn't explicitly state that they are intending to reference a DataONE pid, it's implied, so there may be room for misunderstanding, i.e.a user may have the use case: "I want to use the data at this URL, but i wan't it to have the id I just now made up"
• the dataUrl param is intended for lazy-loaded DataObjects, typically created by 'dataone::getDataPackage', not 'reference' objects, which are essentially empty.

With 'id' plus 'reference' the user has explicitly stated that the id is for an existing DataONE object, so no assumptions have to be made regarding what the user wishes to do.

If you think that the extra param is unneeded or confusing, the 'id' plus first row of params could certainly be made to work, and documented to clear potential confusion.

[gothub]

@cboettig @mbjones btw, another method for creating a DataObject that references a DataONE pid is to lazy load the DataObject

outputObj <- getDataObject(d1c, id="urn:uuid:279ca903-5942-4f9e-9c2b-19acc76f2b3d", lazyLoad=T, limit="1GB", quiet=F)
dp <- addMember(dp, outputObj, metadataObj)

This creates a DataObject that contains system metadata, and has the 'dataUrl' defined. This method has the benefit of insuring that the user has access to the data they are including in a newly composed package. The downside of course is that the sysmeta has to be downloaded.

[mbjones]

Yeah, that is very close to what we need, @gothub, and would probably work for @cboettig. if we could omit the limit and quiet params, it would be quite easy to include an existing object in a new package based solely on its existing identifier.

[gothub]

The default for limit is "1GB". This may have been chosen to satisfy a particular use case. In hindsight, a better option may have been no limit if the param is not included. Should this be changed? The quiet param is optional.

[gothub]

The 'lazyLoad' behavior was changed in the dataone package and explained in issue https://github.com/DataONEorg/rdataone/issues/258.

No new parameters were added to DataObject initialization, and the workflow to create a DataObject that references an existing DataONE object is the one described above in the Aug 31 comments.

[cboettig]

Thanks @gothub , this is fantastic! Just to be clear, I should now do:

outputObj <- getDataObject(d1c,
                                             id="urn:uuid:279ca903-5942-4f9e-9c2b-19acc76f2b3d", 
                                             lazyLoad=T)

to access an existing object (without downloading it)

[gothub]

@cboettig Yes, that is the correct call.

For a summary of this argument behavior for the dataone 2.2.0 release, please skip to the very bottom of this issue