search

DataONEorg / rdataone

R package for reading and writing data at DataONE data repositories
http://doi.org/10.5063/F1M61H5X
36 stars 19 forks source link

For `getDataObject()`: allow argument of `limit=-1` #284

Open gothub opened 2 years ago

gothub commented 2 years ago

Specifying limit=-1 will have the special meaning to always download an item, regardless of it's size.

This is preferred over specifying a large value, for example limit=1TB. This feature will be used for resource maps, which should always be downloaded by getDataPackage().

If there are use cases for which downloading resources maps isn't desired, then I suggest that we develop a different tool for those use cases.

gothub commented 2 years ago

@jeanetteclark what are your thoughts regarding always downloading the resmap (and potentially the metadata object) for getDataPackage(), regardless of requested lazyLoad and limit specified for downloading a package?

jeanetteclark commented 2 years ago

yeah I can't think of a reason to not grab the resource map so that seems reasonable Peter

gothub commented 2 years ago

@jeanetteclark ... and the same for the metadata object?

jeanetteclark commented 2 years ago

yeah same, at least the sysmeta for it. not sure about downloading the entire thing

gothub commented 2 years ago

Do you get many large metadata documents? I'm assuming that the 'limit=' argument is intended for data objects, so may not always be relevant for metadata. Also, is it always necessary to download the metadata object for your use cases? It's probably needed when you want to update it, yes?

Would it be useful to discuss this at the ADC meeting tomorrow for a few minutes?