search

ropensci / datapack

An R package to handle data packages
https://docs.ropensci.org/datapack
44 stars 9 forks source link

getDataPackage appears to fail in datasets with 4000 or more members #128

Closed laijasmine closed 3 years ago

laijasmine commented 3 years ago

getDataPackage appears to fail in datasets with 4000 or more members.

For example a dataset with 4000 members on test.arcticdata.io

d1c <- D1Client("STAGING", "urn:node:mnTestARCTIC")
pkg <- getDataPackage(d1c, identifier="resource_map_urn:uuid:969c14cf-0fa5-451d-84a4-e043fe0e0467", lazyLoad=TRUE, quiet=FALSE)

In the console it looks like we are stuck in the Getting resource map with id part:

...
Downloaded object at URL https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A011bceec-0b21-40ab-9cc5-9fb28282170c
Downloaded object at URL https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A9d7738f1-4883-4b55-86da-c2ecd5ac6ba3
Downloaded object at URL https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3Ad023150d-13d4-45a7-ba32-d8d089abe665
Downloaded object at URL https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A4529aa2b-6e2c-4913-8078-5c0b24fe448c
Downloaded object at URL https://test.arcticdata.io/metacat/d1/mn/v2/object/urn%3Auuid%3A23284959-29d2-4e94-ad00-f85d3b3b731c
Getting resource map with id: resource_map_urn:uuid:47857539-6cad-40ca-94b8-0e99c7e3d4c4
laijasmine commented 3 years ago

@gothub I was wondering if you had a chance to look into this? It will be difficult for us to archive larger data packages if we are unable to use getDataPackage on ones with thousands of members.

laijasmine commented 3 years ago

whoops put this in the wrong place