Ingest Modularity/Improvements

IQSS / dataverse

Open source research data repository software

http://dataverse.org

Other

878 stars 486 forks source link

Ingest Modularity/Improvements #7852

Open djbrooke opened 3 years ago

djbrooke commented 3 years ago

Related to ingest, I'm interested in

allowing installations to ingest larger files ( handled in IQSS/dataverse.harvard.edu#111)
not experiencing system instability due to aforementioned large file ingests, or many ingests ( handled in IQSS/dataverse.harvard.edu#111)
allowing for external developers to more easily utilize the ingest code (I'm thinking TRSAs that want to generate and deposit metadata to a Dataverse installation ex. https://github.com/akio-sone/dataverse-ingest)

akio-sone commented 3 years ago

Before TRSA, Tim Parsons factored out the ingest functionality from Dataverse as follows:

data-file-reader

akio-sone commented 3 years ago

The current ingest functionality relies on a premise that the format of a data source has been identified; now is the time to delegate this step to an established tool such as droid that is running on a different server: droid format list User Guide: pdf

I suspect somebody had already developed DROID's web-service version.

djbrooke commented 3 years ago

Let's set this up on a different server (this may satisfy the system load and larger files question) - let's set this up in prod and see how it works. Document in Guides.
Once this is done, it'll give us some idea about our next steps from a code perspective