how to query the database locally? how to bulk download images matching a query?

[coltongrainger]

immediate goals

• [ ] document queries for image groups given metadata
  • I've asked Philip and Kevin for advice and guidance w.r.t. expected queries.
• [ ] provide instructions for local retrieval of images

long-term goals

• [ ] document HTTPS addresses for individual pages
  • probably not hierarchical storage (c.f., UUID structure #3)
• [ ] bulk transfer to end user via Globus?
• [ ] provide sufficient API support (at a per image level) to write a decent Kaggle challenge for parsing the logbooks with computer vision

in-house options

• [ ] https://rda.ucar.edu/#!rda_web_services
  • [ ] scripts for bulk downloading? e.g., https://rda.ucar.edu/datasets/ds650.0/index.html#cgi-bin/datasets/getWebList?dsnum=650.0&action=customize&disp=&gindex=1
  • [ ] https://github.com/NCAR/rda-apps-clients/tree/master/python
  • [ ] https://www2.cisl.ucar.edu/data-portals/research-data-archive/command-line-subset-requests-%E2%80%93-rdams

Fedora or RESTful API options

• [ ] https://wiki.duraspace.org/display/FF/Fedora+Repository+Home (a front end for a RESTful API)
• [ ] https://apiguide.readthedocs.io/en/latest/build_and_publish/use_RESTful_urls.html (general advice for creating APIs)
• [ ] https://www.biodiversitylibrary.org/docs/api3.html (an example)

[coltongrainger]

document API design

[coltongrainger]

Fortunately, NARA has an API that allows to obtain standard metadata for logbooks included in the manifest.

I've included this metadata in a first draft of the database schema. From the manifest, I've designed the database based on sample data (~10 GB of image files) and metadata (json files from the NARA API) for ~8 logs from ~7 ships.

I can include sample logs from the record groups RG-26 and RG-261 once NARA assigns them URLs.