search

NIAID-Data-Ecosystem / nde-crawlers

Harvesting infrastructure to collect and standardize dataset and computational tool metadata
Apache License 2.0
0 stars 0 forks source link

Import ClinEpiDB metadata #35

Closed flaneuse closed 1 year ago

flaneuse commented 2 years ago

Example dataset: https://clinepidb.org/ce/app/workspace/analyses/DS_4902d9b7ec/new/details and data downloads: https://clinepidb.org/ce/app/workspace/analyses/DS_4902d9b7ec/new/download

To do:

  1. Double check there's no API access
  2. Ensure the web/data usage agreements allow for scraping.
  3. Figure out how to get the study IDs from the overall table: https://clinepidb.org/ce/app/search/dataset/Studies/result
  4. Loop over the ids to grab all the metadata from the various tabs on the dataset pages.