Summer Goal #1: Gather Metadata from Data Repositories

[hglanz]

1) Learn and document how to access and obtain metadata for each of the data repositories 2) Write and contribute code to the repo for obtaining metadata

UCI Machine Learning Repository - done CORGIs - done

[kingsuching]

Hello Dr. Glanz,

I have written code to scrape from any dataset from the datahub.io repository. The .R file is in the main branch while the scraped data is in the Data folder. Thanks!

Sucheen

[kingsuching]

Hello Dr. Glanz,

I have made progress on writing code to scrape datasets from the specified repositories but am having trouble with iterating through the entire repository to obtain all the metadata. The functions I have written can extract the metadata from a single link, but I was mainly having trouble with finding the list of dataset links to iterate over. Can you please point me in the right direction?

Thanks, Sucheen S

[hglanz]

Unfortunately, it's going to vary from repository to repository. I'm going to use the CDC as an example, though.

Step 1) https://data.cdc.gov/ Step 2) Select a topic area (e.g. National Center for Health Statistics) Step 3) Select "Datasets" on the left "View Types" panel Step 4) Now we're on a page with a list of dataset page links.

A couple things to notice:

• There are multiple pages. So the number of pages may need to be hard-coded or scraped from the Step (4) page. (e.g. 12 for National Center for Health Statistics)
  1. From each page we hope to be able to scrape the URL in the links to each dataset mini-page.
  2. Then you should just be able to loop through the links on each page and scrape the metadata from each of those dataset mini-pages.
  3. Repeat steps (1) and (2) for each page.

This process will vary and depend on the structure of each data repository, unfortunately. Let me know if this doesn't make sense or if you have any questions :)

[kingsuching]

Thank you for your feedback. I will try that soon.

[kingsuching]

Hey Dr. Glanz, I was writing code to scrape metadata from data.cdc.gov. I used rvest's read_html(url) %>% html_nodes(css_slector) %>% html_next procedure, and was not obtaining any of the scraped data. I tried this for different pages and css selectors. Can you please point me in the right direction? Thanks!