Looping through plays to scrape all at once

[havishak]

Hi @nrennie,

This is a cool dataset, good work! As I was reviewing the code, I wondered if you could loop through the plays to scrape all together. Something like this -

library(polite)
library(rvest)
library(tidyverse)

# start scraping session 
website_link <- "https://shakespeare.mit.edu/"
start_session <- bow(website_link)

all_works <- scrape(start_session) %>%
    html_table() %>% 
    .[[2]] %>% #it's the second table, and making tidy
    rename(
        "comedy" = "X1", 
        "history" = "X2",
        "tragedy" = "X3",
        "poetry" = "X4"
    ) %>%
    filter(comedy != "Comedy") %>%
    pivot_longer(
        cols = everything(),
        names_to = "genre",
        values_to = "works"
    ) %>%
    separate_longer_delim(cols = works, delim = "\n") %>%
    filter(!works %in% c("", "The")) %>%
    mutate(
        works = ifelse(works == "Merry Wives of Windsor", "The Merry Wives of Windsor", works),
       # now, get the links for the tables
        work_link = scrape(start_session) %>%
            html_elements("table") %>%
            .[[2]] %>%
            html_elements("a") %>%
            html_attr("href"),
       # adding website link
        work_link = paste0(website_link, work_link)
    ) 

# full play scripts exist for everything except 'Poetry'

plays_to_scrape = all_works %>%
    filter(genre != "poetry") %>%
   # changing website address to the full play
    mutate(work_link = gsub("/index.html","/full.html", work_link ))

# loop through plays_to_scrape

map(plays_to_scrape$work_link, extract_data)

[nrennie]

Hi @havishak

Thanks for this! 🎉 Yes, you definitely could do something like this (and it is on my list of things to do), but there were a few reasons I haven't done it yet:

• As you noted, the extract_data() function doesn't work for poetry. That data is formatted in a different way (and so the extract_data() function will be decomposed into two functions extract_play() and extract_poem() first).
• There are some special cases that the extract_data() doesn't capture e.g. in Romeo and Juliet. I'd prefer to fix this first before running it on all datasets to make it easier to debug.
• This will be part of the work in #2, where metadata including the play title, category, and filename get listed together (ideally one function will produce all of the data files and the metadata file)

Thanks for the suggestion and code - I'll likely come back to this issue when I'm ready to run it on everything so I'll leave it open 🎉

A couple of notes:

• the columns names are already in the first row of the data (no need to hard code categories)
• Use delim = "\n\n" to separate on double spaces which get's rid of problem with "" and "The"
• avoid running scrape(start_session) multiple times on the same URL in one function since the data is already there.

[havishak]

Thanks for the notes!