I have a program which reads the URLs of US television news pages to do some scraping. The master list is of some 558 stations.
For some reason, the URLs on the attached list, when queried, return no text.
Here is the code I punched out to just check these URLs, the attached .csv file contains all the "problem" sites.
for(current_url in test_urls) {
raw_page <- read_html(current_url)
if(length(raw_page) > 1) next
print(current_url)
capture <- c(capture,current_url)
print(raw_page)
}
I'll tell you that ALL of the sites, at least the ones I spot checked, appear to be using the same CSS template... so I assume that read_html and that particular CSS don't like each other.
d_test_out.csv
I have a program which reads the URLs of US television news pages to do some scraping. The master list is of some 558 stations. For some reason, the URLs on the attached list, when queried, return no text. Here is the code I punched out to just check these URLs, the attached .csv file contains all the "problem" sites.
for(current_url in test_urls) { raw_page <- read_html(current_url) if(length(raw_page) > 1) next print(current_url) capture <- c(capture,current_url) print(raw_page) }
I'll tell you that ALL of the sites, at least the ones I spot checked, appear to be using the same CSS template... so I assume that read_html and that particular CSS don't like each other.