search

isaactpetersen / ffanalytics

ffanalytics R package
22 stars 19 forks source link

Issue with scraping ESPN #8

Closed singularperturbation closed 7 years ago

singularperturbation commented 7 years ago

When running runScrape, none of the ESPN data is able to be scraped:

Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=0&startIndex=160 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=2&startIndex=280 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=2&startIndex=320 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=2&startIndex=360 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=2&startIndex=400 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=4&startIndex=400 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=4&startIndex=440 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=4&startIndex=480 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=4&startIndex=520 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=6&startIndex=200 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=6&startIndex=240 
Empty data table retrieved from
 http://games.espn.go.com/ffl/tools/projections?&slotCategoryId=6&startIndex=280

Looking at the page, I think they've just changed the startIndex values. I'll work on a PR.

singularperturbation commented 7 years ago

Hmm - looking at this again, I don't think this is a problem - these indexes are for pagination, and are one after the ESPN data ends (so after the script has already paged through all the existing data).

Still looking at runScrape to make sure it's running correctly, but I don't think this is an issue.