search

maksimhorowitz / nflscrapR

R Package for Scraping and Aggregating NFL Data
522 stars 139 forks source link

add furrr processing for season pbp scrape #152

Closed tanho63 closed 3 years ago

tanho63 commented 4 years ago

Hiya! I was experimenting with the furrr package for parallel processing after watching some of the rstudioconf 2020 talks, and got some pretty great results on the scrape_season_pbp function - sharing them here, I can go through and look at more of the functions if you guys think it's worth adding more globally across the package.

Here's my "test" version of the scrape_season_pbp code with changes highlighted - it adds a "furrr" argument to the function which defaults to FALSE - you have to choose to turn it on by sending furrr=TRUE in the function call.

image

On my machine (i7-8850H 6-core CPU) this resulted in 4x speed increases over the original code 😎

image

tanho63 commented 4 years ago

I think you can test this by installing my copy of the package via remotes::install_github("tanho63/nflscrapR")? Might need to uninstall your local version first? [unfamiliar with how it works...]