search

dmaclean / dfs-python

Tools for DFS data collection and projection creation.
11 stars 2 forks source link

Don't scrape players not playing today #45

Closed dmaclean closed 10 years ago

dmaclean commented 10 years ago

In order to optimize the lineup-scraping task, it would be a good idea to avoid scraping players who played yesterday but are not playing today. Currently, we just grab data on every player that was in the starting lineup yesterday. If yesterday was a full slate of games and today there are only 6-7 games, then there's a lot of data being scraped that won't be used immediately.

dmaclean commented 10 years ago

Fixed. No longer scraping players who aren’t playing today. I added a function in RotoworldLineupScraper to parse the lineups page, but only record teams playing and return it as an array. This is then checked against each player’s lineup data from the previous day. If the player’s team isn’t in the list, they get skipped.

Files modified:

dmaclean commented 10 years ago

Further enhanced to determine last lineup used by each team playing today and show those players in the CSV with a column stating they are "unverified".