search

mattheworres / hootdraft-scraper

NodeJS-based scraper for player data in Hoot Draft
1 stars 0 forks source link

Ncaa BB scrape doesnt return current players #9

Closed goose2288 closed 2 years ago

goose2288 commented 2 years ago

So I downloaded this and ran the scrape for NCAABB and it does return a csv file with the players but they are from last year. New players for this year are not in the file. Any ideas? Thanks so much

mattheworres commented 2 years ago

Just checked, and it looks like CBS has finally updated a lot of those pages after years of them never changing (and why I scraped them!). NCAA Mens Basketball seems to be a holder on, because I checked NFL, NBA and NHL and none of those leagues even return pages, just a 404 Fumble (not found) page is returned. I'll have to investigate more tonight and see if there's a quick way to rewrite the scraper, but I may need to revamp how it's done as each sport may need its own specific scrape now.

goose2288 commented 2 years ago

Gotcha! Thanks for the reply!

mattheworres commented 2 years ago

Looks like the others are broken as well - at a very high level (it's been a while since I've done anything with this code, I think I updated it to ES6 last time but didn't rewrite it just updated syntax) I will need to:

Code wise this sounds like a fun problem to solve (and I hope to get something in place for football this summer) but splitting away from 1 source to possibly multiple makes the code more fragile (the entire reason I dragged my feet writing a scraper - its inherently fragile) and I've just not had the free time/gumption to take care of this repo (or any of my other hobby projects).

I encourage folks to fork the repo and try your own hand if you can't wait on me (no hard feelings from me) - but I'll try to get something going here at some point.