search

oseymour / ScraperFC

Python package for scraping soccer data from a variety of sources
GNU General Public License v3.0
222 stars 49 forks source link

More Leagues for Fotmob's Shot Scraper #4

Closed hedonistrh closed 2 years ago

hedonistrh commented 2 years ago

Hey, I was also writing scraper for Fotmob and realized that you already pushed some changes. * 💯 When I check leagues, realized that you can also add following links into here 

            "Champions League":           "https://www.fotmob.com/leagues/42/matches/champions-league",
            "Euro":                       "https://www.fotmob.com/leagues/50/matches/euro",
            "Europa League":              "https://www.fotmob.com/leagues/73/matches/europa-league",

Those links also have shot-maps as that is what code is looking for. (For instance Copa America and Conference League do not have that so possibly adding that does not make sense now.)

oseymour commented 2 years ago

Hey Halil! It's funny you opened this. I just added those links (plus the WC) last night, I just haven't pushed the code yet. I'll close this when I do push.

I'm glad it's inspired you! And I'd love to see your implementation. The code I have right now is really slow, so if you can get something that runs faster that'd be great!

hedonistrh commented 2 years ago

Thanks a lot for the heads-up. 👏 For the implementation, probably I'll finish that by EOW and will share with you. Also probably will profile your code and that implementation. So we can understand how performance differs. 💪 Will update about that as well.

oseymour commented 2 years ago

This is awesome! I'll finish my code and then compare. Have you profiled the time it takes to scrape each shot yet?

oseymour commented 2 years ago

0.7 is really good! My old code took around 50 seconds per shot. I've gotten mine down to around 0.3 seconds/shot, so I'll probably use mine when I release the FotMob module.

Again, the code on the repo is really old and was never tagged for release. It's not expected to work. I'll tag the FotMob release and announce it on twitter when it's done.

hedonistrh commented 2 years ago

Sounds great. 👌 I'll compare when you release it (as profiling depends on the system so much) and if I find any speed-up, will share with you. 💯 Thanks!

hedonistrh commented 2 years ago

Hey Owen, I found the bug with my implementation. 😪 I was thinking that when you open the one of the match page, shotmap starts with the last shot of the game. (This was the case before if i am not missing anything) But today, it is starting with the last shot which is 'goal'. So the code which I shared are able to still get all shots but sorting of that and logging message is not fully correct. I will probably share the fix tomorrow.

AFAIK you are not using that but i will share just for future reference. ^-^

oseymour commented 2 years ago

Closing this issue and deleting any comments with scraping code in them. This is being done at the request of the owner of the data. Please do not post any more code or comments in this issue.