search

HackerSpace-PESU / Best11-Fantasycricket

Predicting the Best 11 for a fantasy cricket game
GNU Affero General Public License v3.0
24 stars 17 forks source link

[FEATURE] Add batting crawler to the webcrawler #42

Closed roysti10 closed 3 years ago

roysti10 commented 3 years ago

Is your feature request related to a problem? Please describe. The crawler currently only crawls for new players , and match ids. The web crawler also needs to crawl for batting stats

Describe the solution you'd like A function similar to parse_player and parse_scorecard Extend function parse_scorecard in crawler/cricketcrawler/cricketcrawler/spiders/howstat.py to batting and get the following

Refer Dataset.md to understand the matchcodes and playercodes

Additional context Crawler can be found in feature-webcrawler branch of the repo. Crawler is built using [scrapy](https://scrapy.org/)

scientes commented 3 years ago

to reduce requests i'd advise to not make a new function, but to just extend the parse_scorecard function so that batting etc. is collected, if the information is on the same page as the matchid stuff. (but maybe im wrong) if anybody has questions about scrapy and its working i'll help but my knowledge of cricket is insufficient to say the least.

roysti10 commented 3 years ago

Ahh right that sounds better , I had a few questions myself. I'll be sure to ask you those once I get free. Im quite busy this and next week myself.

roysti10 commented 3 years ago

to reduce requests i'd advise to not make a new function, but to just extend the parse_scorecard function

Ill make the change for this

roysti10 commented 3 years ago

Closed by commit 6272197

Point 8