omkarcloud / botasaurus

The All in One Framework to build Awesome Scrapers.
https://www.omkar.cloud/botasaurus/
MIT License
1.14k stars 103 forks source link

Is it possible to integrate custom scraping task in a class? #94

Closed LeMoussel closed 1 month ago

LeMoussel commented 2 months ago

By integrating custom scraping task _request_scrape_data into a class in this way

class RequestCrawlasaurus():
    def __init__(self, start_url=""):
        self.start_url= start_url

    @request(
            output=None,
            create_error_logs=False,
            raise_exception=True,
            parallel=3,
    )
    def _request_scrape_data(self, request: AntiDetectDriver, link):
        response = request.get(link)

    def crawl(self):
        urls = self._request_scrape_data(link=self.start_url)

I got this error: "RequestCrawlasaurus._request_scrape_data() missing 1 required positional argument: 'link'"

Is it possible to embed the custom scraping task _request scrape data into a class? If yes, how?

Chetan11-dev commented 1 month ago

Make it static, and remove self from the function