adding header parameter to src/scrapeghost/scrapers.py

jamesturk / scrapeghost

👻 Experimental library for scraping websites using OpenAI's GPT API.

https://jamesturk.github.io/scrapeghost/

Other

1.43k stars 87 forks source link

adding header parameter to src/scrapeghost/scrapers.py #46

Closed ltngonnguyen closed 1 year ago

ltngonnguyen commented 1 year ago

Some pages would need a specific user agent or other HTTP headers to access, so I just added a new header parameter to the scrape method and document it in usage.md. Please also help check to see if it will be applied for chunked tags as well.

jamesturk commented 1 year ago

Hi, thanks for the interest, but this is not the preferred solution, see https://github.com/jamesturk/scrapeghost/pull/34