search

calgo-lab / green-db

The monorepo that powers the GreenDB.
https://calgo-lab.github.io/green-db/
22 stars 2 forks source link

Setting Global `Accept-Language` could interfere with spiders for other languages #51

Closed se-jaeger closed 2 years ago

se-jaeger commented 2 years ago

https://github.com/calgo-lab/green-db/blob/1b88178235ffe5c58378267f6630c7ae5d6ca9f2/scraping/scraping/settings.py#L41-L43

One possibility to solve this could be to move this setting to each spider and set this explicitly.

BigDatalex commented 2 years ago

I would go for spider specific settings 👍 You can use custom_settings for this like in asos-spider:

https://github.com/calgo-lab/green-db/blob/e3c89d6e2453472759edc371bbbc359fa2503025/scraping/scraping/spiders/asos.py#L37-L40

se-jaeger commented 2 years ago

Solved by #58 and #54