search

scrapy-plugins / scrapy-playwright

🎭 Playwright integration for Scrapy
BSD 3-Clause "New" or "Revised" License
1.03k stars 113 forks source link

ScrapyDeprecationWarning for custom_headers #314

Closed blacksteel1288 closed 3 months ago

blacksteel1288 commented 3 months ago

I'm getting a deprecation warning from Scrapy for the custom_headers approach in the docs.

This just started happening this week.

[py.warnings] WARNING: /usr/local/lib/python3.12/dist-packages/scrapy_playwright/handler.py:751: ScrapyDeprecationWarning: Accepting positional arguments in the function passed to the PLAYWRIGHT_PROCESS_REQUEST_HEADERS setting is deprecated. The function should accept three (3) keyword arguments instead: browser_type_name: str, playwright_request: playwright.async_api.Request, scrapy_request_data: dict

elacuesta commented 3 months ago

This is expected. The docs for PLAYWRIGHT_PROCESS_REQUEST_HEADERS were updated in the 0.0.41 release, and a note about the deprecation was added: https://github.com/scrapy-plugins/scrapy-playwright/tree/v0.0.41?tab=readme-ov-file#deprecated-argument-handling

This is in order to be able to pass additional request information, not just the headers: https://github.com/scrapy-plugins/scrapy-playwright/pull/303

blacksteel1288 commented 3 months ago

Thanks. Is there any action I need to take then, to remove the warnings from the log?

elacuesta commented 3 months ago

Yes, refactor your custom_headers function to receive the three keyword arguments that are specified in the docs.

blacksteel1288 commented 2 months ago

Great. I might add, as a tip for the docs:

from typing import Dict