scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
50.98k stars 10.34k forks source link

Improve typing for Spider.parse(). #6274

Open wRAR opened 2 months ago

wRAR commented 2 months ago

Based on https://stackoverflow.com/questions/77160087/correct-typing-of-a-method-without-a-fixed-signature

Should remove most of the mypy errors for parse() in user spiders (if they use older mypy Spider.parse() will appear untyped so no override error either). Doesn't help with scrapy-poet's callbacks that narrow down Response to DummyResponse: https://github.com/scrapinghub/scrapy-poet/issues/179

This should probably have typing tests, I'll do that later.

codecov[bot] commented 2 months ago

Codecov Report

Merging #6274 (84056f4) into master (861646f) will decrease coverage by 0.03%. The diff coverage is 37.50%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #6274 +/- ## ========================================== - Coverage 88.82% 88.79% -0.03% ========================================== Files 161 161 Lines 11863 11867 +4 Branches 1918 1919 +1 ========================================== Hits 10537 10537 - Misses 979 982 +3 - Partials 347 348 +1 ``` | [Files](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | | |---|---|---| | [scrapy/commands/bench.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL2JlbmNoLnB5) | `100.00% <100.00%> (ø)` | | | [scrapy/spiders/\_\_init\_\_.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6274?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3NwaWRlcnMvX19pbml0X18ucHk=) | `88.23% <28.57%> (-5.52%)` | :arrow_down: |