search

scrapy / scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.
https://scrapy.org
BSD 3-Clause "New" or "Revised" License
51.16k stars 10.35k forks source link

Add support for multipart/form-data to FormRequest #6332

Open alexandresgf opened 2 weeks ago

alexandresgf commented 2 weeks ago

Fixes #1897, closes #1954

codecov[bot] commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 43.01%. Comparing base (1d11ea3) to head (6b6af57). Report is 8 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #6332 +/- ## =========================================== - Coverage 83.92% 43.01% -40.92% =========================================== Files 161 161 Lines 11972 12010 +38 Branches 1865 1931 +66 =========================================== - Hits 10048 5166 -4882 - Misses 1588 6421 +4833 - Partials 336 423 +87 ``` | [Files](https://app.codecov.io/gh/scrapy/scrapy/pull/6332?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | | |---|---|---| | [scrapy/http/request/form.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6332?src=pr&el=tree&filepath=scrapy%2Fhttp%2Frequest%2Fform.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2h0dHAvcmVxdWVzdC9mb3JtLnB5) | `19.40% <ø> (-78.36%)` | :arrow_down: | ... and [141 files with indirect coverage changes](https://app.codecov.io/gh/scrapy/scrapy/pull/6332/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy)
wRAR commented 2 weeks ago

Doesn't look like this changes anything except Content-Type, is this correct?

alexandresgf commented 1 week ago

Doesn't look like this changes anything except Content-Type, is this correct?

Right, as suggested by the OP in the ticket, it should be a flag, in this case is_multipart, which changes the Content-Type to multipart/form-data. I also added a new parameter to FormRequest constructor.

wRAR commented 1 week ago

The format of multipart/form-data requests is very different from the application/x-www-form-urlencoded one, see e.g. https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4 or the previous PR on this.