requests / toolbelt

A toolbelt of useful classes and functions to be used with python-requests
https://toolbelt.readthedocs.org
Other
989 stars 186 forks source link

MultipartEncoder Block Size #368

Open StefanGreve opened 11 months ago

StefanGreve commented 11 months ago

The current documentation links to an issue that has been already closed:

https://toolbelt.readthedocs.io/en/latest/uploading-data.html

As mentioned by another commentator, this notice is outdated since Python 3.7+ allows override this default (c.f. docs).

I am not sure if it is already possible to increase the block size in MultipartEncoder. It would be great if somebody could shed some light on the current situation. A lot of information you find on StackOverflow or Google is outdated.

sigmavirus24 commented 11 months ago

This library does not create the HTTPConnection object and thus cannot control the blocksize. That warning is there as people complained about speed of writes when the large part of the issue is inefficient reads. To access that configuration, you will need to work with urllib3 to make that configurable in their integration with httplib in a way that can be passed through requests and thus avoid this issue.

Ostensibly, if urllib3 supports specifying that for a pool of connections, then it could be turned into an Adapter here that could be used in combination with the encoder

sigmavirus24 commented 2 months ago

There's no change to the library, just documentation. Any desire to make the read size configurable needs to be addressed with urllib3. #3066 already raised the default for urllib3 so if further customization is desirable, it should be addressed there. If someone wants to fix the docs, I'll happily merge that