search

psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.02k stars 9.3k forks source link

Make instances of PreparedRequest class pickable #1558

Closed piotr-dobrogost closed 7 years ago

piotr-dobrogost commented 11 years ago

See How can I save a python HTTP requests from python-requets so I can execute it later? question on Stack Overflow.

sigmavirus24 commented 11 years ago

Is that the only remaining class that isn't Pickleable? If so, I'll work on it for 2.0.

Lukasa commented 11 years ago

Looks like it might be. I'd love this in 2.0. =)

ssbarnea commented 11 years ago

:+1: If we make a patch for the current version, would you accept it? -- I want to improve the python-jira library without having to use a non-released version of requests.

Lukasa commented 11 years ago

@ssbarnea To the best of my knowledge we have no plans to release any further 1.2.X point releases. The next planned release is 2.0.0. I'm potentially open to you making a patch against master, but I'd want to check with @kennethreitz first.

kennethreitz commented 11 years ago

Go for it

--

Kenneth Reitz

On Wed, Aug 28, 2013 at 7:34 AM, Cory Benfield notifications@github.com wrote:

@ssbarnea To the best of my knowledge we have no plans to release any further 1.2.X point releases. The next planned release is 2.0.0. I'm potentially open to you making a patch against master, but I'd want to check with @kennethreitz first.

Reply to this email directly or view it on GitHub: https://github.com/kennethreitz/requests/issues/1558#issuecomment-23407592

Lukasa commented 11 years ago

Sweet, feel free to patch against master then. =)

Lukasa commented 11 years ago

Ok, looking at this again we don't currently make any of the trio of Request, PreparedRequest and Response pickleable. Response is potentially difficult to pickle (file descriptors, underlying urllib3 response object isn't pickleable), but the other two are the things worth making pickleable anyway. We should shoot for both of them I think.

sigmavirus24 commented 11 years ago

I'm not (quite frankly) convinced that a Request object should be entirely pickleable. It is an ephemeral object that we use to construct the PreparedRequest. PreparedRequest objects do not store a reference to it, so I'm 90% sure it does not need to be pickleable. We should probably make it pickleable anyway though it is a seriously minor introduction of code and it will cover us for the future.

I'm wondering if @shazow would be opposed to making the urllib3 response object pickleable. If he were to approve, then I think it would make sense to also make the Response object pickleable.

Lukasa commented 11 years ago

The biggest issue with pickling PreparedRequests is pickling hooks. We could be super-clever and try to pickle the function objects but I'm not convinced that's a great plan. Alternative suggestions?

shazow commented 11 years ago

Hrm, yea not sure if it makes a lot of sense. Especially since the Response object has a lot of state within it, state which is network-dependent too (is the socket still open? has the body been read? etc).

What would it take to make urllib3's Response object pickleable?

ssbarnea commented 11 years ago

Maybe I should explain better what I want to achieve and hopefully we will find the best wait to implement it.

I want to save issue.update requests so I can call them later. Why? Two reasons: speed and reliability, Jira may be down for a while and I cannot implement retry in all the scripts but I could save failed requests for later execution via a cron job. If this behaviour could be enabled without changing existing scripts, the better.

On Sunday, 1 September 2013, Andrey Petrov wrote:

Hrm, yea not sure if it makes a lot of sense. Especially since the Response object has a lot of state within it, state which is network-dependent too (is the socket still open? has the body been read? etc).

What would it take to make urllib3's Response object pickleable?

— Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1558#issuecomment-23631177 .

Cheers, Sorin

shazow commented 11 years ago

I think it's reasonable to have PreparedRequest be pickleable without the rest. Sounds like a sensible edge case to me.

ankitson commented 10 years ago

@Lukasa Using cloudpickle from piclouds cloud library(https://pypi.python.org/pypi/cloud) seems like a good option to me. It supports a lot more types than pythons pickle module, including functions (http://docs.picloud.com/client_pitfall.html#nonsupported-objects lists the exceptions). It would require adding a dependency on cloud, but we could just copy the source, the same way as urllib3 and chardet since the serialization code is not likely to change.

If there are no objections, I can implement this and send in a pull request.

sigmavirus24 commented 10 years ago

We only vendor absolutely necessary dependencies and we try to keep that number as low as possible. As I see it, this is not a high priority for us, nor is vendoring a dependency for cloudpickle quite the approach I would like to take at the moment.

sigmavirus24 commented 9 years ago

So 2.7.0 is imminent. Looking at this there are only one corner case that I think are currently problematic.

On a prepared request, the only thing I think can hurt us is the case when the body attribute is a file object. Is there a specific hook that we're concerned about not being pickleable?

frobicatorman commented 8 years ago

i'd like to work on this ticket, if no one else plans to. haven't contributed to requests before.

sigmavirus24 commented 8 years ago

I thought we had fixed this, but I guess not.

nateprewitt commented 7 years ago

With Keyan's work merged in with #3489, I think we have this tested as much as we can. Due to the inability to pickle file-objects and generators, I don't believe PreparedRequest will ever be 100% pickleable but we have the common cases tested now.