Associate arbitrary object to each request?

[waypointsoftware]

I need to capture the request and an instance of an object that contains meta-data about the request to store in the database, how do you recommend that? You only return the response text.

[rtdean]

The short answer is, is that there isn't an easy way to do that right now (read, no existing function you can hit, that provides this functionality).

There's a couple of slightly harder ways you can accomplish this.

You can write your own imap function:

def imap(requests, stream=False, size=2, exception_handler=None):
    pool = Pool(size)

    def send(r):
        return r.send(stream=stream)

    for request in pool.imap_unordered(send, requests):
        if request.response:
            yield request
        elif exception_handler:
            exception_handler(request, request.exception)

    pool.join()

The difference is instead of yielding request.response, you yield request (which is the AsyncRequest object). You could then use it like:

a = grequests.get('http://www.example.com/')
a.metadata = something

for result in imap([a]):
    r = result.result
    metadata = result.metadata

Alternatively, you could use a side dict to store your metadata in, and come back to it later, ala....

import uuid
from datetime import datetime
urls = [ 'http://www.example.com/', 'http://www.google.com/' ]
queue = []
metadata = {}
for url in urls:
    id = uuid.UUID4()
    metadata[id] = { sent: datetime.now() }
    queue.append(grequests.get(url, headers={'x-correlation-id': id}))

for result in grequests.imap(queue):
    corr_id = result.request.headers['x-correlation-id']
    meta = metadata[corr_id]
    # do stuff
    ...
    # clean up the metadata when done
    del metadata[corr_id]

[zuntah]

I'm using the '#' char to pass my data thru the requested url. Any string that comes after '#' means nothing to the request params so it's like sending it without your additional data.

It's useful when it's not much data.. You can specify only index and then find the metadata with the index that provided.

For example, lets say i'm want to send a request to this url: http://google.com I want to "attach" an object to it so I add the object index to the url: http://google.com#{object-id}

then, when i'm iterating the responses, im getting the id via the response.url attribute: index=response.url.split("#")[1] object_data = object_list[index]

Should solve your problem... :)

[tvmanikandan]

Hello, I am using grequests.map function with certain no of requests ( say 100). How do I ensure the response is in the same order of the requests made?

Please help.

-Mani

[treilly]

I needed to do the same thing. The approach I took was to modify the AsyncRequest class to support a metadata argument as follows:

class AsyncRequest(object):
    """ Asynchronous request.

    Accept same parameters as ``Session.request`` and some additional:

    :param session: Session which will do request
    :param callback: Callback called on response.
                     Same as passing ``hooks={'response': callback}``
    :param metadata: optional dict of metadata to return with the executed response object
    """
    def __init__(self, method, url, **kwargs):
        #: Request method
        self.method = method
        #: URL to request
        self.url = url
        #: Associated ``Session``
        self.session = kwargs.pop('session', None)
        if self.session is None:
            self.session = Session()

        callback = kwargs.pop('callback', None)
        if callback:
            kwargs['hooks'] = {'response': callback}

        # Save metadata (if any)
        self.metadata = kwargs.pop('metadata', None)

        #: The rest arguments for ``Session.request``
        self.kwargs = kwargs
        #: Resulting ``Response``
        self.response = None

    def send(self, **kwargs):
        """
        Prepares request based on parameter passed to constructor and optional ``kwargs```.
        Then sends request and saves response to :attr:`response`

        :returns: ``Response``
        """
        merged_kwargs = {}
        merged_kwargs.update(self.kwargs)
        merged_kwargs.update(kwargs)

        try:
            self.response = self.session.request(self.method,
                                                self.url, **merged_kwargs)
            # Save metadata to response
            self.response.metadata = self.metadata

        except Exception as e:
            self.exception = e
            self.traceback = traceback.format_exc()
        return self

If the response successfully executes this dict is available via the response.metadata property. If an exception is thrown it is still available in the request.metadata property

While this seems to work I have not done much testing. Opinions on validity of this solution are welcomed.

[spyoungtech]

Some good recipes exist for doing this and I don't think adding the feature directly is super helpful. Closing for now :)