[Problem] How to fetch another batch of tweets search?

[ryuujo1573]

Thank you a lot for your work, which saved my life from being wasted. I ran into some problem. emmmm code first...

async def main():
    tweets = t.search.tweets(q='china poverty', tweet_mode='extended')
    while 'next_results' in tweets['search_metadata']:
        await save(tweets)
        from urllib import parse

        # Parse url like '?ct=32&op=92&item=98' to dict like { 'ct': 32, 'op': 92, ...} then pass them into the request
        tweets = t.search.tweets(**dict(parse.parse_qsl(tweets['search_metadata']['next_results'][1:])))
    await save(tweets)
    pass

From Twitter API docs giving Example ResponseI know that I can extend my request to another batch result by using param $.search_metadata.next_results directly, but it seems it's not the correct way to do this, which feels like boxing and unboxing

What should I do?

[boogheta]

Hello @ryuujo1573 Indeed the next_results url field is not very useful with this library as it builds the urls for you. What you do should work and I don't see anything incorrect although I agree it does not look very good :) If you look at it closely you will notice it's exactly the same args you passed to your first call, with just "max_id=SOMETWEETID" added within. So what I usually do and would recommand is rather to work with an args dict that you can reuse (notice the count argument to get more tweets by call):

async def main():
    args = {'q': 'china poverty', 'tweet_mode': 'extended', 'count': 100}
    while 'max_id' not in args or len(tweets['statuses']):
        tweets = t.search.tweets(**args)
        await save(tweets)
        args['max_id'] = tweets['search_metadata']['max_id']

[ryuujo1573]

wow, thanks! this looks elegant. closing issue, thank you again xd