Closed factorlive closed 3 years ago
It appears V2 has an inconsisitency with regards to pagination. Some endpoints use pagination_token
and some use next_token
. I'm inquiring whether this is intended before updating the code.
I got a response from Twitter. This is an inconsistency. So, I will have to amend the code.
Probably, the developers of different endpoints are not talking much with each other these days. 😏
It appears 2/tweets/search/recent
is the only endpoint that does not support pagination_token
. I'll have to build in an exception for this until Twitter fixes this.
they will probably make it consistent at some point. an exception would be the best way to catch a different name of the token, indeed.
same error message for 2/tweets/search/all
Will upload a fix tomorrow.
This is the change to TwitterPager.py
:
SEARCH_ENDPOINTS = ['tweets/search/recent', 'tweets/search/all']
pagination_token = 'next_token' if self.resource in SEARCH_ENDPOINTS else 'pagination_token'
if new_tweets:
self.params[pagination_token] = meta['previous_token']
else:
self.params[pagination_token] = meta['next_token']
With this change in TwitterPage.py
, the output shows it only works within the first page and not with the pager to go beyond the first page:
{'id': '1362538499401011200', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538321352794113', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538308094492675', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538285583728643', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362538143665242113', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537885799391234', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537838596743174', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537789972094979', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362537408806277120', 'text': '@MattWallace888 @TwitterDev The cover photo 😩😂😂🤣'}
{'id': '1362537131718176772', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
Traceback (most recent call last):
File "/home/patrick/icp-news/campaign/campaign/campaign.py", line 202, in <module>
for item in pager.get_iterator(new_tweets=False):
File "/home/patrick/icp-news/campaign/TwitterAPI/TwitterAPI/TwitterPager.py", line 123, in get_iterator
self.params[pagination_token] = meta['next_token']
UnboundLocalError: local variable 'pagination_token' referenced before assignment
Is after making the code change I suggested?
yes, my comment was not clear and I revised it. Got your repo now in editable mode.
What happens if you replace pagination_token = 'next_token' if self.resource in SEARCH_ENDPOINTS else 'pagination_token' with pagination_token = 'pagination_token'
Same problem.
Code TwitterPager.py
(...)
else: # VERSION 2
if new_tweets
SEARCH_ENDPOINTS = ['tweets/search/recent', 'tweets/search/all']
pagination_token = 'pagination_token'
# pagination_token = 'next_token' if self.resource in SEARCH_ENDPOINTS else 'pagination_token'
self.params[pagination_token] = meta['previous_token']
else:
self.params[pagination_token] = meta['next_token']
(...)
Output
{'id': '1362542072922734598', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362541974121631744', 'text': '@MattWallace888 @TwitterDev Tron has one as well trx'}
{'id': '1362541629945483267', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362541266412728331', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362540973960658951', 'text': '2021 NASCAR Cup Series DFS Power Rankings - Daytona Road Course - DFS Army https://t.co/FM2MkP2hyZ via @twitterdev'}
{'id': '1362540274031988738', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362540110475112449', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362539439227088900', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362539385019842560', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
{'id': '1362539261250035713', 'text': 'RT @MattWallace888: @TwitterDev https://t.co/YR2kOekDPb'}
Traceback (most recent call last):
File "/home/patrick/icp-news/campaign/campaign/campaign.py", line 202, in <module>
for item in pager.get_iterator(new_tweets=False):
File "/home/patrick/icp-news/campaign/TwitterAPI/TwitterAPI/TwitterPager.py", line 124, in get_iterator
self.params[pagination_token] = meta['next_token']
UnboundLocalError: local variable 'pagination_token' referenced before assignment
I see your problem. You didn't copy code quite correctly. You must define pagination_token before the if...else. The way you have it, pagination_token is undefined for the else clause.
Yep. Thanks. Pull request sent.
Got another enhancement, however, and will open another issue. Hope you don't mind 😁
Fixed in v2.6.7.
Thanks again for implementing an enhancement based on an issue ( #167).
Unfortunately, there is a knock-on effect on the following API v2 endpoint:
Code
Output and error: