search

taspinar / twitterscraper

Scrape Twitter for Tweets
MIT License
2.39k stars 581 forks source link

Twitter returned : 'has_more_items' #305

Closed zhicheng0501 closed 4 years ago

zhicheng0501 commented 4 years ago

I have using twitterscraper to pull twitter data. Every time it ends with Twitter returned:"has_more_items" even it pulled all the data already. Does anyone else see the same problem?

zhicheng0501 commented 4 years ago

this is where the result comes from. has_more_items = json_resp['has_more_items'] if not has_more_items: logger.info("Twitter returned : 'has_more_items' ") I think the code above means that it actually got all the data. Am i right?

Kai292-tech commented 4 years ago

I met the same output. Although I guess it should be the signal for finishing download the data, the number of output tweets is still smaller than actual number. Does anyone know why and how to refine it?

lapp0 commented 4 years ago

@zhicheng0501 yes, that log line is misleading. I incorporated a fix into an existing PR https://github.com/taspinar/twitterscraper/pull/302/commits/8786a5789f8686d4022f14ff44b7ffd86a5abd11