search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.68k stars 2.71k forks source link

noData'globalObjects' #992

Open ghost opened 3 years ago

ghost commented 3 years ago

Initial Check

Command Ran

twint -g="48.880048,2.385939,1km" -o file.csv --csv

Description of Issue

I get the error: CRITICAL:root:twint.run:Twint:Feed:noData'globalObjects'

Environment Details

Linux (Kubuntu 20.04) Running on Jupyter Notebook and on Terminal I get the same error

coreyryanhanson commented 3 years ago

That's probably related to this one: https://github.com/twintproject/twint/issues/985

I've noticed that lately when inputting a location, there's either no results at all or completely unfiltered location results depending on what else is in the query.

ghost commented 3 years ago

Do you have a query that works with geoloc? I ll try it!

coreyryanhanson commented 3 years ago

For me I've noticed the switch from zero results to unfiltered comes when the query uses the or boolean operator. Ie. "this" - zero results "this or that" ungeofiltered results.

The raw query that get's generated (for los angeles) comes out to this: https://api.twitter.com/2/search/adaptive.json?include_can_media_tag=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&cursor=-1&spelling_corrections=1&ext=mediaStats%252ChighlightedLabel&tweet_search_mode=live&f=tweets&l=en&lang=en&q=%20geocode%3A34.0522%2C-118.2437%2C200kmthis%20OR%20that%20since%3A1603584000%20until%3A1603670400

Perhaps the bug is because the geolocation runs on the search term directly rather than having a space in between, It might be an easy fix.

ghost commented 3 years ago

I just run the command again, and it works... Could it be something from twitter? I didn't make any specific changes.

coreyryanhanson commented 3 years ago

That's weird. I don't think it would be twitter as that missing space was problematic.

I just made a tiny revision to the code and created a pull request for it. https://github.com/twintproject/twint/pull/996

ghost commented 3 years ago

Should I upgrade already, or does it take a while until it s checked?

coreyryanhanson commented 3 years ago

This is my first pull request with the project, so I have no clue how long it will take. In the meantime in the short term, if it's urgent, you can reinstall directly from my fork, by replacing the twint github url with https://github.com/coreyryanhanson/twint.git

Just be sure to go back to using the main project url when it is officially updated/fixed :)

himanshudabas commented 3 years ago

@coreyryanhanson hi, sorry I didn't read this thread earlier. Just saw you PR #996 This issue was earlier mentioned in #947 and I put up a patch for it in #955 (along with a number of features and fixes). This is the fix.

ghost commented 3 years ago

Thank you!

ghost commented 3 years ago

Did you get a similar error has the one mentioned in #970 (https://github.com/twintproject/twint/issues/970)? If the geoloc is a space, I would assume it is not related whatsoever to this other one, but...!

coreyryanhanson commented 3 years ago

@himanshudabas I completely missed that a fix was already in the works. Thanks! :)

coreyryanhanson commented 3 years ago

@micaelamaria Taking a look back at the code, it probably would have broken username too since it would be added right before geo and likely be affected by the missing space too. I haven't tested it though.

beedrumms commented 3 years ago

Hi there! was this issue ever resolved? I have tried to follow the conversation but it is not clear which solution worked! I'm having the same error. Happy to provide more details!