twint crashes: The module usage examples from Readme don't work

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

MIT License

15.66k stars 2.72k forks source link

twint crashes: The module usage examples from Readme don't work #1019

Open DataGreed opened 3 years ago

DataGreed commented 3 years ago

Make sure you've checked the following:

[x] Python version is 3.6;
[x] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
[x] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

import twint

# Configure
c = twint.Config()
c.Username = "realDonaldTrump"
c.Search = "great"

# Run
twint.run.Search(c)

Description of Issue

Crashes with

CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)

Environment Details

macOS terminal, python 3.6

AsmaZbt commented 3 years ago

hello, I have the same issue, I'm working on windows. did you solve the problem, please?

thank you

DataGreed commented 3 years ago

@AsmaZbt nope, I gave up on twint. Seems like it has been an issues since August when twitter changed something in their APIs.

I've wrote a custom scraping solution for my specific use-case.

himanshudabas commented 3 years ago

@DataGreed @AsmaZbt You must have installed twint using

pip install twint

try installing using thr following command, and this should work

pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Twint hasn't been updated on pypi for a while now, that's why the first method doesn't work.

AsmaZbt commented 3 years ago

@DataGreed @AsmaZbt You must have installed twint using
pip install twint
try installing using thr following command, and this should work
pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint
Twint hasn't been updated on pypi for a while now, that's why the first method doesn't work.

hello @himanshudabas thank you so much, the command:

pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

solved the issue, twint work now for me, but only on Jupyter Notebook and it doesn't work on Spyder, do have an idea, what can be the problem?

best regards

himanshudabas commented 3 years ago

@AsmaZbt I haven't Personally tested twint on Spyder. Can you share the script that you are running on Spyder along with the environment of Spyder and error that you get while running twint on Spyder.

AsmaZbt commented 3 years ago

@AsmaZbt I haven't Personally tested twint on Spyder. Can you share the script that you are running on Spyder along with the environment of Spyder and error that you get while running twint on Spyder.

I fixed the issue with : Reset Spyder setting Capture d’écran 2020-12-03 182822

my script is very basic like this : c = twint.Config() c.Username = "shugairi" c.Limit = 1

twint.run.Search(c)

and I did not get any error message, it gives nothing in a loop while.

thank you so much @himanshudabas for the HELP ^_^

best regards

nate8020 commented 3 years ago

@DataGreed what did you use for your custom scraping solution?

I'm getting bitten by CRITICAL:root:twint.feed:Follow:IndexError found on issues #1127, #1101, #1077, #1011, #893 and #839...

DataGreed commented 3 years ago

@nate8020 I just checked the xhr requests on twitter page, and created a script that automatically loads new pages using requests library. I don't remember all the details though as it was a one-time research, I probably re-used some iteration logic from twint.

paulowe commented 3 years ago

@DataGreed what did you use for your custom scraping solution?

I'm getting bitten by CRITICAL:root:twint.feed:Follow:IndexError found on issues #1127, #1101, #1077, #1011, #893 and #839...

Did you figure a way around this? I get the same error on CLI when running twint -u username --followers

nate8020 commented 3 years ago

@paulowe I haven't figured out a solution to this yet. I ended up using the official Twitter API as it was enough for my requirements.

paulowe commented 3 years ago

@paulowe I haven't figured out a solution to this yet. I ended up using the official Twitter API as it was enough for my requirements.

Same here

diegokenned commented 3 years ago

@DataGreed o que você usou para sua solução de raspagem personalizada? Estou sendo mordido por CRITICAL:root:twint.feed:Follow:IndexErrorencontrado nos problemas # 1127 , # 1101 , # 1077 , # 1011 , # 893 e # 839 ...

Você descobriu uma maneira de contornar isso? Recebo o mesmo erro na CLI ao executartwint -u username --followers

did you find any solution?