search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.8k stars 2.72k forks source link

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) #915

Open linglin506 opened 4 years ago

linglin506 commented 4 years ago

This error will be reported from 0:00 on September 18, 2020. It is found through debug that the https://twitter.com/i/search/timeline page does not exist. Can you fix it?By changing the IP, it can be determined that Twitter is not banning the IP

CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs

sukioral commented 4 years ago

I experienced the same problem today as well. Yesterday was still fine.

zhixideyu commented 4 years ago

今天用就有事

hcanalesmx commented 4 years ago

I am having the same issue

bigslick87 commented 4 years ago

Glad to see I am not the only one. I just discovered this today and was excited to use it. I hope they can fix it ASAP

hj5992 commented 4 years ago

getting the same error!

gab-santi commented 4 years ago

Getting the same error as well. Hope it gets fixed ASAP

tmantynen commented 4 years ago

I guess Twitter has deleted https://twitter.com/i/search/timeline

Twint is not the only Twitter scraper having the same problem. See: https://github.com/jonbakerfish/TweetScraper/issues/92

Buaasinong commented 4 years ago

bad news

zhixideyu commented 4 years ago

我想Twitter已经删除了https://twitter.com/i/search/timeline

Twint并不是唯一遇到相同问题的Twitter抓取工具。请参阅：jonbakerfish / TweetScraper＃92

Does that mean the project is going to fail

yemen2016 commented 4 years ago

I am also having the same issue. The system was working fine yesterday! I hope the problem will be solved soon.

Sebastokratos42 commented 4 years ago

I guess Twitter has deleted https://twitter.com/i/search/timeline

Twint is not the only Twitter scraper having the same problem. See: jonbakerfish/TweetScraper#92

Problem also appears using GetOldTweets3.

shelu16 commented 4 years ago

https://twitter.com/search?f=live&q=bug%20min_retweets%3A100%20until%3A2020-09-18%20since%3A2020-09-11&src=typed_query but it can work can we do this search here

vishal7892 commented 4 years ago

Even I am facing the same issue. Any luck?

TryAndErr0r commented 4 years ago

We need to find a new endpoint. I loaded the site and saw this link New Api link?. It is important to note that if you copy the request and past it to notepad you can see that the request is provided with headers and you can see that it provides a x-csrf-token and x-guest-token which are required. You can find these tokens in the cookie fields "gt" and "ct0".

This is somewhat the official twitter api and you need a bearer token to access this. The bearertoken is provided in the https://abs.twimg.com/responsive-web/client-web/main.6b1bdbc5.js file. Just search for "AAAAAA" and you can find it.

Now you can craft a GET request in postman with the x-csrf-token, the x-guest-token and the bearer token for authentification.

yunusemrecatalcam commented 4 years ago

Booom! I found a new endpoint, implemented the new parser for it. Check this out! https://github.com/twintproject/twint/pull/917 @pielco11

estebanpdl commented 4 years ago

Having the same issue as well. Just tried yesterday (2020-09-17) and everything worked fine. I tried Twint today (2020-09-18) and I got the same loop.

CRITICAL:root:twint.get:User:
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 15 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 15 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 27.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 64.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 125.0 secs
CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0)
sleeping for 216.0 secs
boshez commented 4 years ago

I thought I was the only person who was facing this. Hope they fix this.

hasobi commented 4 years ago

i'm having the same problem here.

guilherme-sal commented 4 years ago

Same problem here. Yesterday was working fine...

jeremy1392 commented 4 years ago

Everything is working except to fetch tweets. Yesterday was working fine

ivanlewin commented 4 years ago

Hi I just want to add that I also had no problems yesterday but today I can't get any data.

Also, I was trying other tool to see if it worked and they seem to be having similar issues

aradiayu commented 4 years ago

I think I'm the only one facing this 'twint' problem. I hope it is fixed soon

Dbroug01 commented 4 years ago

I am also facing this issue

tmantynen commented 4 years ago

Booom! I found a new endpoint, implemented the new parser for it. Check this out! #917 @pielco11

Big thanks to @yunusemrecatalcam

I'm not using Twint but instead my own similar Twitter scraper program writen in PHP. So I can't comment directly on @yunusemrecatalcam 's changes to Twint.

Anyway. Couple of days ago my own scraper stopped working just like all Twitter scrapers using i/search/timeline endpoint did. Following @yunusemrecatalcam 's example, I implemented similar changes to my program that @yunusemrecatalcam did to Twint in PR #917.

I haven't had the time to finished all changes, but preliminary impressions are mixed. The new mobile endpoint appears to function similarly to the old one, but Twitter appears to return a lot fewer tweets as a result to queries than previously. Queries that resulted hundreds of tweets earlier now result just tens of tweets. I can't yet comment on speed or rate limits etc. And again, these are just preliminary impressions.

(Edit: I incorrectly tagged @pielco11 when I should have tagged @yunusemrecatalcam. My apologies.)

adrival13 commented 4 years ago

Same problem CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 15 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 15 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 27.0 secs CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 64.0 secs

rayhannr commented 4 years ago

got the same problem though fetching followers and followings are just fine. anyway, is there any good alternative to twint?

shizia commented 4 years ago

今天用就有事

以前是都能用吗？才刚发现这个就发现好像出问题了

lukaspistelak commented 4 years ago

they changed end point url to :+1:

https://twitter.com/explore/

akuma527 commented 4 years ago

they changed end point url to 👍

https://twitter.com/explore/

What do you mean exactly? How to get it to run?

estebanpdl commented 4 years ago

they changed end point url to 👍

https://twitter.com/explore/

isn't it /explore/ the URL for trending topics?

fwgood commented 4 years ago

今天用就有事

以前是都能用吗？才刚发现这个就发现好像出问题了

是的，我用了几个月了，前几天突然就不行了

fwgood commented 4 years ago

they changed end point url to 👍 https://twitter.com/explore/

isn't it /explore/ the URL for trending topics?

@estebanpdl “/explore” indeed is the trending topics，but you can use advanced search in this page. If you switch to the second tab you will fetch the same result as before. hopefully there are someone can write the new parser ASAP

zhixideyu commented 4 years ago

How on earth can I crawl my way to a tweet, which is important to me。

imfht commented 4 years ago

@zhixideyu looks like twitter api is a way.

zhixideyu commented 4 years ago

Please recommend a project that can use

lukaspistelak commented 4 years ago

for JS render here is nice lib :+1:

https://github.com/mherrmann/selenium-python-helium

rodolfopardo commented 4 years ago

Twint is broken? OMG

Any solution??

tahmidrashid commented 4 years ago

Twint is broken? OMG

Any solution??

Try this: https://github.com/twintproject/twint/pull/917

shadenshabayek commented 4 years ago

I am facing the same issue too!

ETedward commented 4 years ago

also still facing the same issue

DavidPerea commented 4 years ago

What is the solution for this?

simahashemi commented 4 years ago

the same for me. Is there any solution or do you know any other scrapper as good as Twint?

Wendy-Chang commented 4 years ago

same here! I'm struggling.

Tasaqi commented 4 years ago

same for me..

eabanoz commented 4 years ago

Hi, I would like to use twint package (2.1.20) with Python 3.8 via pycharm IDE and I have gotten 'CRITICAL:root:twint.run:Twint:Feed:noDataExpecting value: line 1 column 1 (char 0) sleeping for 1.0 secs'. I couldn't find a solution about it. PS: I have already read all posts which are related with this issue.

jiwoochoi-kor commented 4 years ago

I'm facing the same issue...

hnodaro commented 4 years ago

Any idea to resolve this issue ?

himanshudabas commented 4 years ago

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

hnodaro commented 4 years ago

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Thank you so much !!

estebanpdl commented 4 years ago

@eabanoz @jiwoochoi-kor @hnodaro Uninstall the previous installation of twint using the following command

pip3 uninstall twint

And then reinstall twint using the following command

pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint

This should work.

Does this version install the command-line tool as well?