[REQUEST] twint.get:User:'NoneType' object is not subscriptable

Laura2797 commented 4 years ago

I've checked the following:

[] Python version is 3.6;
[] Updated Twint with pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
[] I have searched the issues and there are no duplicates of this issue/question/request.

Description of Issue

As commented in the Issue #777 the same problem still there, so I reopen the issue. The error: "twint.get:User:'NoneType' object is not subscriptable" appears again, but yesterday was working for me. I also update the Twint with: pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint

Environment Details

Linux Anaconda Terminal

aksub99 commented 4 years ago

I'm facing the same issue since today morning. Everything was fine until yesterday.

153 commented 4 years ago

Experiencing same issue

rifkiachmads commented 4 years ago

i''m also experiencing the same issue. i've tried to run the code from windows & mac

ghost commented 4 years ago

Twint is not functioning correctly at this time, probably due to Twitter site changes on June 1st. Hopefully there will be a Twint patch soon.

Edit: 18hrs later, now working for me.

o7n commented 4 years ago

The patched code (with X-Requested-With: XMLHttpRequest) is now returning a 400 Bad Request, it seems like Twitter has again changed something today.

nathmo commented 4 years ago

Same for me. Hope this tool will be fixed soon :/

pboettch commented 4 years ago

Is there a coordinated attempt to find a (new) solution to this problem?

jfrancoisponcet16 commented 4 years ago

Hi, twitter just changed back its last update so the headers patch in the function "User" in get.py should work now (that was the patch ): response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) made by pielco11 If you have the latest version of python it should work

rifkiachmads commented 4 years ago

Hi @jfrancoisponcet16!

i've tried it again just now, and mine still returning CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

i've updated my twint using the command pip3 install --user --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;

i am using python 3.6 in windows and mac env

do you have any solution to my problem?

o7n commented 4 years ago

It's indeed working again @rifkiachmads ...

aksub99 commented 4 years ago

Yes, it's working for me too.

rifkiachmads commented 4 years ago

its working fine now. thanks guys!

update: its working in my mac but it doesnt on my windows

Laura2797 commented 4 years ago

Is working for me now on Linux, but I didnt change anything. I hope the same error doesn't appear again.

jfrancoisponcet16 commented 4 years ago

Hi, If you have the latest version of Twint, this should work fine the patch was already integrated in it. The operating system should not affect the way your getting the data from twitter. Requests to twitter are made using fake_useragent so there is not much useful information twitter servers can get from your machine to block you (it doesn't see that you are using python)

EvanUp commented 4 years ago

twint.run.Search() and twint.run.Lookup() are both working after updating twint, but twint.run.Followers() and twint.run.Following() are both returning errors or timing out. It doesn't seem to be an IP issue.

--REPREX-- a = twint.Config() a.Username = 'ladygaga' a.Limit = 10 a.User_full = True a.Pandas = True twint.run.Followers(a)

pandas 1.0.3 twint 2.1.20

jfrancoisponcet16 commented 4 years ago

Hi @EvanUp , Indeed I see a CRITICAL:root:twint.feed:Follow:IndexError however I still get the name of followers. Not sure why yet, I will look into it.
The reason why you may have a problem with saving followers with panda might be because of issue#173.

An update was added and this is how it should work : a = twint.Config() a.Username = 'ladygaga' a.Limit = 10 a.User_full = True a.Pandas = True twint.run.Followers(a) follow_df = twint.storage.panda.Follow_df

Otherwise you can store the followers in a list and save the list item with pandas: follow_list = [] def get_followers(): a = twint.Config() a.Username = 'ladygaga' a.Store_object = True twint.run.Followers(a) follow_list = twint.output.follows_list

get_followers()

Then save the list items (followers' names) where you want.

EvanUp commented 4 years ago

Thank you @jfrancoisponcet16 !

It seems my issue was that nest_asyncio was not up to date. I tried updating nest_asyncio and follower pulls are working fine now. Thank you again for the help!

mripani commented 4 years ago

Hello, I'm experiencing the same issue, till yesterday was working but not today. I run the following code, and I get the 'twint.get:User:'NoneType' object is not subscriptable' error. The code I run:

c = twint.Config()
c.Limit = 10
c.Username = 'MaxiRipani'
c.User_full = True
c.Pandas = True
c.Hide_output = True
twint.run.Followers(c)
prueba = twint.storage.panda.Follow_df

I'm using google colab, and before run everything I run:

!pip3 install twint !pip install nest_asyncio

DV777 commented 4 years ago

Is the error "twint.get:User:'NoneType' object is not subscriptable" appearing again for you guys ?

agombert commented 4 years ago

response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) made by pielco11

@jfrancoisponcet16 thx, it solved the problem for me !

DV777 commented 4 years ago

At the moment I keep on getting this "twint.get:User:'NoneType' object is not subscriptable" error. Here is my code :

#summary
c = twint.Config() 
c.Store_csv = True 
c.Stats = True 
c.Count = True  
c.User_full = True
c.Hide_output = True 
c.Location = True 
for i in range(len(df)): 
    c.Username = df.loc[i,'username']  
    c.Output = df.loc[i, 'username'] + '_summary_june2020.csv' 
    twint.run.Lookup(c)

jfrancoisponcet16 commented 4 years ago

Hi @DV777 If you just want to store a CSV file with look up info, this is how you'd do it : c = twint.Config() for i in names_BF: c.Store_csv = True c.Output = "User_BF" c.Username = i twint.run.Lookup(c)

Configurations that you get in config.py only apply tweet.py while twint.run.Lookup(c) fires the function Lookup(config) at run.py where config is looking just for a tweeter username (if function in get.py gets something else it throws a none type. c.Count = True its useless in this case as this counts Tweets from from twint.run.Search(c) in run.py which focuses on methods at twee.py Keep the config within a function or a loop to avoid problems :) Hope that helps

skywd commented 4 years ago

嗨，twitter刚刚改回了其最后一次更新，因此get.py中的“ User”函数中的标头补丁现在应该可以工作了（即该补丁）： response = await Request（url，connector = _connector，headers = {“ X- pielco11提出的Requested-With“：” XMLHttpRequest“}）如果您使用的是最新版本的python，它应该可以工作

I use response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) in windows but I still get an error

vijayrawatsan commented 4 years ago

I have updated the twint version but still have this error. Can someone help here?

DV777 commented 4 years ago

Hi @DV777 If you just want to store a CSV file with look up info, this is how you'd do it : c = twint.Config() for i in names_BF: c.Store_csv = True c.Output = "User_BF" c.Username = i twint.run.Lookup(c)

Configurations that you get in config.py only apply tweet.py while twint.run.Lookup(c) fires the function Lookup(config) at run.py where config is looking just for a tweeter username (if function in get.py gets something else it throws a none type. c.Count = True its useless in this case as this counts Tweets from from twint.run.Search(c) in run.py which focuses on methods at twee.py Keep the config within a function or a loop to avoid problems :) Hope that helps

Thanks ! My code was working a couple months ago (I am a true beginner) so I guess your version makes it more sound & efficient :) but the current error remains.

Cheryl520 commented 4 years ago

Is the problem for getting user profile appearing again? 'User:'NoneType' object is not subscriptable' appeared for me again while a few weeks ago it worked.

jfrancoisponcet16 commented 4 years ago

hi @DV777 Looking at my message again, I don't think I was really clear with regards to the core functioning of twint. However it should normally work. I know I kept the twint version 2,1,18 and added {"X-Requested-With": "XMLHttpRequest"} for the headers in the functio User in get.py. I tested the code and I do get the basic infos. @Cheryl520 Twitter still seems to accept AJAX request as it still allows XMLHttpRequest. Which twint version do you use ? Whats your code I'm not sure why its not working for other of you, but I know that in post twint version 2,1,18 the get.py doc was modified, I'm gonna look into it.

bushjavier commented 4 years ago

for my is working this example of my code.

c.Username = "meomeo127"

twint.run.Lookup(c)

for anyone that have some problems using twint while getting user data, use the last version in the repository.

YikSanChan commented 4 years ago

@bushjavier Hi, I am using the latest version of twint but I still have the exact issue.

$ conda list | grep twint
# packages in environment at /Users/evan/anaconda3/envs/twint-playground:
twint                     2.1.20                   pypi_0    pypi
$ python
Python 3.7.7 (default, May  6 2020, 04:59:01)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

Do I miss anything?

Cheryl520 commented 4 years ago

hi @DV777 Looking at my message again, I don't think I was really clear with regards to the core functioning of twint. However it should normally work. I know I kept the twint version 2,1,18 and added {"X-Requested-With": "XMLHttpRequest"} for the headers in the functio User in get.py. I tested the code and I do get the basic infos. @Cheryl520 Twitter still seems to accept AJAX request as it still allows XMLHttpRequest. Which twint version do you use ? Whats your code I'm not sure why its not working for other of you, but I know that in post twint version 2,1,18 the get.py doc was modified, I'm gonna look into it.

I changed to another computer and it works now. Not sure what happened. Thanks for the reply!

fengkaijia commented 4 years ago

The AJAX header should solve get.User() but won't help on get.Username() (#798), can confirm c.Username works, but c.User_id not. Maybe we should move on to #798.

DV777 commented 4 years ago

hi @DV777 Looking at my message again, I don't think I was really clear with regards to the core functioning of twint. However it should normally work. I know I kept the twint version 2,1,18 and added {"X-Requested-With": "XMLHttpRequest"} for the headers in the functio User in get.py. I tested the code and I do get the basic infos. @Cheryl520 Twitter still seems to accept AJAX request as it still allows XMLHttpRequest. Which twint version do you use ? Whats your code I'm not sure why its not working for other of you, but I know that in post twint version 2,1,18 the get.py doc was modified, I'm gonna look into it.

I really did not understand this one. Only thing is: using usernames to look up accounts seems not to work anymore with twint 2.1.20

c = twint.Config() 
for i in range(df):
    c.Store_csv = True 
    c.User_full = True 
    c.Username = '@'+df.loc[i,'my_usernames_column']
    c.Output = df.loc[i,'my_usernames_column']+'_summary.csv' 
    twint.run.Lookup(c)

CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

bushjavier commented 4 years ago

@bushjavier Hi, I am using the latest version of twint but I still have the exact issue.

$ conda list | grep twint
# packages in environment at /Users/evan/anaconda3/envs/twint-playground:
twint                     2.1.20                   pypi_0    pypi
$ python
Python 3.7.7 (default, May  6 2020, 04:59:01)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

Do I miss anything?

I just tested in a google cloud server, and in my home i live in south america and with both IPs worked

python
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
151928509 | meo blek | @meomeo127 | Private: 0 | Verified: 0 | Bio:  | Location:  | Url:  | Joined: 4 Jun 2010 9:17 AM | Tweets: 3614 | Following: 42 | Followers: 41 | Likes: 0 | Media: 0 | Avatar: https://abs.twimg.com/sticky/default_profile_images/default_profile_400x400.png
>>>

DV777 commented 4 years ago

@bushjavier Hi, I am using the latest version of twint but I still have the exact issue.

$ conda list | grep twint
# packages in environment at /Users/evan/anaconda3/envs/twint-playground:
twint                     2.1.20                   pypi_0    pypi
$ python
Python 3.7.7 (default, May  6 2020, 04:59:01)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

Do I miss anything?

I just tested in a google cloud server, and in my home i live in south america and with both IPs worked

python
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
151928509 | meo blek | @meomeo127 | Private: 0 | Verified: 0 | Bio:  | Location:  | Url:  | Joined: 4 Jun 2010 9:17 AM | Tweets: 3614 | Following: 42 | Followers: 41 | Likes: 0 | Media: 0 | Avatar: https://abs.twimg.com/sticky/default_profile_images/default_profile_400x400.png
>>>

I tested exactly the same with Python 3.7.7 & Twint's last version and still got the error message. How come ?

DV777 commented 4 years ago

@bushjavier Hi, I am using the latest version of twint but I still have the exact issue.

$ conda list | grep twint
# packages in environment at /Users/evan/anaconda3/envs/twint-playground:
twint                     2.1.20                   pypi_0    pypi
$ python
Python 3.7.7 (default, May  6 2020, 04:59:01)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

Do I miss anything?

I just tested in a google cloud server, and in my home i live in south america and with both IPs worked

python
Python 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import twint
>>> c = twint.Config()
>>> c.Username = "meomeo127"
>>> twint.run.Lookup(c)
151928509 | meo blek | @meomeo127 | Private: 0 | Verified: 0 | Bio:  | Location:  | Url:  | Joined: 4 Jun 2010 9:17 AM | Tweets: 3614 | Following: 42 | Followers: 41 | Likes: 0 | Media: 0 | Avatar: https://abs.twimg.com/sticky/default_profile_images/default_profile_400x400.png
>>>

I tested exactly the same with Python 3.7.7 & Twint's last version and still got the error message. How come ?

Ok, after reading more topics, it seems that this error is due to the last update of Twitter & that people working on Twint may be able to come up with a solution in the coming weeks / months but lack time at the moment. I hope this is more or less accurate...

jfrancoisponcet16 commented 4 years ago

hi @DV777 Looking at my message again, I don't think I was really clear with regards to the core functioning of twint. However it should normally work. I know I kept the twint version 2,1,18 and added {"X-Requested-With": "XMLHttpRequest"} for the headers in the functio User in get.py. I tested the code and I do get the basic infos. @Cheryl520 Twitter still seems to accept AJAX request as it still allows XMLHttpRequest. Which twint version do you use ? Whats your code I'm not sure why its not working for other of you, but I know that in post twint version 2,1,18 the get.py doc was modified, I'm gonna look into it.

I really did not understand this one. Only thing is: using usernames to look up accounts seems not to work anymore with twint 2.1.20
c = twint.Config() 
for i in range(df):
    c.Store_csv = True 
    c.User_full = True 
    c.Username = '@'+df.loc[i,'my_usernames_column']
    c.Output = df.loc[i,'my_usernames_column']+'_summary.csv' 
    twint.run.Lookup(c) 
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable

No worries @DV777. I get the same problem with the version 2.1.20 (the latest version). What I did instead, was to go back to the version 2.1.18. The reason is because in later versions the Lookup storing list meant to store the data, returns a NoneType (not sure why). Once I set the twint version 2.1.18 in a virtual environment , I went through the twint files, got the file get.py , then found function User and I changed the response to response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) Now it work fine for me. Twitter has not changed its Ajax response so it should work fine if you are using usernames. Hope that helps

DV777 commented 4 years ago

hi @DV777 Looking at my message again, I don't think I was really clear with regards to the core functioning of twint. However it should normally work. I know I kept the twint version 2,1,18 and added {"X-Requested-With": "XMLHttpRequest"} for the headers in the functio User in get.py. I tested the code and I do get the basic infos. @Cheryl520 Twitter still seems to accept AJAX request as it still allows XMLHttpRequest. Which twint version do you use ? Whats your code I'm not sure why its not working for other of you, but I know that in post twint version 2,1,18 the get.py doc was modified, I'm gonna look into it.

I really did not understand this one. Only thing is: using usernames to look up accounts seems not to work anymore with twint 2.1.20
c = twint.Config() 
for i in range(df):
    c.Store_csv = True 
    c.User_full = True 
    c.Username = '@'+df.loc[i,'my_usernames_column']
    c.Output = df.loc[i,'my_usernames_column']+'_summary.csv' 
    twint.run.Lookup(c) 
CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable
No worries @DV777. I get the same problem with the version 2.1.20 (the latest version). What I did instead, was to go back to the version 2.1.18. The reason is because in later versions the Lookup storing list meant to store the data, returns a NoneType (not sure why). Once I set the twint version 2.1.18 in a virtual environment , I went through the twint files, got the file get.py , then found function User and I changed the response to response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) Now it work fine for me. Twitter has not changed its Ajax response so it should work fine if you are using usernames. Hope that helps

I did exactly that - created my own virtual environment, installed twint & all, modified "response" in User - inside "get.py" as you mentioned... and it works ! Thanks a lot 👍

shreyagopal commented 4 years ago

Hello, I'm experiencing the same issue, till yesterday was working but not today. I run the following code, and I get the 'twint.get:User:'NoneType' object is not subscriptable' error. The code I run:
c = twint.Config()
c.Limit = 10
c.Username = 'MaxiRipani'
c.User_full = True
c.Pandas = True
c.Hide_output = True
twint.run.Followers(c)
prueba = twint.storage.panda.Follow_df
I'm using google colab, and before run everything I run:

!pip3 install twint !pip install nest_asyncio

@mripani Even I tries scraping the followers of a Twitter user using Google Colab and have the same issue. The code is working fine for Search and Lookup but gives the error for Followers and Following.

Did you find the workaround for this? Please let me know.

agombert commented 4 years ago

Hi, I'm experiencing again the error:

I have this function:

def twint_normal(us):
    c = twint.Config()
    c.Username = us
    c.Custom["user"] =  ["id", "username", "join_date",  "tweets", "following", "followers", "likes"]
    c.Output = "username.csv"
    c.Hide_output = True
    c.Store_csv = True
    twint.run.Lookup(c)

I already changed the line 182 with response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) and 163 with response = await Request(url, headers={"X-Requested-With": "XMLHttpRequest"}) as suggested by @jfrancoisponcet16 , and it solved the pb (cf: my comment on june 17th)

But since yersteday the problem has come back :/.

I came back to version 2.1.18, changed again the lines 163 and 182, same problem, I use VPN and tried it on an other macOS machine, same problem, thus I imagine they have not blocked my mac adress.

skywd commented 4 years ago

Hi, I'm experiencing again the error:

I have this function:
def twint_normal(us):
    c = twint.Config()
    c.Username = us
    c.Custom["user"] =  ["id", "username", "join_date",  "tweets", "following", "followers", "likes"]
    c.Output = "username.csv"
    c.Hide_output = True
    c.Store_csv = True
    twint.run.Lookup(c)
I already changed the line 182 with response = await Request(url, connector=_connector, headers={"X-Requested-With": "XMLHttpRequest"}) and 163 with response = await Request(url, headers={"X-Requested-With": "XMLHttpRequest"}) as suggested by @jfrancoisponcet16 , and it solved the pb (cf: my comment on june 17th)

But since yersteday the problem has come back :/.

I came back to version 2.1.18, changed again the lines 163 and 182, same problem, I use VPN and tried it on an other macOS machine, same problem, thus I imagine they have not blocked my mac adress.

There may be a problem with XMLHttpRequest, and other twitter crawlers also have the same problem

agombert commented 4 years ago

There may be a problem with XMLHttpRequest, and other twitter crawlers also have the same problem

What do you reckon we should use?

paulej commented 4 years ago

Twitter is apparently rejecting requests with the X-Requested-With: XMLHttpRequest header. Just a basic curl query like this fails:

curl -v -H 'X-Requested-With: XMLHttpRequest' 'https://twitter.com/jack' >/dev/null

It returns a 400, whereas excluding that header returns a 200. And this same query used to return 200 before. Something definitely changed on Twitter's backend, but I'm not sure what it is.

Kikobeats commented 4 years ago

looks like the X-Requested-With: XMLHttpRequest thing is no longer working 😢

tito421 commented 4 years ago

I keep getting this error. Has someone found a solution? Is there anything I can do to help fix it? For what it's worth. I tracked down the url Response that seems to be causing this issue, the functions that get called are

get.User get.Request get.Response

and I got the following response(soup.text):

We've detected that JavaScript is disabled in your browser. Would you like to proceed to legacy Twitter?
Yes
Something went wrong, but don’t fret — let’s give it another shot.

Now, I don't really know what to do with this. Maybe if someone could point out a path to fix it, I could try Cheers!

jfrancoisponcet16 commented 4 years ago

@agombert Indeed Twitter X-Requests have changed. I'm gonna look into it. Best, Jean FP

rjay98 commented 3 years ago

Any updates here? @jfrancoisponcet16

laz08 commented 3 years ago

Has anyone found a workaround for this? I'm also stuck on this :(

helloqwerasdf commented 3 years ago

hi!has anyone solve this problem? 400 bad requests because of headers?

Waffa commented 3 years ago

I first thought its related to private fallow only accounts.

but yes, the twint -u username --following -o amirtaaki_fallowing.txt still gives: CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable CRITICAL:root:twint.feed:Follow:IndexError

Developers should make some good friends in Twitter Development section, besides many are a bit pissed off about what is going on.. so there might come a lot of help about it, or from former developers who dare off the record share things that lawyers will allow not.

bushjavier commented 3 years ago

I first thought its related to private fallow only accounts.

but yes, the twint -u username --following -o amirtaaki_fallowing.txt still gives: CRITICAL:root:twint.get:User:'NoneType' object is not subscriptable CRITICAL:root:twint.feed:Follow:IndexError

Developers should make some good friends in Twitter Development section, besides many are a bit pissed off about what is going on.. so there might come a lot of help about it, or from former developers who dare off the record share things that lawyers will allow not.

@Waffa that is not working right now use:

twint -s "from:username" replaces twint -u username, works as expected

that was answered read:

https://github.com/twintproject/twint/issues/863

twintproject / twint