search

twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.82k stars 2.73k forks source link

CRITICAL:root:twint.output:_output:CSV:Error:'user' object has no attribute 'url' #1094

Open irisdemented opened 3 years ago

irisdemented commented 3 years ago

Issue Template

Please use this template!

Initial Check

If the issue is a request please specify that it is a request in the title (Example: [REQUEST] more features). If this is a question regarding 'twint' please specify that it's a question in the title (Example: [QUESTION] What is x?). Please only submit issues related to 'twint'. Thanks.

Make sure you've checked the following:

I confirm that I have checked all of the above. There are similar issues regarding 'config' object and 'tweet' object, but unfortunately I have not been able to figure out a fix for my 'user' object issue.

Command Ran

Please provide the exact command ran including the username/search/code so I may reproduce the issue. username = ['TTP_updates', 'fullsunmyno1'] for u in username: name = u c = twint.Config() c.Username = name c.User_full = True c.Output = f"output.csv" c.Store_csv = True twint.run.Lookup(c)

Description of Issue

Please use as much detail as possible.

I grabbed some random usernames to duplicate what I was seeing with my list, which is basically that any user object that has no URL will not output to csv.

In the usernames above, the first one, TTP_updates, will output to csv, no problem. The second one will not.

This is the terminal output:

4894323374 | Tech Transparency Project | @TTP_updates | Private: False | Verified: False | Bio: The Tech Transparency Project (TTP) is a research initiative of @Accountable_Org that seeks to hold large technology companies accountable. | Location: Washington, D.C. | Url: https://t.co/b3Wtm4lFNu | Joined: 2016-02-11 21:28:33 UTC | Tweets: 3602 | Following: 605 | Followers: 3950 | Likes: 218 | Media: 708 | Avatar: https://pbs.twimg.com/profile_images/1235218504191791111/h79m07CJ_normal.jpg 'user' object has no attribute 'url' CRITICAL:root:twint.output:_output:CSV:Error:'user' object has no attribute 'url' 'user' object has no attribute 'url' [x] output._output 1097842897733378049 | titi | @fullsunmyno1 | Private: False | Verified: False | Bio: | Location: | Url: | Tweets: 4114 | Following: 517 | Followers: 66 | Likes: 10479 | Media: 258 | Avatar: https://pbs.twimg.com/profile_images/1320816756126932992/WdJljbF0_normal.jpg

I tried adding a try / except block around URL in user.py and was able to get the script to continue - but I could not get any csv output, unfortunately.

I looked at the user object for a failing one and ur['data']['user']['legacy'] does not have a ['url']. I think it should be able to output to csv still, just haven't figured out where I need to tweak it.

Any ideas?

Environment Details

Using Windows, Linux? What OS version? Running this in Anaconda? Jupyter Notebook? Terminal?

This is running on a server running Ubuntu 18.04. I am running via python3 script_name.py

ghost commented 3 years ago

Did you manage to figure this out? I'm encountering the same thing.

irisdemented commented 3 years ago

I have not yet figured it out - I stopped trying to get the user info and moved on to a new project. Would still lvoe some resolution though!

Did you manage to figure this out? I'm encountering the same thing.

batmanscode commented 3 years ago

@irisdemented @jomorrcode Try using the branch mentioned in #1002. I tried your sample code and it worked for me!

pip install --upgrade git+https://github.com/himanshudabas/twint.git@origin/fix-parser#egg=twint

BIC312 commented 3 years ago

@irisdemented @jomorrcode Try using the branch mentioned in #1002. I tried your sample code and it worked for me!

pip install --upgrade git+https://github.com/himanshudabas/twint.git@origin/fix-parser#egg=twint

I tried this branch and it seems the problem still exists. I am trying to scrape profiles of a short list of users. It works for the most of users, but for some, it gives me the following error:

CRITICAL:root:twint.get:User:'user'