iSarabjitDhiman / TweeterPy

TweeterPy is a python library to extract data from Twitter. TweeterPy API lets you scrape data from a user's profile like username, userid, bio, followers/followings list, profile media, tweets, etc.
MIT License
138 stars 20 forks source link

can't get twitter list from search method #2

Closed freedom-wy closed 1 year ago

freedom-wy commented 1 year ago

i use tweeterpy request twitter search

from tweeterpy import TweeterPy
from tweeterpy import config

config.PROXY = {"http": "127.0.0.1:7891", "https": "127.0.0.1:7891"}
config.TIMEOUT = 10

twitter = TweeterPy()
data = twitter.search(search_query="hello world")
print(data)

response is none list

'content-type'

'content-type'
{'data': [], 'cursor_endpoint': None, 'has_next_page': True}
iSarabjitDhiman commented 1 year ago

Twitter recently put restrictions on the guest sessions, meaning you can't fetch data without log Ins anymore. If you try to open twitter without login, they will redirect you to the login. So you gonna have to use a logged in (authenticated) session before you start using TweeterPy. Check Login feature to use a logged in session. Also, make sure you are using the actual proxies OR you can comment the proxy code if you don't want to use any proxies.

Example Code:

from tweeterpy import TweeterPy

twitter = TweeterPy()
twitter.login("username/email","password")
twitter.save_session() # Save session if you don't want to login everytime you use the tool again.
data = twitter.search(search_query="hello world",total=50)

If you saved a session. Next time you use the tool again, you don't have to use twitter.login() method again. You can load saved session with twitter.load_session() Check example below

from tweeterpy import TweeterPy

twitter = TweeterPy()
twitter.load_session()
data = twitter.search(search_query="hello world",total=50)
freedom-wy commented 1 year ago
from tweeterpy import TweeterPy

from tweeterpy import config

config.PROXY = {"http": "127.0.0.1:7891", "https": "127.0.0.1:7891"}

twitter = TweeterPy()
twitter.login("XXXX", "XXXX")
twitter.save_session()  # Save session if you don't want to login everytime you use the tool again.
data = twitter.search(search_query="hello world", total=50)
print(data)

response

API Updated Successfully.
Successfully Logged In..
Expecting value: line 1 column 2 (char 1)
Expecting value: line 1 column 2 (char 1)
{'data': [], 'cursor_endpoint': None, 'has_next_page': True}
iSarabjitDhiman commented 1 year ago
from tweeterpy import TweeterPy

from tweeterpy import config

config.PROXY = {"http": "127.0.0.1:7891", "https": "127.0.0.1:7891"}

twitter = TweeterPy()
twitter.login("XXXX", "XXXX")
twitter.save_session()  # Save session if you don't want to login everytime you use the tool again.
data = twitter.search(search_query="hello world", total=50)
print(data)

response

API Updated Successfully.
Successfully Logged In..
Expecting value: line 1 column 2 (char 1)
Expecting value: line 1 column 2 (char 1)
{'data': [], 'cursor_endpoint': None, 'has_next_page': True}

That's weird. Its working just fine on my end. Have you tried using it without the proxy as I showed above? If it works without the proxy then its your proxy. Also let me know what type of proxies are you using( datacenter, residentails)? If it still doesn't work, let me know , I will look into it.

iSarabjitDhiman commented 1 year ago

Hey, thanks @codilau @freedom-wy for drawing my attention to this bug. After testing it on a different machine, I realized it had some issues parsing the web response. Adding Brotli library to the package as a requirement fixed the bug.

Committing changes in a short while, please install the latest build.