Allow me to scrape by username and since, until dates - Add from_date and to_date functionality

[ihabpalamino]

hello can i have example pf code that would allow me to scrape by username and since until dates the tweets and their number of likes retweet comments and url

[iSarabjitDhiman]

Hey there ! Here is an example :

from twitter import TweeterPy
twitter = TweeterPy()

twitter.login("username","password")
# OR generate session manually with auth-token (if if you're unable to log in)
# twitter.generate_session(auth_token="auth_token_here")

# set with_replies to True if you are also interested in tweet replies, if you only need tweets, leave it as it is. Feel free to see total number of tweets you want to get, but default it gets all if set to None.
user_tweets = twitter.get_user_tweets("username_here", with_replies=False, total=None)

If you want to reuse the logged in session in future without logging in again, check #2

Check Documentation for more use cases/examples.

[ihabpalamino]

while login i got this error Traceback (most recent call last): File "C:\Users\HP Probook\PycharmProjects\firstproject\TikTokScrap.py", line 55, in twitter.login("","") File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\tweeterpy\tweeterpy.py", line 189, in login TaskHandler().login(username, password) File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\tweeterpy\login_util.py", line 91, in login task_id = [task_id for task_id in task_flow_mapper.keys() if task_id in tasks][0] IndexError: list index out of range

[iSarabjitDhiman]

Seems like you might have two factor authentication enabled or twitter may have detected it as a suspicious login as it's not from your regular browser.

Solution : Use auth-token to generate a session.

How to get auth-token : Go to your browser, login into twitter if not already. Then go to developer tools, (google : how to open developer tools in chrome(your browser name). From there you should see the application menu, In the application menu, copy auth-token from the cookies section from the left panel.

Use auth-token to generate a session :

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.generate_session(auth_token="auth token you copied")

# Now you are good to go, add the other code below as you would normally.
user_tweets = twitter.get_user_tweets("username",total=5) 

# you can save the session if you don't want to do it every time you run the script 
# To save the session
twitter.save_session()

# If you saved a session, next time you will start as follows

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.load_session()

If there is still some issue, let me know.

[huulanka]

@ihabpalamino

Unfortunately, it appears that in your comment https://github.com/iSarabjitDhiman/TweeterPy/issues/6#issuecomment-1634466922, you inadvertently shared your username and password publicly. This poses a significant security risk as it exposes your personal account to potential unauthorized access or misuse. It's crucial to act swiftly and remove this sensitive information as soon as possible to protect your account and maintain your online security.

Thank you for your prompt attention to this issue. Stay safe and secure!

Kind regards, Andi

[ihabpalamino]

Seems like you might have two factor authentication enabled or twitter may have detected it as a suspicious login as it's not from your regular browser.

Solution : Use auth-token to generate a session.

How to get auth-token : Go to your browser, login into twitter if not already. Then go to developer tools, (google : how to open developer tools in chrome(your browser name). From there you should see the application menu, In the application menu, copy auth-token from the cookies section from the left panel.

Use auth-token to generate a session :

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.generate_session(auth_token="auth token you copied")

# Now you are good to go, add the other code below as you would normally.
user_tweets = twitter.get_user_tweets("username",total=5) 

# you can save the session if you don't want to do it every time you run the script 
# To save the session
twitter.save_session()

# If you saved a session, next time you will start as follows

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.load_session()

If there is still some issue, let me know.

is there any other way to login or get the token without doing it manually? @huulanka thank u for making me noticed it

[iSarabjitDhiman]

Seems like you might have two factor authentication enabled or twitter may have detected it as a suspicious login as it's not from your regular browser.

Solution : Use auth-token to generate a session.

How to get auth-token : Go to your browser, login into twitter if not already. Then go to developer tools, (google : how to open developer tools in chrome(your browser name). From there you should see the application menu, In the application menu, copy auth-token from the cookies section from the left panel.

Use auth-token to generate a session :

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.generate_session(auth_token="auth token you copied")

# Now you are good to go, add the other code below as you would normally.
user_tweets = twitter.get_user_tweets("username",total=5) 

# you can save the session if you don't want to do it every time you run the script 
# To save the session
twitter.save_session()

# If you saved a session, next time you will start as follows

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.load_session()

If there is still some issue, let me know.

is there any other way to login or get the token without doing it manually? @huulanka thank u for making me noticed it

It depends, if you are having login issues just because you have two fac auth enabled, I can add some feature to the code to take care of that. But if it's just a suspicious login, you can try changing the user agent to that of your twitter app or browser you usually use.

Let me try to replicate it, if I could get the same error, I will be able to fix it by making changes to the code. In the meantime, please let me know if you have two fac enabled.

[ihabpalamino]

Seems like you might have two factor authentication enabled or twitter may have detected it as a suspicious login as it's not from your regular browser. Solution : Use auth-token to generate a session. How to get auth-token : Go to your browser, login into twitter if not already. Then go to developer tools, (google : how to open developer tools in chrome(your browser name). From there you should see the application menu, In the application menu, copy auth-token from the cookies section from the left panel. Use auth-token to generate a session :

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.generate_session(auth_token="auth token you copied")

# Now you are good to go, add the other code below as you would normally.
user_tweets = twitter.get_user_tweets("username",total=5) 

# you can save the session if you don't want to do it every time you run the script 
# To save the session
twitter.save_session()

# If you saved a session, next time you will start as follows

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.load_session()

If there is still some issue, let me know.

is there any other way to login or get the token without doing it manually? @huulanka thank u for making me noticed it

It depends, if you are having login issues just because you have two fac auth enabled, I can add some feature to the code to take care of that. But if it's just a suspicious login, you can try changing the user agent to that of your twitter app or browser you usually use.

Let me try to replicate it, if I could get the same error, I will be able to fix it by making changes to the code. In the meantime, please let me know if you have two fac enabled.

okey thank you brother and if there is any possibility to scrape using since and until by date please tell me

[iSarabjitDhiman]

while login i got this error Traceback (most recent call last): File "C:\Users\HP Probook\PycharmProjects\firstproject\TikTokScrap.py", line 55, in twitter.login("","") File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\tweeterpy\tweeterpy.py", line 189, in login TaskHandler().login(username, password) File "C:\Users\HP Probook\PycharmProjects\firstproject\venv\lib\site-packages\tweeterpy\login_util.py", line 91, in login task_id = [task_id for task_id in task_flow_mapper.keys() if task_id in tasks][0] IndexError: list index out of range

@ihabpalamino Now you should be able to login without any issues. Duplicate & Fixed : Check #5

[iSarabjitDhiman]

Seems like you might have two factor authentication enabled or twitter may have detected it as a suspicious login as it's not from your regular browser. Solution : Use auth-token to generate a session. How to get auth-token : Go to your browser, login into twitter if not already. Then go to developer tools, (google : how to open developer tools in chrome(your browser name). From there you should see the application menu, In the application menu, copy auth-token from the cookies section from the left panel. Use auth-token to generate a session :

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.generate_session(auth_token="auth token you copied")

# Now you are good to go, add the other code below as you would normally.
user_tweets = twitter.get_user_tweets("username",total=5) 

# you can save the session if you don't want to do it every time you run the script 
# To save the session
twitter.save_session()

# If you saved a session, next time you will start as follows

from tweeterpy import TweeterPy
twitter = TweeterPy()
twitter.load_session()

If there is still some issue, let me know.

is there any other way to login or get the token without doing it manually? @huulanka thank u for making me noticed it

It depends, if you are having login issues just because you have two fac auth enabled, I can add some feature to the code to take care of that. But if it's just a suspicious login, you can try changing the user agent to that of your twitter app or browser you usually use. Let me try to replicate it, if I could get the same error, I will be able to fix it by making changes to the code. In the meantime, please let me know if you have two fac enabled.

okey thank you brother and if there is any possibility to scrape using since and until by date please tell me

Yes, I am working on it. I will update here once its done, maybe in a couple of hours.

[iSarabjitDhiman]

Hey @ihabpalamino I have added the from_date and to_date functionality as you suggested. But I have added it to the async-await branch as It needed some features from that branch. Besides it needs to be tested before its pushed to the master branch. I might merge it into master branch soon. For the time being, you will have to git clone the async-await branch and you are good to use the feature. You can check the usage here.

[sol080]

Hello @iSarabjitDhiman!

I found this thread and wanted to try out the from_date to_date functionality, but I could not clone the async-await branch (possibly because it's now a stale branch?). Can you help me to proceed?