twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.75k stars 2.72k forks source link

[QUESTION] How to save tweets into mysql with twint #599

Closed dangjinghao closed 4 years ago

dangjinghao commented 4 years ago

it is seems that twint support Mysql ,but now i just know how to save tweets as db file. i have search issues(and wiki) but i could not find anything about it.so can you help me?

i am a python,github starter and not a person that have native English. thankyou.

pielco11 commented 4 years ago

Mysql support has been removed since a while

dangjinghao commented 4 years ago

Mysql support has been removed since a while

very thankyou,but can you tell me what had happened? i may need write a file to save tweets into mysql by myself. 谢谢。

bushjavier commented 4 years ago

Attached are files to enable twint to support mysql, i am not expert in python but i changed few lines in the file db.py, to use my files please create a your database using the script attached named create_bd_twint.sql then replace the db.py located in twint/storage After that you need to install the mysql support using the command

pip3 install pymysql

finally you need to run twint using the parameter c.Database = "twitter" or if you are calling it from the command line add the database parameter use any name it does not matter, because the data base IP, username and password are located in the db.py please change it to your database, search this line

conn = pymysql.connect("10.10.10.10","root","password","twitter")

and modify it, there are 2 lines like that one.

in my mysql database i have 4 million of tweets scraped in few days using multiple servers.

mysql support.zip

pielco11 commented 4 years ago

@dangjinghao there were compatibility problems, so we just dropped it

bushjavier commented 4 years ago

@dangjinghao you could use my files to support mysql in the lasted version of twint, i use the cheapest server in google cloud around 4 usd per month i use 8 and with that i get around 1 million tweets per day, all the twint scripts write to the same database. I hope this help other people