twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.69k stars 2.71k forks source link

[QUESTION] insert my own variable into csv #671

Closed amandaashleywoo closed 4 years ago

amandaashleywoo commented 4 years ago

72DFC5A3-671E-4673-8A19-964D11C9ADCA

Command Ran

I’ve written this code attached above.

Description of Issue

lstn is a list with the same number of elements as lstf, and I’m trying to write a code that will add a column that includes my own custom value (from lstn) in the .csv file. How can I do that? However, in my code I can’t seem to change “x” to take on another value in c.Format.

Environment Details

MacOS Catalina, running in terminal

pielco11 commented 4 years ago

You might want to edit twint/tweet.py adding your field, and twint/storage/write_meta.py to let it be saved into the output file

amandaashleywoo commented 4 years ago

I think everything’s saved into the output file as the loop is run, but I’m not sure how to make it read the variable {x}, and not read it as a string?

pielco11 commented 4 years ago

c.Format is just for stdout, not what it's saved to the file

To change the saved fields you have to do as cited above, to format the stdout you just need to add that x field to each tweet. To achieve this you need to save the tweets to a list (read here https://github.com/twintproject/twint/wiki/Storing-objects-in-RAM#tweets) and specify c.Hide_output = True. Then, once the scraping process if finished, you have to iterate over tweets_list and then print

Basically something like

import twint

c = twint.Config()
c.Username = "user"
c.Hide_output = True
c.Store_object = True

twint.run.Search(c)
tweets = twint.output.tweets_list

for t in tweets:
    print(f"{t.id} {x}"
amandaashleywoo commented 4 years ago

alright, thank you so much!