twitter is scraping tweet links instead of the actual tweets

[arceuss]

im trying to scrape someones tweets for ai but its just giving me links

[hkim2636]

Did you tried out --jsonl option? It will give you the content of tweets. snscrape --jsonl --max-results 100 twitter-hashtag archiveteam

[arceuss]

does it do tweets of a specific user

[arceuss]

also i tried but it just says its not a valid arguement

[JustAnotherArchivist]

--jsonl only exists in the current development version. You can also use --format '{content}' on the release version (or the dev one), but be aware that tweets can contain line breaks, so one line in the output will not necessarily be one tweet.

And yes, those options are independent of the scraper, so you can use them with twitter-user as well. (The available fields for --format do depend on the social network though.)

[mvitha]

got it to run -

installed the development version using: pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git

ran from command line: snscrape --jsonl --max-results 10 twitter-user textfiles > text_files_test_10.json

then in py script or jupyter notebook script

import json import pandas

table = [] with open('text_files_test_10.json', 'r') as f: for line in f: table.append(json.loads(line))

pd.DataFrame.from_records(table).to_csv('text_files_test_10.csv')

thanks this is awesome!

[arceuss]

everything looks like this tho i dont know how to train this with ai lol

[arceuss]

oh

[DiameterEffect]

got it to run -

installed the development version using: pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git

ran from command line: snscrape --jsonl --max-results 10 twitter-user textfiles > text_files_test_10.json

then in py script or jupyter notebook script

import json import pandas

table = [] with open('text_files_test_10.json', 'r') as f: for line in f: table.append(json.loads(line))

pd.DataFrame.from_records(table).to_csv('text_files_test_10.csv')

thanks this is awesome!

for those having an error like name pd is not defined use import pandas as pd