Data collection - soldiers - Rachit & Sumit

sumitmukhija / Sentiment-analysis-study

0 stars 0 forks source link

Data collection - soldiers - Rachit & Sumit #3

Closed sumitmukhija closed 4 years ago

sumitmukhija commented 4 years ago

[x] Get usernames (/Veterans & Civilians.ipynb)
[x] Get all their bios (/supporting scripts/username-to-info)
[x] Filter bios (/supporting scripts/filter.py)
[x] Get Tweets of filtered users (/supporting scripts/username-to-tweets.sh)
[x] create data frame for soldier tweets

sumitmukhija commented 4 years ago

Had to change the csv file to trim all the special characters. Bash script used to fetch bio json below.

cat iava.csv | tr -d '\015' > new_iava.csv

Bash script to get user information for all users - /supporting scripts/script.sh

LOGFILE=test.log
exec < new_sample_file.csv || exit 1
read header 
a="twint --user-full -o file.json --json -u "
while IFS='\n' read name; do
    a="twint --user-full -o file.json --json -u $name"
    eval $a
    test -n "$ip" && echo ip = "$ip"
done

sumitmukhija commented 4 years ago

@irachitrastogi Did we get the tweets?

irachitrastogi commented 4 years ago

@sumitmukhija - I got the tweets and have pushed the same.