Closed dolsysmith closed 2 years ago
Also need to create a shared directory on prod for storing the full extracts (distinct from the shared TS/SFM directory used for loading them).
Copy JSON files to full extracts directory instead of creating them anew.
Target number of tweets per extract type to achieve 2GB per file:
json: 511792 tweets csv: 10865173 tweets mentions/edges 137920613 tweets mentions/nodes: 259158572 tweets agg-mentions: 348209632 tweets ids: 222774986 tweets users: 49982150 tweets
The numbers were derived from extracts created on the Congress 115 dataset. To arrive at these estimates, I counted the number of lines in each file, divided the size of each file by the number of lines, and took the average score (= number of bytes per row) for each extract. Then I divided our target max file size (2GB) by this score to find the number of rows (tweets) that can fit into each extract type, assuming our target size.
Possible language to add to the full datasets page:
Windows users may need to use an application such as 7-Zip to unzip and open files with a .gz extension.
Spark always creates a directory when writing files to disk. As a result, extracts created during ingest will be in subdirectories as follows:
-- dataset_directory |tweet_ids |__tweet_json |tweet_csv
The current implementation stores all extracts in the same directory (under
full_datasets
). We could handle this in one of at least two ways:tweetset_server.py
to check for individual directories.