Storage Proposal

[ArthurMinovsky]

It should have

• Why do we need more storage

1. Model Checkpoint
2. Data Sourcing
3. Common Crawl Creating

[boat1603]

• We will use 80TB to keep dataset for multimodal dataset (5B files of LAION image dataset will use 50TB [reference]) and 30TB for other multimodal dataset (COYO-700M, Conceptual 12M and other datasets).
• We will use 40TB for our multimodal experiments (Keep weights, keep preprocess or cleaned data)