Ingest aggregate data into MongoDB

htrc / torchlite-backend

Backend API service for Torchlite web dashboard

2 stars 0 forks source link

Ingest aggregate data into MongoDB #78

Closed jswatsch closed 3 days ago

jswatsch commented 1 month ago

1.3 TB in size right now. Boris thinks there is enough room.

jswatsch commented 1 month ago

Ingest will probably happen late next week (4/22-4/26)

jswatsch commented 1 month ago

Some files exceeded the 16 MB file limit. There is a way around it by splitting into chunks that hasn't been setup yet. Boris is trying to find out how many files it is an issue for: ~1200 files exceeding 16MB.

@dkudeki is trying to set up a test MongoDB to see what is involved with chunking