search

internetarchive / liveweb

Liveweb proxy of the Wayback Machine project
https://web.archive.org/
44 stars 13 forks source link

Write large files to disk, rather than 1 file-per-request #27

Closed rajbot closed 12 years ago

rajbot commented 12 years ago

Ops would like to have a few large files, rather than a lot of files on the filesystem. Since we arc files get very large before being uploaded, the number of files on disk can get problematic.

anandology commented 12 years ago

Writing to one single file might require too much locking to avoid race conditions. We are thinking of using a file-pool with 10 files or so.

anandology commented 12 years ago

This is implemented now.