Small File Packing (Satellite)

iglesiasbrandon commented 2 years ago

Summary:

Currently storing small files (5 MB or less) is not healthy for the satellites. for each file stored on the network, the satellite must store some metadata. If the size of the object is small the amount of metadata does not change so the ratio of data stored to metadata stored is unhealthy.

Pain Point:

For Each segment that is stored on the network the satellite must store some metadata. The to store the metadata on the satellite for a signal small file and a large (single segment) file is the same. So in terms of
Meta data in CRDB is very expensive.

Small objects take as much metadata as large objects so we want to optimize this.

We can not scale the satellite DB without having a satellite solution to packing files so that the cost of the metadata relative to objects stays low.
Small files end up becoming small Pieces that are stored on the Storage Nodes. Small Pieces are bad for the Nodes on the network because they are not optimally stored on the hard drive. data on hard drives is stored in blocks; small pieces occupy an entire block even if they are smaller than a block. Nodes are only paid for the size of the piece they store not for the entire size of the block.

Intended Outcome:

Backwards compatible for users
The satellite will have a process to pack small files uploaded by customers.

How will it work?

Blueprint: https://review.dev.storj.io/c/storj/storj/+/6543

wthorp commented 2 years ago

The link https://review.dev.storj.io/c/storj/storj/+/6680 is for Gateway-ST which isn't the satellite. I think the How will it work link should be https://review.dev.storj.io/c/storj/storj/+/6543?

derek-storj commented 3 months ago

This would be a requirement in order to pursue Veeam for M365 Backups. The backups of M365 environments includes individual emails (not packaged mailboxes), so think millions of 1KB files in this use-case.

storj / roadmap