search

holepunchto / hyperdrive-next

Hyperdrive is a secure, real-time distributed file system
https://docs.holepunch.to
Apache License 2.0
45 stars 9 forks source link

What are the storage limits for hyperdrive? #2

Open KrishnaPG opened 2 years ago

KrishnaPG commented 2 years ago

This is originally posted here, but since this is the next version preview going on, thought of adding here so that this can be discussed and can possibly contribute to the design / architecture.

When using a local file system, it looks like all the hyperdrive block data is stored into a single file (/data) in the storage. As such that will have problems with the OS file size limits.

What happens when the hyperdrive content becomes large? Will the hyperdrive (or hyperspace / hypercore) continue to append the blocks to the same underlying storage data file or will it spill them into different data files?

Where can one find more information on how the file system is organized underlying for the blobs of Hyperdrive?

mafintosh commented 2 years ago

It continues to append to that file yes, but the file impl is flexible so close to trivial to make it use multiple if you wanted to (see random-access-storage)

mafintosh commented 2 years ago

No file limits, flexible indexing with the btree.