FungiStore: version 1 - Githubissues

threefoldtech / home

Starting point for the threefoldtech organization

https://threefold.io

Apache License 2.0

9 stars 4 forks source link

FungiStore: version 1 #1467

Closed despiegk closed 7 months ago

despiegk commented 1 year ago

features

export to fs
import from fs (blobs to ZDB, meta to FungiList) with support for relative location e.g. /data/fs/1/etc in FL is /etc
mount FL optional copy on write to other dir (for the changes)
metadata where blobs are stored is in the FungiList (multiple blob sources can be attached to 1 FL)
FungiList is based on sqlite and nicely normalized tables (easy to read/write from any language)
caching of blobs in local filesystem and can be used for multiple FungiMounts
populate 1 ZDB from other ZDB based on flist (pre-warm a ZDB), do in X co-routines for more speed
compress/encrypt blobs (encryption based on the hash)

Concepts

FungiMount = mounted FL to a dir
FungiList = FL = the list of metadata
ZDB = backend where blobs are stored
ZDB Cluster = group of ZDB's in 1 cluster so we have redundancy for ZDB storage

muhamadazmy commented 1 year ago

Changes to (rfs) (the 0-fs rust implementation) are here (PR is still open). please read the readme and the specs and let me know if u have any comment

branch v2-meta-rewrite

https://github.com/threefoldtech/rfs/tree/v2-meta-rewrite

muhamadazmy commented 11 months ago

https://gist.github.com/muhamadazmy/adf46289173ab65ca509c65f21b02f1e

despiegk commented 11 months ago

kristof will test this week

muhamadazmy commented 10 months ago

update:

From my side, fungistore rfs implementation is complete. The command line tool rfs supports creation, and mounting of .fl format, and upload blobs to different store types. Also ZOS now supports the new format as well (beside the hub) and has been tested to run workloads from user "custom" stores.

The only missing part of this story now is the section described under Fungi Cluster. Should we start workign on that?