flokli / nix-casync

A more efficient way to store and substitute Nix store paths
80 stars 4 forks source link

pkg/store: use blobstore and metadatastore #21

Closed flokli closed 2 years ago

flokli commented 2 years ago

This further disentangles Metadata and Blob storage.

We also stop storing (redundant) .narinfo files on disk, but breaks the data into two types, NarMeta and PathInfo.

This brings a bunch of basic consistency sanity checks.

For example, a .narinfo file can now only uploaded if it refers to a .nar file that already exists with that narhash. NarSize also needs to be correct.

In the future, we could do some reference detection on uploaded .nar files, and validate this matches with what (each) Narfile specifies.

The new interface allows easy addition of new methods, which can be used to ask GC-related questions.

The metadatastore.FileStore simply serializes all structs as json to disk, but we can now also store this in a database without a lot of effort.

Related: #9

flokli commented 2 years ago

I want to add some more tests before merging this, mostly integrity-related:

I guess adding a NAR reference scanner, and validating it with what Nix sends in the uploaded .narinfo files can be moved to a separate PR.

flokli commented 2 years ago

I added some more tests to ensure the "foreign key constraints" are met, too.

flokli commented 2 years ago

This also fixes #23.