In some cases, we are seeing manifests reach into the hundreds of megabytes in size. Additionally, there is duplicated data between manifests of spk packages when minor changes happen in one or two sub directories in a large package with thousands of files.
Proposal
SPFS already has an object kind that holds trees. These represent a single directory and are combined together in a manifest to create a full filesystem hierarchy.
Instead of always storing the reference to this complete manifest in layers, we would allow the reference of the root tree object. This means that trees could/would be stored individually as objects in the repo, and that the manifest would/could be loaded incrementally as all trees are accessed.
Other Considerations
What about a single file with deep structure? This case would create many trees and slow down the loading process because of back-and-forth with the repository. Maybe it's atypical and we don't care, or we have smarter ways to request the entire manifest via gRPC to reduce round-trips when networked.
Background and Problem Space
In some cases, we are seeing manifests reach into the hundreds of megabytes in size. Additionally, there is duplicated data between manifests of spk packages when minor changes happen in one or two sub directories in a large package with thousands of files.
Proposal
SPFS already has an object kind that holds trees. These represent a single directory and are combined together in a manifest to create a full filesystem hierarchy.
Instead of always storing the reference to this complete manifest in layers, we would allow the reference of the root tree object. This means that trees could/would be stored individually as objects in the repo, and that the manifest would/could be loaded incrementally as all trees are accessed.
Other Considerations