spkenv / spk

A Package Manager for high velocity software environments, built on spfs.
https://spkenv.dev
Apache License 2.0
40 stars 6 forks source link

Make spfs diff only check sizes on blobs #999

Closed jrray closed 8 months ago

jrray commented 8 months ago

The main issue is the size of directories. If a path is committed on one machine with a different filesystem type or with/without fuse, the directory sizes captured into the manifest are platform-dependent, and then rendering that manifest on a different system may cause spfs diff to report changes.

One easy way to make this happen is to spk build a package while fuse is enabled and then spk env the built package with fuse disabled.

$ spk env stdfs -- spfs diff
~/spfs/bin [size {0=>6}]
~/spfs/etc [size {1=>18}]
~/spfs/etc/spfs [size {1=>23}]
~/spfs/etc/spfs/startup.d [size {2=>53}]
~/spfs/lib [size {0=>6}]
~/spfs/spk [size {1=>17}]
~/spfs/spk/pkg [size {1=>19}]
~/spfs/spk/pkg/stdfs [size {1=>19}]
~/spfs/spk/pkg/stdfs/1.1.0 [size {1=>22}]
~/spfs/spk/pkg/stdfs/1.1.0/PF6B6TLF [size {5=>77}]
jrray commented 8 months ago

Fun fact, no tests were harmed in the making of this PR.

This is affecting us when launching jobs on our render farm because spawn sees these changes from spfs diff and thinks it needs to create an spfs snapshot and capture the changes.

IMO this should be taken a step further and change the Entry type(s) to only contain sizes for blob-type entries. I have a branch with this change but the problem is that it affects the digest calculation so existing manifests that contain sizes for non-blobs would be readable but then encode back out into an object with a different digest from what it originally had.