search

project-stacker / stacker

Build OCI images natively from a declarative format
https://stackerbuild.io
Apache License 2.0
201 stars 34 forks source link

feat(erofs): initial commit for erofs support #626

Open rchincha opened 3 months ago

rchincha commented 3 months ago

Fixes https://github.com/opencontainers/image-spec/issues/1190

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

hsiangkao commented 3 months ago

Hi @rchincha, I'm currently working on incremental builds and I plan to release erofs-utils v1.8 this month (with multi-threaded mkfs, incremental builds, Intel QPL, etc.), but anyway, I'd suggest use cgo for initial EROFS support (with formal liberofs APIs exported) for now (or even binary integration as the very first step) since this go implemention seems somewhat incomplete (and maybe even broken.. I don't have enough time to look into that since other prioritied stuffs are on hands..)

rchincha commented 3 months ago

@hsiangkao I suspect that cgo path may be what ends up happening. Currently, just prototyping quickly to understand the interfaces and scope of changes/work.

hsiangkao commented 2 weeks ago

One thing I might need to mention here is that erofs supports external blobs or chunks since Linux 5.16 compared to Squashfs because Dragonfly Nydus once asked this feature to deduplicate data in chunks among different container images.

I'm not sure if it's worthwhile to highlight this, you could just make a tiny metadata with external blobs (which can be used for multiple images) for reference. Currently only Nydus has userspace tools to generate chunk blobs, if that is interested in other use cases, I could seek more time to implement this in mkfs.erofs too.