Exporter plugin (like frontend but for exporters)

[ktock]

Related #2581

There is a demand for exporting oci-incompatible build artifacts but doing the integration in-tree and the future maintenance looks very challenging.

To move such integration forward, we can support out-of-tree exporter plugins that allows extracting artifacts in arbitrary formats, it'll make the integration and maintenance feasible for advanced artifacts.

It've also been discussed in #2045 (comment)

@tonistiigi There's a concern of future maintenance cost of this code should it be merged. Would you be interested on taking a look at the problem of providing out-of-tree exporters(or blob converters per https://github.com/moby/buildkit/pull/2045#pullrequestreview-625890371) support instead that would run in containers like buildkit frontends do and what capabilities they would need.

As discussed above, maybe we can have a containerized exporter plugin like done in frontends.

cc @tonistiigi @sipsma @AkihiroSuda @imeoer

[AkihiroSuda]

How would it be feasible? With some sort of gRPC plugin? Or shared object plugin?

[ktock]

One possible design can be having an executable binary (or container) plugin that performs exporter.ExporterInstance.Export().

https://github.com/moby/buildkit/blob/55ba9d14360a6c6fb30c110016c09d823fad805b/exporter/exporter.go#L17

• BuildKit executes that plugin everywhere exporter.ExporterInstance.Export() needs to called. The plugin exports the artifacts to the destination (with creating the image in an arbitrary format) and returns the exporter response (map[string]string) via stdout when it exits.
• As Source (2nd arg in the above), BuildKit can provide gRPC APIs (via pipe?) and gives access to the source content blobs (and snapshots).
  • access to the content blobs might be the minimal requirement for creating the build artifact and pushing it to somewhere.
• access to the session from the plugin will be unsupported (or the future work).

[imeoer]

@ktock Does the exporter plugin need to do these by itself?

• Manage the blob cache;
• Mount snapshot & compute diff tar;
• Make config, manifest, index;
• Push blob & manifest;

[ktock]

• Make config, manifest, index;
• Push blob & manifest;

Yes, I think they should be done by expoter.

• Manage the blob cache;
• Mount snapshot & compute diff tar;

They depends on the expoter implementation. At least BuildKit should provide APIs to give the access to the exporting ref's blob and snapshot mounts.

[tonistiigi]

What are the examples of exporters that would use it?

I think there are at least 2 distinct options. One is for example an ISO exporter. That would be a container where the snapshots are mounted into and the container creates a write stream out of it. CLI side this would behave like -o type=tar exporter. The advantage of this compared to just creating iso as part of the build and exporting with -o type=local is that it doesn't remain in build cache(although that could be controlled in some other way as well).

Another would be something that deals with the blobs. I guess API for that exporter would be something like the contentstore API. Exporter can access the blobs directly and push them somewhere. CLI side would probably not create anything in this case.

For something like nydus I'm not sure what requirements it has. It may be possible that it is more like a differ-plugin rather than exporter plugin?

[sipsma]

That would be a container where the snapshots are mounted into and the container creates a write stream out of it

Would this mean that the snapshots have to be unlazied? Or are you imagining something else?

Another would be something that deals with the blobs. I guess API for that exporter would be something like the contentstore API. Exporter can access the blobs directly and push them somewhere.

When I've thought about exporter plugins in the past, I ended up sort of near here. It might need to be a combination of the content store API plus an API that lets you retrieve buildkit-specific metadata from CacheManager. Unless I'm not thinking of something, that might be the only way to be able to support lazy optimizations (e.g. skip unlazy if it already exists in export destination).

[sipsma]

Also, while there is risk of this derailing the idea here, I do think it's worth mentioning that a design here could have a lot of overlap with a similar design to support "cache export plugins". Given all the cache export backends (registry, GHA, S3, Azure, etc.), maybe the same idea could be beneficial.

Again, don't want to derail this design here, just maybe something to keep in mind in case there's opportunities to solve multiple problems at once.

[tonistiigi]

Would this mean that the snapshots have to be unlazied?

Yes, like any other mountpoint. If you are creating an iso stream, you need to read the files.