Support arbitrary container compressions

[nirbar]

User story

As a setup developer, I would like to use a propriety compression method for containers. More generally, I would like to have the option to use any container compression algorithm using some sort of extension.

Proposal

Custom container compressions will allow 7z, zip, etc. Additionally, encryption may be utilized to protect the binaries.

I suggest to extend bundle extensions to support decompression on runtime, and WiX extensions to do the compression on build time.

Considerations

No response

[barnson]

Encryption is an interesting addition. But extensibility in this space is a high bar -- and we shouldn't maintain the current container API in the engine because it's heavily geared toward the CAB API. For example, I'm looking at LZMA in WiX v6 and it would have a very different API without streams. I'm not sure there's a generic approach but if you have an API proposal, please post it.

[nirbar]

I was thinking to add methods to bundle extensions to extract containers. I played with LZMA and it can be extracted with similar API as cab by customizing some lzma-sdk cpp classes, so I don't think that the API should change significantly

[barnson]

LZMA only does compression so it doesn't do files/streams. That's a big change in the API.

[nirbar]

lzma-sdk project creates/extracts 7z files https://www.7-zip.org/sdk.html

[barnson]

I believe 7-Zip is overkill for Burn. If we can avoid the overhead, we should. Most other compression libraries, like XZ and ZSTD, are also just container formats, not archive formats, so if we're going to muck with the Burn container API, it should support other and newer formats.

[nirbar]

Maybe we can allow both, something like this set of functions for IExtensionArchive:

ContainerOpen(..)
ContainerNextStream(..)
ContainerStreamToFile(..)
ContainerStreamToBuffer(..)
ContainerSkipStream(..)
ContainerClose(..)

Or this for IExtensionDecompressor:

ContainerDecompress(..)

But that would force us to handle the archiving metadata such as file names, sizes, modify time, attributes etc. Isn't that redundant?

[barnson]

The manifest already handles most of that data so a container API that used the manifest for the file metadata could use any arbitrary blob (compressed and/or encrypted) for the file content.

[nirbar]

Right. How about changing the API to get a dictionary of files to extract and their target paths?

ContainerOpen()
ContainerExtractFiles()
ContainerClose()

With a few changes cab can be used as the default implementation, while extensions can implement any compression- whether in a standard archive, or in a blob with the file metadata in bundle extension node?

[barnson]

Yep. The container API manages the file metadata. That would include a flag, to avoid extracting files that aren't needed, after planning.