Documentation of the APPX/MSIX format

ben221199 commented 1 year ago

Last week I unzipped an APPX/MSIX file and zipped them again. The files themselves unchanged. When trying to install the first file, it gave me some signature error, but I think I can solve that one by adding the right files. However, when I clicked the second file (my self-made zip), it gave me some parsing error. When comparing the zip files with each other, I saw some differences.

Is there any documentation about the requirements of the zip file? What files should be stored, what files should be deflated, what files shouldn't have any flags and what files should have flag 0x08? What should be the order of the files? What files need a signature header? Etcetera, etcetera. I think it is really important if you want to make software that also is able to generate APPX/MSIX files.

For now, this what worked for me (in this order):

LocalFileHeader: AppxBlockMap.xml (Deflated, no flags)
LocalFileHeader: AppxManifest.xml (Deflated, flag 0x08)
Signature: crc, compressed_size, uncompressed_size
LocalFileHeader: [Content_Types].xml (Deflated, no flags)

CentralFileHeader: AppxBlockMap.xml (Deflated, no flags)
CentralFileHeader: AppxManifest.xml (Deflated, flag 0x08)
CentralFileHeader: [Content_Types].xml (Deflated, no flags)
EndOfCentralDirectory: ...

I did some tests and for now it seems I have to stick to this order, have the flags as given above, have a correct CRC in both LocalFileHeader and CentralFileHeader and have a version that is the same in both LocalFileHeader and CentralFileHeader, no matter the value.

However, if it is possible to documentate this rules somewhat more, it would be great. Anyway, I will add my findings in this thread for if this rules aren't known by somebody else here.

DrusTheAxe commented 1 year ago

.msix files are .zip files with some additional rules about their content / structure. You can't simply "unzip" a .msix and "zip" the files using ZIP savvy tools and get a valid ".msix" file. You need tools that are savvy to MSIX's file format (typically makeappx.exe and signtool.exe).

Why do you ask? What are you trying to accomplish?

ben221199 commented 1 year ago

I am aware that just unzipping and zipping doesn't work and that I need MakeAppx and SignTool.

However, I ask this because of two things:

I think this additional rules should be part of the documentation.
I want to write some tools that support APPX/MSIX without them needing MakeAppx and SignTool.

ben221199 commented 1 year ago

In EBNF, this are the rules of the order and the type of Zip-sections:

zipfile = (local_file, data_descriptor?, data_bytes)+, central_directory
central_directory = central_file+, end_central_directory

local_file = "PK\3\4", header_bytes
data_descriptor = "PK\7\8", header_bytes
central_file = "PK\1\2", header_bytes
end_central_directory = "PK\5\6", header_bytes

header_bytes = (* Dependent on the header type, the remaining bytes of the header. See APPNOTE for details. *)
data_bytes = (* Dependent on the compression type, the STORED or DEFLATED data of the file. *)

For every header type, this are the rules for the fields:

Local file

Field	Description
Signature	Always `PK\3\4`.
Extract version	Can be anything, but should be the same in the central file.
Flag	If bit 4 is set, a data descriptor follows directly after the data, else it doesn't.
Compression method	`0` (STORED) or `8` (DEFLATED).
Time	No rules.
Date	No rules.
CRC*	Exact CRC32 of the file.
Compressed size*	The compressed size.
Uncompressed size*	The uncompressed size.
Filename length	The length of the filename.
Extra field length	The length of the extra field.
Filename	The name of the file.
Extra field	The extra field. Can be anything compatible.

Some could possibly be set to zero because of the data descriptor, but I am not sure.

Data descriptor

Field	Description
Signature	Always `PK\7\8`.
CRC*	Exact CRC32 of the file.
Compressed size*	The compressed size.
Uncompressed size*	The uncompressed size.

Some could possibly be set to zero, but I am not sure.

Central file

Field	Description
Signature	Always `PK\1\2`.
Made version	Can be anything. No rules.
Extract version	Can be anything, but should be the same in the central file.
Flag	If bit 4 is set, a data descriptor follows directly after the data, else it doesn't. Same as local file.
Compression method	`0` (STORED) or `8` (DEFLATED). Same as local file.
Time	No rules.
Date	No rules.
CRC*	Exact CRC32 of the file.
Compressed size*	The compressed size.
Uncompressed size*	The uncompressed size.
Filename length	The length of the filename.
Extra field length	The length of the extra field.
File comment length	The length of the comment.
Disk number start	Set to `0` in my case.
Internal File Attributes	Set to `0` in my case, but can be something compatible, I assmume.
External File Attributes	Set to `0` in my case, but can be something compatible, I assmume.
Offset to local header	The offset of the local header relative to the start of the file. Should be correct.
Filename	The name of the file.
Extra field	The extra field. Can be anything compatible.
File comment	A comment on the file. Can be anything.

Some could possibly be set to zero (because of the data descriptor), but I am not sure.

End of central directory

Field	Description
Signature	Always `PK\5\6`.
Number of disk	Set to `0` in my case.
Number of disk with start of central directory	Set to `0` in my case.
Total number of entries on this disk	Should be exact amount of entries.
Total number of entries	Should be exact amount of entries.
Size of central directory	Should be correct size of central directory.
Offset of central directory	Should be correct offset of central directory.
Comment length	The length of the comment.
Comment	A comment on the whole file. Can be anything.

Some could possibly be set to zero, but I am not sure.

I hope this can be of any use for somebody here and I also hope that the verification rules can be documented somewhere in the official documentation.

Thanks in advance.

florelis commented 1 year ago

I want to write some tools that support APPX/MSIX without them needing MakeAppx and SignTool.

Would using the AppxPackaging API suit your needs? That should let you consume and produce .appx/.msix packages programmatically without using MakeAppx.

DrusTheAxe commented 1 year ago

Would using the AppxPackaging API suit your needs? That should let you consume and produce .appx/.msix packages programmatically without using MakeAppx.

+1

That's how much of makeappx.exe is implemented - Packaging API does the heavy lifting.

The docs you shared are the ZIP file format. MSIX has additional rules beyond that so yes, technically an MSIX file can often be used as a .zip file, but not entirely e.g. when signed the Signature.p7x file must be the last file in the archive (ZIP has no such requirement over ordering of its content), as one example. I'd thought the file format was fully doc'd but I'm not sure where. I'll ask around and if not then yes, it's something we should work to improve. Thanks for bringing it up.

BTW this project and Windows App SDK are open source so you may find their implementations interesting reading e.g. MakeMSIX.

ben221199 commented 1 year ago

This repository helped me a little bit yes. Some information was useful. For example, there is an extension list that tells what file types should be STORED and what should be DEFLATED. That is information I can use.

What I want to know additionally is if it is required that AppxManifest.xml, AppxBlockMap.xml and [Content_Types].xml are deflated. At the moment I have to use a hardcoded byte sequence of each deflated file content to pack into the zip. If I use the STORED method, it doesn't work and if use the DEFLATED method and calculate the deflation data myself it also doesn't work. Maybe I'm missing something undocumentated there too.

Would using the AppxPackaging API suit your needs? That should let you consume and produce .appx/.msix packages programmatically without using MakeAppx.

Using that API could be used in some languages compatible with it, yes, but for me, who also works with PHP and Java, it isn't easily usable. I wrote already some basic implementation in Java for APPX/MSIX already.

mikehearn commented 7 months ago

@ben221199 If you still need a JVM implementation of MSIX and your need is commercial, email contact@hydraulic.dev because we have done all this and can make signed MSIX files successfully from Java (well, Kotlin). Maybe we can help you.

ben221199 commented 7 months ago

@mikehearn It is not about an implementation. It is about documentation. If I know the rules, I can build anything.

Andrew-web-coder commented 4 months ago

MS just doesn't care because you don't bring money. I want to replace the images in the msix package because this tool produces crappy logo images and it is not possible to provide a custom logo. And there is no other method than to do it manually.

microsoft / msix-packaging