Faster conversion

[kroese]

The current code takes quite a bit of time to convert all parts of the downloaded ESD to .wim files (boot.wim and install.wim), especially the WinPE part is always very slow.

However, I have seen some ISO's in the wild that have boot.esd and install.esd files instead of .wim files.

So wouldnt it be possible to skip the whole ESD->WIM conversion, and just generate an ISO containing only .esd files? That should make the whole process a lot quicker.

[Technogeezer50]

I'll take a look at it as a possible enhancement, but I'm not sure exactly sure of how much speed up we're going to get.

Background:

The file formats are somewhat immaterial. Both WIM and ESD are similar types of files containing multiple images. From what I understand the major difference is in the style of compression used.

The ESD file doesn't contain a boot.wim/boot.esd or install.wim/install.esd images. They have to be built from the images found in the ESD and placed in the ISO image.

Since we're building on non-Windows platforms, we don't have access to DISM. We're dependent on the open-source wimlib-imagex.

The ESD isn't bootable as-is. Index 1 of the ESD has to be extracted to get the files needed to boot the ISO and use WinPE/Setup from boot.wim. boot.wim(or boot.esd) has to be created from two images in the downloaded ESD -- WinPE (index 2 in the ESD) and Windows Setup (Index 3). This phase is relatively quick.

The majority of the time in the conversion is spent moving/copying (not unpacking) images 4 and later in the ESD to install.wim so that Windows Setup can use them. That's about 5GB or so of compressed image. Plus I believe there's a compress/decompress in the process that may add time.

At one point in time I did think of a short cut: moving the ESD file into the ISO image and renaming it to install.esd. It was possible to "logically" delete the first 3 images in the ESD. It worked, but the down side of using the ESD in this manner was that I lost the original ESD for debugging purposes, and the resulting ISO was slightly bigger. I can revisit that again to see how much it cuts down the creation time.

[kroese]

I am using your code in my project (see https://github.com/dockur/windows ) and during development I literally ran the conversion hundreds of times.

So I can tell from experience that on my hardware (recent CPU/SSD) the majority of the time spend is always spend in the WinPE phase, not in the extraction of the large image later on.

I also thought that was weird (because its much smaller than the other parts), but my guess is that its related to that chunk-size used for compression. Because I once tried with a different chunksize/compression, and suddenly it went much quicker. But the resulting image did not boot for some reason, so I had to revert that change. Maybe the Windows installer has the chunksize hardcoded or something.

[Technogeezer50]

That's an interesting behavior. I've always seen the majority of the time spent in the build of install.wim (and of course the download itself).

Can you augment the code with a few time stamps at the points before and after calls to "wimlib-imagex apply" and "wimlib-imagex export" in the extractEsd() function and run the conversion with the -d flag?

I'll do the same and compare notes.

That will also give me an indication of the components of the source ESD.

[kroese]

Benchmark of Win10 x64:

Total of 1.890205964 seconds elapsed for: wimlib-imagex apply 1 Total of 48.578550912 seconds elapsed for: wimlib-imagex export 2 Total of 8.329110186 seconds elapsed for: wimlib-imagex export 3 Total of 17.615331200 seconds elapsed for: wimlib-imagex export 9

As you can see, export 2 takes 48 seconds which is almost three times longer than export 9, and also more than all other steps combined.

Benchmark of Win11 x64:

Total of 1.740064621 seconds elapsed for: wimlib-imagex apply 1 Total of 54.834511187 seconds elapsed for: wimlib-imagex export 2 Total of 8.432699849 seconds elapsed for: wimlib-imagex export 3 Total of 58.280453731 seconds elapsed for: wimlib-imagex export 9

[Technogeezer50]

I just ran some tests and It's the choice of compression algorithm making the difference. Try setting the compression to LZMS instead of LZX and a 128K block size. There's a big difference in time. I'll have to see if the resulting ISO is bootable.

[kroese]

When I change to LZMS 128k it decreases the time for WinPE to only 4 seconds (instead of 1 minute) but Windows will report error code 0xc0000225 when booting the image.

EDIT: It seems like --compress=none can do WinPE in 8 seconds and the filesize of the ISO is really not important to me. So I guess this issue is fixed. Unless you have alternative ideas to make it faster (your initial shortcut sounded very interesting).

[Technogeezer50]

I've tried a number of options for compressions and the only one that seems to not give me the issue on boot is LZX with a 32K chunk size.

I'm going to keep this open for the time being and do some more research to see if I can find faster alternatives. One of the things I tried to do was to keep this as close to what I would see in a Microsoft ISO. I'm going to download a Windows 11 x86_64 ISO from Microsoft and see if its boot.wim/esd can give me any clues.

[kroese]

I solved it by setting compress=none for the boot.wim. It doesnt seem to impact the overall size of the ISO in any significant way.

[Technogeezer50]

I double checked an official Windows 10 ISO I had downloaded. The boot.wim file in that ISO is built with LZX compression and a 32K block size.

Given that LZX compression is what Microsoft seems to have used, and given the inability to get the ISO to boot with LZMS and/or a larger block size, I'm closing this request. I thank you for bringing this up, but the difference in ISO creation time doesn't warrant changing the compression from what Microsoft uses in their own ISO.

[Technogeezer50]

Closed - decided not to deviate from the compression settings used by Microsoft in their downloadable ISO.