Capture image take too much time

[JerryYCChang]

Hi,

I use this WimgApi build a UI to capture image, I had try different compression type and all of it will take a long time. If with none compression type, it will take about 60-70 minutes, with Lzx compression type will need about 80 minutes. If I use Dism command to capture image with maximum compression, it need just about 25 minutes and with fast compression, it need about 20 minutes. The source file and the hardware environment are the same. Is it normal that use WimgApi will need more time than use Dism command to capture image? Thank you~

[Herohtar]

I thought I was experiencing the same issue, but after doing some direct comparison tests there doesn't seem to be much difference in capture time for me, though Dism did appear to be slightly faster.

I made a backup using the WimgApi with Lzx compression and it seemed like it didn't report any progress for a long time -- it took a few minutes before it even moved to 1%, and then stayed at 1% for a long time after that, but in the end it completed the capture in ~78 minutes.

I then tried the same backup just using Dism from the command line with compression set to max. The Dism capture completed in ~67 minutes.

[josemesona]

@JerryYCChang do you have some sample code of how you're calling the managed API? In the end, it simply calls the native API so it shouldn't be any slower or faster. But you could be registering a callback that is slowing things down or something else could be awry.

@Herohtar do you want to post a repro of code that reports progress as you experienced? I'd like to dig into what's wrong.

[JerryYCChang]

@josemesona Here is part of my code, the compression type will be change in different type. string TempPath = Application.StartupPath;

        try
        {
            //MessageBox.Show("Start CreateFile !");
            Microsoft.Wim.WimHandle wimHandle = Microsoft.Wim.WimgApi.CreateFile(imageFilePath, Microsoft.Wim.WimFileAccess.Write, Microsoft.Wim.WimCreationDisposition.CreateAlways, Microsoft.Wim.WimCreateFileOptions.None, Microsoft.Wim.WimCompressionType.None);

            if(wimHandle != null)
            {
                Microsoft.Wim.WimgApi.SetTemporaryPath(wimHandle, TempPath);

                try
                {
                    //MessageBox.Show("Start Capture Image !");

                    using (Microsoft.Wim.WimHandle imageHandle = Microsoft.Wim.WimgApi.CaptureImage(wimHandle, volumeDir, Microsoft.Wim.WimCaptureImageOptions.None))
                    {
                        //MessageBox.Show("Start GetImageCount !");

                        int imageCount = Microsoft.Wim.WimgApi.GetImageCount(wimHandle);

                        //MessageBox.Show("Capture Image Finished !");
                    }

                }
                catch (Exception ex)
                {
                    MessageBox.Show("CaptureImage error : " + ex.Message);
                }

            }
            else
            {
                MessageBox.Show("wimHandle = null ");
            }

        }
        catch (Exception ex)
        {
            MessageBox.Show("CreateFile error : " + ex.Message);
        }

[jeffkl]

Ok I’m going to look into it. My guess is that dism is filtering out some stuff by default. You’re capturing a whole drive right? Have you compared the file size of the two WIM files?

[JerryYCChang]

@jeffkl I capture the OS volume on the drive. The original volume size is 12.5G, use WimgApi with compress type of none will take about 15 minutes and the WIM file size is 10.4G, with compress type of Lzx will take about 50 minutes and the WIM file size is 4.72G. Use dism command with default compress type will take about 7~8 minutes and the WIM file size is 5.05G. It may cause by the version of SDK, I use windows 10 version 1803 with the latest version of SDK.

[jeffkl]

I did some experiments and this is what I found:

Compressing a folder with 70,165 files, 16,718 folders, and 3.74 GB total

	Dism.exe		Microsoft.Wim
Compression	Time	Size	Time	Size
Default	1:00	1.93 GB
None	0:42	3.30 GB	0:40	3.30 GB
Max	1:25	1.88 GB	1:34	1.88 GB
Fast	1:00	1.93 GB	1:00	1.93 GB

Dism.exe has a /compress command-line argument that appears to default to fast which translates to WimCompressionType.Xpress. The best compression /compress:max maps to WimCompressionType.Lzx which is slower for sure.

Can you try WimCompressionType.Xpress in your code and see if it performs as fast as Dism.exe?

[JerryYCChang]

@jeffkl I did some experiments, too. The original OS volume have 12.5GB and the .wim file like the follow table. Sorry, I don't know how to set the table correctly, it's the same as your table.

	Dism.exe	Microsoft.Wim

Compression	Time	Size	Time	Size
Default	7:35	5.05 GB
None	14:17	9.8 GB	14:30	10.4 GB
Max	12:15	4.7 GB	49	4.72 GB
Fast	7:35	5.05 GB	28:45	5.15 GB

[ied206]

There are possibility that antivirus interfered with ManagedWimgApi, dropping the performance. While I was testing ManagedWimgApi and ManagedWimLib (My C# wrapper of wimlib), I experienced similar issue. It was solved after I turn off my antivirus (V3 Lite). Many antivirus solutions are notorious for suspecting and interfering non-signed binary, which I guess this is the reason.

@JerryYCChang What is your test environment? (OS, antivirus, etc.)

[JerryYCChang]

@jeffkl I forgot to tell you that I was use the capture back under WinPE, is there some thing may have different to cause the capture time to be different? @ied206 The OS is WinPE, build from Win10 version 1803. There are no antivirus.

[jeffkl]

There could be a major performance difference with .NET Framework on WinPE vs Windows. I know that assemblies are NGen'd which improves performance. But I am not aware of any known issues with .NET Framework apps running slow in WinPE. Even if there was, I wouldn't expect it to be 4 times slower. Especially in this case since Microsoft.Wim.dll is mostly just calling the native API.

I'm stumped and unfortunately have very little time at the moment to investigate. I'm definitely interested in what's causing it, so please keep trying to get to the bottom of it. In a few weeks I should have time to set up a WinPE test environment to see if I can figure it out.

[Herohtar]

In my case, running a full OS drive capture with Microsoft.Wim was taking significantly longer than the same capture using DISM, and also resulting in a much larger file. After a lot of tests I finally realized that the problem was that I was assuming that Microsoft.Wim had the default exclusion list that DISM does. It turns out that the API does not exclude any files by default, while DISM excludes this following list of files/folders by default:

• C:\$ntfs.log
• C:\hiberfil.sys
• C:\pagefile.sys
• C:\swapfile.sys
• C:\System Volume Information
• C:\RECYCLER
• C:\Windows\CSC

Specifically, the page file and hibernation file were both rather large and caused a major increase in capture time and image size. After modifying my project to exclude those files, there was no noticeable difference in Microsoft.Wim vs DISM captures.

As an interesting side note, the API does automatically skip compression on certain files (at least .zip, .cab, .exe, .dll, and .mp3, not sure if there are more).

[javmarquez13]

I have a issue when try to capture the image,

When try to capture with the wimcompress.none option the code working fine, and capture the image but when y try to capture with other option for example with wimcompress.Xpress create the .wim but never finish the capture...

this is mi code that i'm using

void WIMGAPI_CaptureImage()
    {
        try
        {
            // Open a handle to the .wim file
            WimHandle _handle = WimgApi.CreateFile(@"W:\Test_LZX_Compress.wim",
                                                    WimFileAccess.Write,
                                                    WimCreationDisposition.CreateNew,
                                                    WimCreateFileOptions.Verify,
                                                    WimCompressionType.Xpress);
            try
            {
                //Always create a temporal path to create a temporal files during the capture
                WimgApi.SetTemporaryPath(_handle, @"W:\");

                //Create delegate to get information during de capturing image
                //wimMessageCallback = new WimMessageCallback(WimCallBack); verify if is necessary create a instance

                //Register the callback method with the specific handle returned by wimgapi.CreateFile
                WimgApi.RegisterMessageCallback(_handle, WimCallBackMethod);

                //Call function to Capture the image
                _handleByCapturingImg = WimgApi.CaptureImage(_handle, @"E:\", WimCaptureImageOptions.None);
            }
            finally
            {
                // Be sure to unregister the callback method                    
                WimgApi.UnregisterMessageCallback(_handle, WimCallBackMethod);
            }

            //Be sure that close the handle for the capturing image next close the handle for the createfile
            _handleByCapturingImg.Close();
            _handle.Close();

            MessageBox.Show("Capture Image Finished");
        }
        catch (Exception ex)
        {
            MessageBox.Show(ex.Message);
        }
    }

 private WimMessageResult WimCallBackMethod(WimMessageType _MessageType, object _message, object _userData)
    {
        // This method is called for every single action during the process being executed.
        // In the case of apply, you'll get Progress, Info, Warnings, Errors, etc
        //
        // The trick is to determine the message type and cast the "message" param to the corresponding type
        //
        _WimMessageResult = WimMessageResult.Success;

        switch (_MessageType)
        {
            case WimMessageType.Progress:  // Some progress is being sent

                // Get the message as a WimMessageProgress object
                //
                WimMessageProgress progressMessage = (WimMessageProgress)_message;

                // Print the progress
                //
                lblProgress.Text =  "progress: " +  progressMessage.PercentComplete.ToString("#0.##%");
                lblEstimatedTime.Text = "Estimated Time Remaining: " + progressMessage.EstimatedTimeRemaining.TotalMinutes.ToString();                    
                lblEstimatedTime.Refresh();
                lblProgress.Refresh();
                break;

            case WimMessageType.Process:  // Some progress is being sent

                // Get the message as a WimMessageProgress object
                //
                WimMessageProcess processMessage = (WimMessageProcess)_message;

                // Print the Current file 
                //
                lblFilesInfo.Text = "File: " + processMessage.Path;                   
                lblFilesInfo.Refresh();
                break;

            case WimMessageType.Scanning:  // Some progress is being sent

                // Get the message as a WimMessageProgress object
                //
                WimMessageScanning scanningMessage = (WimMessageScanning)_message;

                // Print the Current file 
                //
                lblCountFiles.Text = "Count Files: " + scanningMessage.Count.ToString();
                lblCountFiles.Refresh();
                break;

            case WimMessageType.Compress:  // Some progress is being sent

                // Get the message as a WimMessageProgress object
                //
                WimMessageCompress compressMessage = (WimMessageCompress)_message;

                // Print the Current file 
                //
                lblFilesInfo.Text = "Compress File: " + compressMessage.Path;
                lblFilesInfo.Refresh();
                break;

            case WimMessageType.Warning:  // A warning is being sent

                // Get the message as a WimMessageProgress object
                //
                WimMessageWarning warningMessage = (WimMessageWarning)_message;

                // Print the file and error code
                //
                _WimMessageResult = WimMessageResult.Abort;
                break;

            case WimMessageType.Error:  // An error is being sent

                // Get the message as a WimMessageError object
                //
                WimMessageError errorMessage = (WimMessageError)_message;

                // Print the file and error code
                //
                Console.WriteLine("Error: {0} ({1})", errorMessage.Path, errorMessage.Win32ErrorCode);
                _WimMessageResult = WimMessageResult.Abort;
                break;
        }

        // Depending on what this method returns, the WIMGAPI will continue or cancel.
        //
        // Return WimMessageResult.Abort to cancel.  In this case we return Success so WIMGAPI keeps going

        return _WimMessageResult;      
    }

[jonmiller1]

@javmarquez13 I see you manipulating form controls inside of your callback event. That is going to cause some serious performance issues and may be causing your other issues too. Updating UI controls is slow, especially when that callback is getting called hundreds of thousands of times. Consider running the Microsoft.Wim calls on a different thread/background worker, pass updated info through the ProgressChanged callback in the background worker, and finally throttle the raising of that event, don't raise it more than once a second. You should see a noticeable performance gain.