Component ID generation across upgrades.

monty241 commented 5 years ago

I'm having problems understanding the implications of the component ID generation process (v1.9.3). Originally I sometimes had the problem that after an upgrade essential assembly files were missing.

Today I've switched the compiler to deterministically generate the same ID across upgrades:

In the WXS files I can see that the ID of each Component is the same across new versions.
When I do a fresh install, it installs fine in the folder. When I do an upgrade, absolutely everything is removed.

However, I did not expect that everything would be removed on an upgrade.

Questions:

Should the WXS IDs of components be different across upgrades?
Or should the WXS IDs be persistent per file (and therefor deterministically based upon file name/path)?

If so, suggestion is to more clearly document this in the properties. In know that it is all coming from WIX, but some extra help would be welcome in the documentation.

Some details:

MSI ProductCode of the two MSIs are different (ProductCode in MSI, product ID in WXS).
UpgradeCode are equal, both in WXS as in MSI.

Samples given:

Version 17.29.39

WXS contains:

<Component Id="Component.File_xxx.dll_484619357" Guid="4a77cce5-8beb-4b41-9881-aa603c886e2d">
  <File Id="File_xxx.dll_484619357" Source="..\..\solutions\Awesomething\obin\xxx.dll">

MSI analyzer displays as ComponentId: {4A77CCE5-8BEB-4B41-9881-AA604D5AEE2D}

Version 17.29.40

WXS contains:

<Component Id="Component.File_xxx.dll_484619357" Guid="4a77cce5-8beb-4b41-9881-aa603c886e2d">
  <File Id="File_xxx.dll_484619357" Source="..\..\solutions\Awesomething\obin\xxx.dll">

MSI analyzer displays as ComponentId: {4A77CCE5-8BEB-4B41-9881-AA604D5AFE2D}

The Component Id in the WXS are identical, but the compiled ComponentId is different.

oleg-shilo commented 5 years ago

Yes Components are the topic that was not covered enough. I have just updated the documentation to fill the gap: https://github.com/oleg-shilo/wixsharp/wiki/Deployment-scenarios#component-id-considerations

You may also want to read this rather long exchange that was triggered by the similar problem: https://github.com/oleg-shilo/wixsharp/issues/452

Now your questions....

However, I did not expect that everything would be removed on an upgrade.

Neither did I. To be honest I cannot comment on this one. This is one of those pure-MSI issues that requires a deep dive into MSI/WiX documentation.

Should the WXS IDs of components be different across upgrades?

It should not matter. The Major Upgrade is an automatic uninstall followed by the newer install. Thus compId should not make any difference. Well, I expect it to be the case.

Or should the WXS IDs be persistent per file (and therefor deterministically based upon file name/path)?

They are currently based on name/path. See Project.HashedTargetPathIdAlgorithm(WixEntity entity) method. The value 90490314 is the hash of the destinationdirectory.

<File Id="Manual.txt_90490314" Source="Files\Docs\Manual.txt" />

The Component Id in the WXS are identical, but the compiled ComponentId is different.

This is how the component guid is generated:

XElement comp = dirItem.AddElement(
    new XElement("Component",
        new XAttribute("Id", compId),
        new XAttribute("Guid", WixGuid.NewGuid(compId))));

And in my test it produced consistent GUID values:

If indeed it behaves this way in your case then please share a simple hello world project that shows the problem. I would prefer to fix it.

monty241 commented 5 years ago

Thanks for sharing your current knowledge. Hmm, I think it is time for a deep dive since I see some differences in behavior. I will try to work on it wednesday and update issue with my findings.

monty241 commented 5 years ago

Have a happy new year

monty241 commented 5 years ago

Little addition:

It should not matter. The Major Upgrade is an automatic uninstall followed by the newer install. Thus compId should not make any difference. Well, I expect it to be the case.

From logging, the actual order during execution of the MSI found is:

First install the assemblies and other files.
Then run an uninstall with the MSI from the previous version using REMOVE=ALL.

monty241 commented 5 years ago

It seems that two wxs files are generated when the solution is built. Both projects in the sample solution only have target framework net47.

The one found in the archive location has consistent GUIDs, the one in the obin folder changes.

Running candle with trace, pedantic and verbose displays the line number collection showing that the wxs file in the obin folder is used.

It appears that the archive WXS file is created by a call to BuildWxs and the second one by BuildMsi. They don't share the file path (suggested enhancement #563).

1>------ Build started: Project: Invantive.Data.Generator, Configuration: Debug Any CPU ------
1>Invantive.Data.Generator -> C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\bin\net47\Invantive.Data.Generator.exe
2>------ Build started: Project: Invantive.Data.Generator.Setup, Configuration: Debug Any CPU ------
2>Properties\AssemblyInfo.cs(30,32,30,55): warning CS7035: The specified version string does not conform to the recommended format - major.minor.build.revision
2>Invantive.Data.Generator.Setup -> C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\ibin\net47\Invantive.Data.Generator.Setup.exe
2>Building MSI
2>Searching for custom action entry points in Invantive.Deploy.CA.dll
...
2>Wix project file has been built: c:\temp\invantive-deployment\Invantive Data Generator\17.29.31\Invantive Data Generator-17.29.31-INTERNAL.wxs
...
2>Searching for custom action entry points in Invantive.Deploy.CA.dll
...
2>Wix project file has been built: C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs
...
2>candle.exe : warning CNDL1108: The command line switch 'trace' is deprecated.
2>Windows Installer XML Toolset Compiler version 3.11.0.1701
2>Copyright (c) .NET Foundation and contributors. All rights reserved.
2>
2>Invantive Data Generator-17.29.31-INTERNAL.wxs
2><?xml version="1.0" encoding="IBM437"?>
2><?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*2?>
2><Wix xmlns="http://schemas.microsoft.com/wix/2006/wi" xmlns:netfx="http://schemas.microsoft.com/wix/NetFxExtension">
2>  <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*3?>
2>  <Product Id="c1c5c08d-d98f-4b84-8e56-c55c2ed21100" Name="Invantive Data Generator 17.29" Language="1033" Codepage="Windows-1252" Version="17.29.31" UpgradeCode="c1c5c08d-d98f-4b84-8e56-c55c1d001100" Manufacturer="Invantive Software B.V.">
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*4?>
2>    <Package InstallerVersion="200" Compressed="yes" Description="Invantive Data Generator 17.29" SummaryCodepage="Windows-1252" Languages="1033" InstallScope="perMachine" />
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*5?>
2>    <Media Id="1" Cabinet="Invantive_Data_Generator_17.29_.cab" EmbedCab="yes" />
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*7?>
2>    <Condition Message="Invantive Data Generator requires Microsoft .NET Framework 4.7 or greater to be installed. Please install a supported Microsoft .NET Framework version, such as from https://www.microsoft.com/en-us/download/details.aspx?id=55167."><![CDATA[Installed OR (NETFRAMEWORK45 AND NETFRAMEWORK45 >= "#460798")]]></Condition>
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*8?>
2>    <Condition Message="Invantive Data Generator requires at least 1,000 MB of physical memory. Please configure more physical memory for use by applications."><![CDATA[PhysicalMemory >= 1000]]></Condition>
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*9?>
2>    <Condition Message="Invantive Data Generator is only supported on Windows 7, Windows Server 2012R2, or higher current or terminal release. The operating system must also be 64-bit. Please upgrade to a supported Windows version."><![CDATA[Installed OR VersionNT >= 601]]></Condition>
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*10?>
2>    <Condition Message="Invantive Data Generator is only supported on 64-bit versions of Windows 7, Windows Server 2008R2, or higher current or terminal release. Please upgrade to a supported Windows version with 64-bit.">Installed OR VersionNT64</Condition>
2>    <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*12?>
2>    <Directory Id="TARGETDIR" Name="SourceDir">
2>      <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*13?>
2>      <Directory Id="ProgramFilesFolder" Name="ProgramFilesFolder">
2>        <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*14?>
2>        <Directory Id="ProgramFilesFolder.Invantive_Software_BV" Name="Invantive Software BV">
2>          <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*15?>
2>          <Directory Id="INSTALLDIR" Name="Invantive Data Generator 17.29">
2>            <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*17?>
2>            <Component Id="Component.File_IKVM.OpenJDK.Charsets.dll_981093327" Guid="c1c5c08d-d98f-4b84-8e56-c55ceb8a597c">
2>              <?ln C:\ws\Invantive.Data.Generator\solutions\Invantive Data Generator\obin\Invantive Data Generator-17.29.31-INTERNAL.wxs*18?>
...

monty241 commented 5 years ago

The two WXS files are generated from the same Project sequentially.

However, the contents are different:

Different Namespaces

Namespaces differ. It seems that the XML is constructed and read-back, rewriting the tag without loosing validity.

First WXS being generated: <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">

Second WXS being generated: <Wix xmlns="http://schemas.microsoft.com/wix/2006/wi" xmlns:netfx="http://schemas.microsoft.com/wix/NetFxExtension">

Rewritten components

Besides changing namespaces, including references as here in NativeImage, the Guid of components are changed too by Wix#.

First WXS:

            <Component Id="Component.File_IKVM.OpenJDK.Charsets.dll_981093327" Guid="c1c5c08d-d98f-4b84-8e56-c55cdab8597c">
              <File Id="File_IKVM.OpenJDK.Charsets.dll_981093327" Source="..\..\solutions\Invantive Data Generator\obin\net47\IKVM.OpenJDK.Charsets.dll">
                <NativeImage Id="NativeImage" Platform="all" xmlns="http://schemas.microsoft.com/wix/NetFxExtension" />
              </File>
            </Component>

Second WXS:

            <Component Id="Component.File_IKVM.OpenJDK.Charsets.dll_981093327" Guid="c1c5c08d-d98f-4b84-8e56-c55ceb8a697c">
              <File Id="File_IKVM.OpenJDK.Charsets.dll_981093327" Source="..\..\solutions\Invantive Data Generator\obin\net47\IKVM.OpenJDK.Charsets.dll">
                <netfx:NativeImage Id="NativeImage" Platform="all" />
              </File>
            </Component>

This not only holds for files, but also for directories, such as:

<Component Id="Component.ProgramMenuFolder.Invantive_Software_BV.Invantive_Data_Generator.EmptyDirectory" Guid="c1c5c08d-d98f-4b84-8e56-c55c56d69a65">

changing into

<Component Id="Component.ProgramMenuFolder.Invantive_Software_BV.Invantive_Data_Generator.EmptyDirectory" Guid="c1c5c08d-d98f-4b84-8e56-c55c67a8aa65">

Note that the Component Id itself does not change between generations of the WXS.

monty241 commented 5 years ago

The Guid is different across generation of the WXS.

It seems that the Guids are re-generated on every creation of WXS; they are not persistent on the WixEntity object as is the Id (string).

Tried making generation process consistent by using:

WixGuid.ConsistentGenerationStartValue = new WixGuid.SequentialGuid(CONSTANTGUID);

before every dump of WXS, but they still differ on every generation.

Then changed hashing algorithm based upon a copy from WixSharp:

        static int[] orderMap = { 15, 14, 13, 12, 11, 10, 9, 8, 6, 7, 4, 5, 0, 1, 2, 3 };

        /// <summary>
        /// Hashes the Guid by the specified integer hash value.
        /// </summary>
        /// <param name="guid">The GUID.</param>
        /// <param name="hashValue">The hash value.</param>
        /// <returns>GUID value.</returns>
        public static Guid InvantiveHashGuidByInteger(Guid guid, int hashValue)
        {
            byte[] bytes = guid.ToByteArray();
            byte[] hashBytes = BitConverter.GetBytes(hashValue);
            for (int i = 0; i < hashBytes.Length; i++)
            {
                int bytesIndex = orderMap[i];
                bytes[bytesIndex] = (byte)(bytes[bytesIndex] + hashBytes[i]);
            }

            return new Guid(bytes);
        }

        private static Guid baseGuidForWixGuidGeneration_;

        private static Guid BaseGuidForWixGuidGeneration
        {
            get
            {
                return baseGuidForWixGuidGeneration_;
            }
            set
            {
                if (baseGuidForWixGuidGeneration_ != value)
                {
                    Utility.LogLine($"Change base GUID for WIX GUID generation to '{value}'.");
                    baseGuidForWixGuidGeneration_ = value;
                }
            }
        }

and always starting with a fixed GUID:

                Compiler.GuidGenerator = InvantiveGuidGenerator;
                BaseGuidForWixGuidGeneration = this.WixManagedProject.GUID.Value;

This case generates consistent Guids, independent of how many times you create a WXS. Even better (or stranger), there is no need to reset for instance the initial seeded value.

monty241 commented 5 years ago

Tested again with old code using GuidGenerator.Default (checked). The ConsistentGenerationStartValue.CurrentGuid keeps the same value during generation of GUIDs. So the cause of the ever changing GUIDs can not be the initial seeded value.

The compId used to seed the files has been checked to be the ID string of the component (Component.File_name_hashofdir).

By looking at the source code of WIX# I seem unable to find the root cause of every changing GUIDs on the same component.

For now, I will resort to the custom implementation of the GUID generator.

monty241 commented 5 years ago

@oleg-shilo Is it maybe a good idea to store the calculated Guid (which becomes MSI ComponentID) in the object instead of calculating it every time again?

That also makes it possible for external debugging code to print the values to a file. But also ensures consistent labeling such as with Sequential generator and allows introspection.

oleg-shilo commented 5 years ago

Hi Guido, sorry or the delay. I wanted to get some time to go through your posts properly.

You have correctly identified (practically reverse engineered) what WixSharp does under the hood.

Yes the WXS code is generated in multiple steps, which include dynamic resolving of temporary placeholders and merging various WiX XML namespaces.

The only real requirement for the generated code is to be a valid WiX code and to be consistent.

From WixSharp point of view WXS code for building MSI is what the IL code for building a .NET assembly. It can be as ugly as you want but it has to be valid.

Saying that, I would use any opportunity to make the wxs code generation more deterministic if it does not require an excessive effort. But unfortunately this very effort can easily become a deciding factor for the feature/change to be included. Thus can you please assist me with solving that "of the ever changing GUIDs" problems? Can you share the simplest possible "hello-world" style sample that demonstrates the problem?

I am asking this because I don't see the described behavior in the InstallFiles sample from WixSharp samples library:

The wxs file is generated once meaning that it is either your runtime conditions are different to mine or we are not testig exactly the same code.

Is it maybe a good idea to store the calculated Guid (which becomes MSI ComponentID) in the object instead of calculating it every time again?

This is rather complicated and kind of non-beneficial as the GUID is not generated "every time again" but only once. At least it was the intention.

Let's start with the sample that would let me to see the problem.

BTW the latest release contains a new property WixEntity.ComponentId which can be used to force comp-Id if required. But it is not exactly what you are looking for as it does not allow to analyze the auto-generated IDs but only overrides them.

oleg-shilo / wixsharp