[Windows] Support Nightly Toolchains for main and release/5.10

[shahmishal]

Windows toolchain needs to be built on ci-external.swift.org for main and release/5.10 branch.

[shahmishal]

It failed again for same error:

T:\toolchains\ : warning WIX8503: Failed to extract attached container. This most often happens when extracting a stripped bundle from the package cache, which is not supported.
Error: msiexec.exe exited with code 1619.

[compnerd]

@shahmishal we don't delete the installer if it existing, it might require some manual intervention to clean that up.

[shahmishal]

Where does the installer get installed? is it in T:\?

[compnerd]

Yeah, the installer itself is downloaded to T: but we extract the content into T:\toolchains IIRC

[shahmishal]

Can we delete this dir if it invalid and re-try extracting the toolchain?

[shahmishal]

@compnerd its still failing on another node, https://ci-external.swift.org/job/swift-main-windows-toolchain/29/console

[shahmishal]

I thought this would delete the virtual drive:

            subst T: /d

[compnerd]

Yes, that removes the "virtual drive", but does not remove the contents that were mapped there.

[shahmishal]

I also have rd /s /q %WORKSPACE%\\build in the job, so it should download new toolchain every time.

            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b)

[compnerd]

Interesting - and the download doesn't say that the toolchain installer is invalid. So, it could be that it is failing to pass the signature check :/ I don't think that I have a way to diagnose this without actually interacting with the host.

[shahmishal]

Can we add verbose mode to the signature check?

[shahmishal]

msiexec_failure.log Attached log from

msiexec.exe /lv T:\toolchains\log.log /qn /a T:\toolchains\a0 TARGETDIR=T:\toolchains\swift-5.9-RELEASE-windows10

MSI (s) (E8:5C) [01:08:01:773]: Found shell folder  by spelunking through the registry.
MSI (s) (E8:5C) [01:08:01:773]: Note: 1: 2103 2: 26 
MSI (s) (E8:5C) [01:08:01:774]: Error: Failed to Create AppData Cache Path with Product ID
MSI (s) (E8:5C) [01:08:01:774]: Note: 1: 2205 2:  3: Error 
MSI (s) (E8:5C) [01:08:01:774]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 2103 
DEBUG: Error 2103:  Could not resolve path for shell folder 26.
MSI (s) (E8:5C) [01:08:01:778]: Note: 1: 2205 2:  3: Error 
MSI (s) (E8:5C) [01:08:01:778]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709 
MSI (s) (E8:5C) [01:08:01:778]: Product: Swift Runtime for Windows x86_64 -- The installer has encountered an unexpected error installing this package. This may indicate a problem with this package. The error code is 2103. The arguments are: 26, , 

[shahmishal]

bat '''
            "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\vsdevcmd.bat -arch=amd64 -host_arch=amd64"

            rd /s /q %LocalAppData%\\clang\\ModuleCache

            set CMAKE_BUILD_TYPE=Release
            set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

            set SourceRoot=%WORKSPACE%
            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b 1)

            set BuildRoot=T:

            set PackageRoot=%BuildRoot%\\artifacts

            md %PackageRoot%

            :: Setup temporary directories
            md %BuildRoot%\\tmp
            set TEMP=%BuildRoot%\\tmp
            set TMP=%BuildRoot%\\tmp
            set TMPDIR=%BuildRoot%\\tmp

            set NINJA_STATUS=[%%f/%%t][%%p][%%es]

            set TestArg=-Test swift,dispatch,foundation,xctest

            set "SkipPackagingArg= "
            set "swift="

            rd /s /q %WORKSPACE%\\build

            powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
            -SourceCache %SourceRoot% ^
            -BinaryCache %BuildRoot% ^
            -ImageRoot %BuildRoot% ^
            -BuildType %CMAKE_BUILD_TYPE% ^
            %SkipPackagingArg% ^
            %TestArg% ^
            -Stage %PackageRoot% || (exit /b 1)
            '''

Thats the current script, let me know if I missed something Re-running it https://ci-external.swift.org/job/swift-main-windows-toolchain/31/console

[compnerd]

I was thinking of adding the /lvx! ... to the arguments, seems that you did the same thing! This is interesting - the error is definitely Error 2103: Could not resolve path for shell folder 26.. Now, the question is why is that happening 🤔

[compnerd]

I wonder if the builders are running as different users. LOCAL SYSTEM (service account) might encounter some issues - I wonder if we can quickly test with:

msiexec.exe /lvx! T:\toolchains\log.log /qn /a T:\toolchains\a0 TARGETDIR=T:\toolchains\swift-5.9-RELEASE-windows10 ALLUSERS=1

[compnerd]

https://github.com/apple/swift/pull/69852 adds the logging for future use as well.

[compnerd]

@shahmishal seems that the last re-run succeeded

[shahmishal]

@compnerd sent link in private for review for nightly main toolchain.

[compnerd]

@shahmishal - reviewed, seems ready to go to me!

[shahmishal]

main branch nightly Windows toolchain available on swift.org

[compnerd]

@shahmishal - it seems that the main branch nightlies do not auto-update? There was a snapshot from 11/27, but the other platforms are on 12/7.

[compnerd]

@shahmishal any progress on the auto-updated nightlies? It seems that the Windows target is out of date by a month.

[compnerd]

ping again on this @shahmishal - the Windows snapshot on main is now ~2m out of date and we still do not have a 5.10 release snapshot. This really risks the 5.10 release as we will need extra time to validate the Windows releases.

[shahmishal]

@compnerd I am looking into getting this configured, due to other tasks I was not able to get to this. Hopefully, I will be able to focus on this soon.

[shahmishal]

Started new builds

• 5.10 - https://ci-external.swift.org/job/swift-5.10-windows-toolchain/11/console
• main - https://ci-external.swift.org/job/swift-main-windows-toolchain/33/console

[shahmishal]

main failed:

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1             -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain             -BinaryCache T:             -ImageRoot T:             -BuildType Release                           -Test swift,dispatch,foundation,xctest             -Stage T:\artifacts   || (exit /b 1 ) 
C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1 : A parameter cannot be found 
that matches parameter name 'BuildType'.
    + CategoryInfo          : InvalidArgument: (:) [build.ps1], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : NamedParameterNotFound,build.ps1

cc: @compnerd

[compnerd]

Ah, right, we dropped the -BuildType parameter a while ago to allow us to support sccache. You should just be able to remove the -BuildType Release from the invocation (it will always generate release mode builds).

[shahmishal]

New build started: https://ci-external.swift.org/job/swift-main-windows-toolchain/34/console

[shahmishal]

Ah, right, we dropped the -BuildType parameter a while ago to allow us to support sccache. You should just be able to remove the -BuildType Release from the invocation (it will always generate release mode builds).

Can we also cherry-pick this to release/5.10?

[compnerd]

Sure, let me look into that!

[shahmishal]

Nightly 5.10 released on swift.org - https://github.com/apple/swift-org-website/pull/488

[compnerd]

Trying to pull in the changes for -BuildType into 5.10 at https://github.com/apple/swift/pull/71117