search

swiftlang / swift

The Swift Programming Language
https://swift.org
Apache License 2.0
67.39k stars 10.34k forks source link

[Windows] Support Nightly Toolchains for main and release/5.10 #69484

Open shahmishal opened 11 months ago

shahmishal commented 11 months ago

Windows toolchain needs to be built on ci-external.swift.org for main and release/5.10 branch.

shahmishal commented 11 months ago

Currently blocked on https://github.com/apple/swift/issues/69483

shahmishal commented 11 months ago

cc: @grynspan @compnerd

shahmishal commented 11 months ago

Currently blocked on getting this PR merged https://github.com/apple/swift/pull/69555

shahmishal commented 11 months ago

Job created for main branch: https://ci-external.swift.org/job/swift-main-windows-toolchain/20/

shahmishal commented 11 months ago

Job failed to find zlib 

CMake Error: The source directory "C:/Users/swift-ci/jenkins/workspace/swift-main-windows-toolchain/zlib" does not exist.

Found the issue, and started new build: https://ci-external.swift.org/job/swift-main-windows-toolchain/21/console

shahmishal commented 11 months ago

@compnerd We are seeing following test failing, it might be config issue. However, if you can look into this would be helpful. Thanks!

https://ci-external.swift.org/job/swift-main-windows-toolchain/21/console

Failed Tests (65):
  Swift(windows-x86_64) :: Driver/android-link.swift
  Swift(windows-x86_64) :: Driver/driver_mode.swift
  Swift(windows-x86_64) :: Driver/help.swift
  Swift(windows-x86_64) :: Driver/options-repl.swift
  Swift(windows-x86_64) :: Driver/print_target_info.swift
  Swift(windows-x86_64) :: Driver/sdk.swift
  Swift(windows-x86_64) :: Driver/working-directory.swift
  Swift(windows-x86_64) :: IDE/complete_macros_expanded.swift
  Swift(windows-x86_64) :: IDE/complete_optionset.swift
  Swift(windows-x86_64) :: Index/index_macros.swift
  Swift(windows-x86_64) :: Interop/CxxToSwiftToCxx/bridge-cxx-struct-back-to-cxx-execution.cpp
  Swift(windows-x86_64) :: Interop/SwiftToCxx/stdlib/swift-stdlib-in-cxx.swift
  Swift(windows-x86_64) :: Macros/accessor_macros.swift
  Swift(windows-x86_64) :: Macros/attached_macros_diags.swift
  Swift(windows-x86_64) :: Macros/composed_macro.swift
  Swift(windows-x86_64) :: Macros/extension_macro_plugin.swift
  Swift(windows-x86_64) :: Macros/if_expr.swift
  Swift(windows-x86_64) :: Macros/imported_conformance_macro.swift
  Swift(windows-x86_64) :: Macros/init_accessor.swift
  Swift(windows-x86_64) :: Macros/lazy_parsing.swift
  Swift(windows-x86_64) :: Macros/macro_and_typealias.swift
  Swift(windows-x86_64) :: Macros/macro_expand.swift
  Swift(windows-x86_64) :: Macros/macro_expand_attributes.swift
  Swift(windows-x86_64) :: Macros/macro_expand_closure.swift
  Swift(windows-x86_64) :: Macros/macro_expand_codeitems.swift
  Swift(windows-x86_64) :: Macros/macro_expand_conformances.swift
  Swift(windows-x86_64) :: Macros/macro_expand_conformances_multi.swift
  Swift(windows-x86_64) :: Macros/macro_expand_extensions.swift
  Swift(windows-x86_64) :: Macros/macro_expand_member_with_conformances.swift
  Swift(windows-x86_64) :: Macros/macro_expand_other.swift
  Swift(windows-x86_64) :: Macros/macro_expand_peers.swift
  Swift(windows-x86_64) :: Macros/macro_expand_primary.swift
  Swift(windows-x86_64) :: Macros/macro_expand_synthesized_members.swift
  Swift(windows-x86_64) :: Macros/macro_expand_throwing.swift
  Swift(windows-x86_64) :: Macros/macro_expand_variadic.swift
  Swift(windows-x86_64) :: Macros/macro_fixits.swift
  Swift(windows-x86_64) :: Macros/macro_invalid.swift
  Swift(windows-x86_64) :: Macros/macro_keywordname.swift
  Swift(windows-x86_64) :: Macros/macro_mixedpluginpath.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_basic.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_error.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_searchorder.swift
  Swift(windows-x86_64) :: Macros/macro_plugin_server.swift
  Swift(windows-x86_64) :: Macros/macro_shadowing.swift
  Swift(windows-x86_64) :: Macros/macro_swiftdeps.swift
  Swift(windows-x86_64) :: Macros/macro_vfs.swift
  Swift(windows-x86_64) :: Macros/magic_literal_expansion.swift
  Swift(windows-x86_64) :: Macros/top_level_freestanding.swift
  Swift(windows-x86_64) :: ModuleInterface/Observable.swift
  Swift(windows-x86_64) :: ModuleInterface/macros.swift
  Swift(windows-x86_64) :: Profiler/coverage_macros.swift
  Swift(windows-x86_64) :: SIL/Serialization/deserialize_stdlib.sil
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-clang-implicit.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-forwarding-module.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map-typecheck-module-from-interface.swift
  Swift(windows-x86_64) :: ScanDependencies/explicit-module-map.swift
  Swift(windows-x86_64) :: Serialization/macros.swift
  Swift(windows-x86_64) :: SourceKit/Macros/diags.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_across_modules.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_formatting.swift
  Swift(windows-x86_64) :: SourceKit/Macros/macro_semantic_token.swift
  Swift(windows-x86_64) :: SourceKit/Macros/syntactic_expansion.swift
  Swift(windows-x86_64) :: SymbolGraph/Symbols/Macro.swift
  Swift(windows-x86_64) :: SymbolGraph/Symbols/Mixins/DeclarationFragments/Full/Macros.swift
  Swift(windows-x86_64) :: stdlib/llvm-support-odr-violation.test-sh

Testing Time: 759.15s
  Skipped          :    1
  Unsupported      : 2459
  Passed           : 7591
  Expectedly Failed:   47
  Failed           :   65
compnerd commented 11 months ago

@shahmishal super exciting!

Just looking through a couple of things stand out:

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\Tools\vsdevcmd.bat" -arch=amd64 -host_arch=amd64 'C:\Program' is not recognized as an internal or external command, operable program or batch file.

With the switch to build.ps1, this shouldn't be needed (build.ps1 will properly setup the VS Dev environment to support building the SDK for all the architectures). Something that might help simplify the Jenkins configuration.

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>d /s /q C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\build 'd' is not recognized as an internal or external command, operable program or batch file.

Odd, I'm not sure where this is coming from. We should investigate this later.

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1 -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain -BinaryCache T: -ImageRoot T: -BuildType Release -Test swift,dispatch,foundation,xctest, -Stage T:\artifacts

This looks right ... and if this is failing, I doubt it's a configuration issue and more likely this is an issue in build.ps1. I'll need a little bit to dig into this a bit further to try to understand what is going on.

compnerd commented 11 months ago

There seem to be a few failures due to %swift_driver_plain being expanded to main. This would mean that config.swift = inferSwiftBinary('swift') returned main? I don't see that in the log, so I expect that it found it in the path.

shahmishal commented 11 months ago

If it helps, here is the build step: 

            "C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\Common7\\Tools\\vsdevcmd.bat" -arch=amd64 -host_arch=amd64"

            rd /s /q %LocalAppData%\\clang\\ModuleCache

            set CMAKE_BUILD_TYPE=Release
            set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

            set SourceRoot=%WORKSPACE%
            set BuildRoot=%SourceRoot%\\build

            md %BuildRoot%
            subst T: /d
            subst T: %BuildRoot% || (exit /b)

            set BuildRoot=T:

            set PackageRoot=%BuildRoot%\\artifacts

            md %PackageRoot%

            :: Setup temporary directories
            md %BuildRoot%\\tmp
            set TEMP=%BuildRoot%\\tmp
            set TMP=%BuildRoot%\\tmp
            set TMPDIR=%BuildRoot%\\tmp

            set NINJA_STATUS=[%%f/%%t][%%p][%%es]

            set TestArg=-Test swift,dispatch,foundation,xctest,

            set "SkipPackagingArg= "

            rd /s /q %WORKSPACE%\\build

            powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
            -SourceCache %SourceRoot% ^
            -BinaryCache %BuildRoot% ^
            -ImageRoot %BuildRoot% ^
            -BuildType %CMAKE_BUILD_TYPE% ^
            %SkipPackagingArg% ^
            %TestArg% ^
            -Stage %PackageRoot%
shahmishal commented 11 months ago

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>d /s /q C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\build 'd' is not recognized as an internal or external command, operable program or batch file.

Odd, I'm not sure where this is coming from. We should investigate this later.

I have a fix for this.

compnerd commented 11 months ago

It appears that a number of the failures are referencing ../lib/swift/windows and not being able to find sources under there. I think that this might be due to the split of the build tree and the source tree.

I wonder if there was something that we had done before that results in the build tree being linked or something?

compnerd commented 11 months ago

Hmm, I wonder if it is the PYTHON_HOME that is causing the issue ... we use python 3.9, but PYTHON_HOME is set to 3.7. Would it be possible to test with PYTHON_HOME unset?

compnerd commented 11 months ago

Bad Run:

lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:210: note: Using cmake: C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-frontend: t:\1\bin\swift-frontend.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swiftc: t:\1\bin\swiftc.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-opt: t:\1\bin\sil-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-func-extractor: t:\1\bin\sil-func-extractor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-llvm-gen: t:\1\bin\sil-llvm-gen.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-nm: t:\1\bin\sil-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sil-passpipeline-dumper: t:\1\bin\sil-passpipeline-dumper.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using lldb-moduleimport-test: t:\1\bin\lldb-moduleimport-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-ide-test: t:\1\bin\swift-ide-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-dependency-tool: t:\1\bin\swift-dependency-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-reflection-dump: t:\1\bin\swift-reflection-dump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-remoteast-test: t:\1\bin\swift-remoteast-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-indent: t:\1\bin\swift-indent.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-symbolgraph-extract: t:\1\bin\swift-symbolgraph-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-api-extract: t:\1\bin\swift-api-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using clang: t:\1\bin\clang.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using clang++: t:\1\bin\clang++.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-link: t:\1\bin\llvm-link.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-llvm-opt: t:\1\bin\swift-llvm-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-profdata: t:\1\bin\llvm-profdata.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-cov: t:\1\bin\llvm-cov.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-strings: t:\1\bin\llvm-strings.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using FileCheck: t:\1\bin\filecheck.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-dwarfdump: t:\1\bin\llvm-dwarfdump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-readelf: t:\1\bin\llvm-readelf.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-dis: t:\1\bin\llvm-dis.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-nm: t:\1\bin\llvm-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-readtapi: t:\1\bin\llvm-readtapi.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using llvm-size: t:\1\bin\llvm-size.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using sourcekitd-test: t:\1\bin\sourcekitd-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using complete-test: t:\1\bin\complete-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-api-digester: t:\1\bin\swift-api-digester.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-cache-tool: t:\1\bin\swift-cache-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-refactor: t:\1\bin\swift-refactor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-demangle-yamldump: t:\1\bin\swift-demangle-yamldump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-demangle: t:\1\bin\swift-demangle.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_O' program, try setting BENCHMARK_O in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_Driver' program, try setting BENCHMARK_DRIVER in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using wasm-ld: t:\1\bin\wasm-ld.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-plugin-server: t:\1\bin\swift-plugin-server.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-parse-test: t:\1\bin\swift-parse-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:385: note: Using resource dir: ..\lib\swift
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:432: note: Compiling with -swift-version 4
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:528: note: Using Clang module cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\clang-module-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:529: note: Using test results dir: T:/1/./swift-test-results/x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:534: note: Using code completion cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\completion-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:1496: note: Testing Windows x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:2253: note: Using platform module dir: ..\lib\swift\windows
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:289: note: using swift-reflection-test-windows-x86_64: t:\1\bin\swift-reflection-test-windows-x86_64.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\test\lit.cfg:2861: note: Available features: CMAKE_GENERATOR=Ninja, CODEGENERATOR=AArch64, CODEGENERATOR=ARM, CODEGENERATOR=WebAssembly, CODEGENERATOR=X86, CPU=x86_64, MSVC_VER=17.0, OS=windows-msvc, PTRSIZE=64, SWIFT_VERSION=4, VENDOR=unknown, asserts, concurrency, concurrency_runtime, cplusplus_driver, differentiable_programming, distributed, executable_test, no_asan, no_lto, nonexecutable_test, objc_codegen, observation, optimized_stdlib, optimized_stdlib_x86_64, reflection, sourcekit, string_processing, swift_only_stable_abi, swift_stdlib_no_asserts, swift_swift_parser, swift_test_mode_optimize_none, swift_test_mode_optimize_none_x86_64, swift_tools_extra, thread_safe_runtime, threading_win32

Good Run:

lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:210: note: Using cmake: C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-frontend: t:\1\bin\swift-frontend.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift: t:\1\bin\swift.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swiftc: t:\1\bin\swiftc.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-opt: t:\1\bin\sil-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-func-extractor: t:\1\bin\sil-func-extractor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-llvm-gen: t:\1\bin\sil-llvm-gen.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-nm: t:\1\bin\sil-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sil-passpipeline-dumper: t:\1\bin\sil-passpipeline-dumper.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using lldb-moduleimport-test: t:\1\bin\lldb-moduleimport-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-ide-test: t:\1\bin\swift-ide-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-dependency-tool: t:\1\bin\swift-dependency-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-reflection-dump: t:\1\bin\swift-reflection-dump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-remoteast-test: t:\1\bin\swift-remoteast-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-indent: t:\1\bin\swift-indent.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-symbolgraph-extract: t:\1\bin\swift-symbolgraph-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-api-extract: t:\1\bin\swift-api-extract.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using clang: t:\1\bin\clang.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using clang++: t:\1\bin\clang++.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-link: t:\1\bin\llvm-link.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-llvm-opt: t:\1\bin\swift-llvm-opt.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-profdata: t:\1\bin\llvm-profdata.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-cov: t:\1\bin\llvm-cov.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-strings: t:\1\bin\llvm-strings.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using FileCheck: t:\1\bin\filecheck.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-dwarfdump: t:\1\bin\llvm-dwarfdump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-readelf: t:\1\bin\llvm-readelf.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-dis: t:\1\bin\llvm-dis.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-nm: t:\1\bin\llvm-nm.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-readtapi: t:\1\bin\llvm-readtapi.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using llvm-size: t:\1\bin\llvm-size.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using sourcekitd-test: t:\1\bin\sourcekitd-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using complete-test: t:\1\bin\complete-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-api-digester: t:\1\bin\swift-api-digester.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-cache-tool: t:\1\bin\swift-cache-tool.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-refactor: t:\1\bin\swift-refactor.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-demangle-yamldump: t:\1\bin\swift-demangle-yamldump.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-demangle: t:\1\bin\swift-demangle.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_O' program, try setting BENCHMARK_O in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:292: warning: couldn't find 'Benchmark_Driver' program, try setting BENCHMARK_DRIVER in your environment
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using wasm-ld: t:\1\bin\wasm-ld.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-plugin-server: t:\1\bin\swift-plugin-server.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-parse-test: t:\1\bin\swift-parse-test.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:385: note: Using resource dir: t:\1\lib\swift
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:432: note: Compiling with -swift-version 4
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:528: note: Using Clang module cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\clang-module-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:529: note: Using test results dir: T:/1/./swift-test-results/x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:534: note: Using code completion cache: T:\1\swift-test-results\x86_64-unknown-windows-msvc\completion-cache
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:1496: note: Testing Windows x86_64-unknown-windows-msvc
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:2253: note: Using platform module dir: t:\1\lib\swift\windows
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:289: note: using swift-reflection-test-windows-x86_64: t:\1\bin\swift-reflection-test-windows-x86_64.exe
lit.py: C:\Users\swift-ci\jenkins\workspace\swift-PR-windows\swift\test\lit.cfg:2861: note: Available features: CMAKE_GENERATOR=Ninja, CODEGENERATOR=AArch64, CODEGENERATOR=ARM, CODEGENERATOR=WebAssembly, CODEGENERATOR=X86, CPU=x86_64, MSVC_VER=17.0, OS=windows-msvc, PTRSIZE=64, SWIFT_VERSION=4, VENDOR=unknown, asserts, concurrency, concurrency_runtime, cplusplus_driver, differentiable_programming, distributed, executable_test, no_asan, no_lto, nonexecutable_test, objc_codegen, observation, optimized_stdlib, optimized_stdlib_x86_64, reflection, sourcekit, string_processing, swift_only_stable_abi, swift_stdlib_no_asserts, swift_swift_parser, swift_test_mode_optimize_none, swift_test_mode_optimize_none_x86_64, swift_tools_extra, thread_safe_runtime, threading_win32

This seems to be the diff:

- lit.py: swift\test\lit.cfg:289: note: using swift: t:\1\bin\swift.exe
- lit.py: swift\test\lit.cfg:385: note: Using resource dir: t:\1\lib\swift
+ lit.py: swift\test\lit.cfg:385: note: Using resource dir: ..\lib\swift
- lit.py: swift\test\lit.cfg:2253: note: Using platform module dir: t:\1\lib\swift\windows
+ lit.py: swift\test\lit.cfg:2253: note: Using platform module dir: ..\lib\swift\windows

This does match the majority of the failures. Now the question is, why is the resource dir and platform module dir path different?

compnerd commented 11 months ago

The missing

note: using swift: t:\1\bin\swift.exe

along with https://github.com/apple/swift/blob/main/test/lit.cfg#L273-L298

seems to indicate that swift was picked up from Path. Is the image identical to the PR builder? The default value is ..\lib\swift.

shahmishal commented 11 months ago

@compnerd Let me know if I missed setting an env

compnerd commented 11 months ago

@shahmishal I don't see off hand any missing environment variables. The only ones that should be concerning are the ones that determine the flags (which repositories to test, whether to package or not, build type), but those seem to be setup.

I would say that removing the PYTHON_HOME setting might be good. One of the nice things about build.ps1 is that we shouldn't need to set up a whole slew of environment variables, and if there is something missing we should try to detect it. If we could remove running the VsDevCmd, it would also simplify the run.

Just to double check my assumption - the image is identical to the PR testing one right?

compnerd commented 11 months ago

Hmm ... @tristanlabelle pointed out that I misread the code: swift/test/lit.cfg at main · apple/swift

It seems to imply that there is a SWIFT environment variable being set?

shahmishal commented 11 months ago

@compnerd Can you try running this locally? 

 rd /s /q %LocalAppData%\\clang\\ModuleCache

 set CMAKE_BUILD_TYPE=Release
 set PYTHON_HOME="%ProgramFiles(x86)%\\Microsoft Visual Studio\\Shared\\Python37_64"

 set SourceRoot=%WORKSPACE%
 set BuildRoot=%SourceRoot%\\build

 md %BuildRoot%
 subst T: /d
 subst T: %BuildRoot% || (exit /b)

 set BuildRoot=T:

 set PackageRoot=%BuildRoot%\\artifacts

 md %PackageRoot%

 :: Setup temporary directories
 md %BuildRoot%\\tmp
 set TEMP=%BuildRoot%\\tmp
 set TMP=%BuildRoot%\\tmp
 set TMPDIR=%BuildRoot%\\tmp

 set NINJA_STATUS=[%%f/%%t][%%p][%%es]

 set TestArg=-Test swift,dispatch,foundation,xctest,

 set "SkipPackagingArg= "

 rd /s /q %WORKSPACE%\\build

 powershell.exe -ExecutionPolicy RemoteSigned -File %WORKSPACE%\\swift\\utils\\build.ps1 ^
 -SourceCache %SourceRoot% ^
 -BinaryCache %BuildRoot% ^
 -ImageRoot %BuildRoot% ^
 -BuildType %CMAKE_BUILD_TYPE% ^
 %SkipPackagingArg% ^
 %TestArg% ^
 -Stage %PackageRoot%
compnerd commented 11 months ago

@shahmishal everything now makes sense - there is an environment variable swift that is being set by Jenkins at Parameters [Jenkins] that is set to main. That is then passed along to the test suite which then uses main as the swift command instead which results in the failures.

shahmishal commented 11 months ago

this is used to pick up the correct SHA, what is the best way to unset variable?

compnerd commented 11 months ago

Depends on the shell. With cmd it would be set swift= (or whatever the variable name is). For PowerShell it would be Delete-Item env:\swift -ErrorAction Ignore.

shahmishal commented 11 months ago

Going to try set "swift= "

shahmishal commented 11 months ago

New build started: https://ci-external.swift.org/job/swift-main-windows-toolchain/25/console

compnerd commented 11 months ago

@shahmishal if there is a trailing space, I think it might fail as it will set it to the string containing a space, there should be nothing after the =.

compnerd commented 11 months ago

C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>set "swift= "

The trailing space is setting the string to " " rather than deleting the environment variable.

shahmishal commented 11 months ago

main branch built: https://ci-external.swift.org/job/swift-main-windows-toolchain/26/

compnerd commented 11 months ago

Awesome! The 5.10 cherry-pick is at https://github.com/apple/swift/pull/69608

shahmishal commented 11 months ago

release/5.10 job created: https://ci-external.swift.org/job/swift-5.10-windows-toolchain/

shahmishal commented 11 months ago 
swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-5.10-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-5.10-windows-toolchain\swift\utils\build.ps1             -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-5.10-windows-toolchain             -BinaryCache T:             -ImageRoot T:             -BuildType Release                           -Test swift,dispatch,foundation,xctest,             -Stage T:\artifacts 
Swift toolchain not found. Downloading from swift.org...
Installing Swift toolchain...
T:\toolchains\ : warning WIX8503: Failed to extract attached container. This most often happens when extracting a stripped bundle from the package cache, which is not supported.
Error: msiexec.exe exited with code 1619.

Wonder if it has anything to do with job name

shahmishal commented 11 months ago

Removed . from the job name, let's see if it helps https://ci-external.swift.org/job/swift-5_10-windows-toolchain/5/console

shahmishal commented 11 months ago

@compnerd do you know whats going on?

compnerd commented 11 months ago

Hmm, not offhand. Error 1619 is ERROR_INSTALL_PACKAGE_OPEN_FAILED.

This installation package couldn't be opened. Verify that the package exists and is accessible, or contact the application vendor to verify that this is a valid Windows Installer package.

Can you verify if the path (T:\toolchains\a0) exists and is non-zero sized? Either way, I think that we should hardcode the checksum for the expected binaries and ensure that they match. I'll try to figure out how to do that in a separate change.

compnerd commented 11 months ago

I think I might know what is going on. There is a bug in the download URL, and so we were failing to download the installer. This will need a little bit of help (we want to clean up the old 404 output). This should be caught in the future by the pinned SHA checks.

shahmishal commented 11 months ago

Do we know why this is only happening on release/5.10 and not on main?

compnerd commented 11 months ago

I suspect that the URL was correct at one point and the artifact was cached. Without being able to inspect the filesystem, it is difficult to understand what is going on.

compnerd commented 11 months ago

Put up https://github.com/apple/swift/pull/69663 to cherry-pick the changes to 5.10.

shahmishal commented 11 months ago

New build started with the cherry-pick merged into release/5.10 - https://ci-external.swift.org/job/swift-5_10-windows-toolchain/7/console

compnerd commented 10 months ago

Put up https://github.com/apple/swift/pull/69693 to cherry-pick some changes to help with the code signing.

shahmishal commented 10 months ago

@compnerd do we have any integration tests we can run on installer? If not, can we add them to swift-integration-tests repository

compnerd commented 10 months ago

@shahmishal I don't think that there is in the current state. If we had docker, we could easily build and test https://github.com/compnerd/swift-win32 as a smoke test (which is something that I do).

shahmishal commented 10 months ago

What is the status for hub.docker.com docker image?

compnerd commented 10 months ago

Pinged the Docker folks again, all the feedback has been addressed.

compnerd commented 10 months ago

@shahmishal is that a blocker for getting the snapshots posted? I'm trying to understand where we are - we haven't had any snapshots for 5.10 yet, and I'm worried that we might get into a state where we cannot get the necessary changes merged and thus end up with a release on Windows that is not very usable.

shahmishal commented 10 months ago

No, it's not blocked on Windows integration testing or Docker image.

The next step is for me to integrate the trigger from ci.swift.com to ci-external.swift.org.

Plan on working on this integration in the next few days. Hopefully, it will be done before Nov 16th.

shahmishal commented 10 months ago

Hopefully with this revert we will be able to get nightly toolchain unblocked. https://github.com/apple/swift/pull/69807

compnerd commented 10 months ago

The 5.10 toolchain is not usable due to missing files: https://github.com/apple/swift-installer-scripts/pull/258 should resolve that part.

shahmishal commented 10 months ago

New build being created for main branch for Windows nightly: https://ci-external.swift.org/job/swift-main-windows-toolchain/27/console

shahmishal commented 10 months ago

@compnerd Do you know what's going here? 

swift-ci@EC2AMAZ-JCVOGM7 C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain>powershell.exe -ExecutionPolicy RemoteSigned -File C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain\swift\utils\build.ps1             -SourceCache C:\Users\swift-ci\jenkins\workspace\swift-main-windows-toolchain             -BinaryCache T:             -ImageRoot T:             -BuildType Release                           -Test swift,dispatch,foundation,xctest,             -Stage T:\artifacts 
WiX not found. Downloading from nuget.org...
Extracting WiX...
Swift toolchain not found. Downloading from swift.org...
Extracting Swift toolchain...
T:\toolchains\ : warning WIX8503: Failed to extract attached container. This most often happens when extracting a stripped bundle from the package cache, which is not supported.
Error: msiexec.exe exited with code 1619.
compnerd commented 10 months ago

@shahmishal hmm, that sounds like a download failure for the swift toolchain from swift.org? That is very weird, I don't think that I've seen that since we fixed this in build.ps1 to do the validation of the downloaded image.

shahmishal commented 10 months ago

Rebuilding see if it helps https://ci-external.swift.org/job/swift-main-windows-toolchain/28/console