search

bazelbuild / continuous-integration

Bazel's Continuous Integration Setup
https://buildkite.com
Apache License 2.0
253 stars 134 forks source link

Bazel binaries built with the latest MSVC version (14.40.33807) crash with older vcruntime #1967

Open fmeum opened 1 month ago

fmeum commented 1 month ago

Description of the bug:

In GitHub Actions on a Windows runner, Bazelisk fails to launch Bazel 7.2.0rc3, while 7.2.0rc2 on Windows and 7.2.0rc3 on other OSes work just fine (see https://github.com/fmeum/buildozer/pull/15).

The log output from the runner:

2024-06-06T07:15:48.7574869Z ##[group]Run bazelisk run //:run_buildozer
2024-06-06T07:15:48.7575484Z bazelisk run //:run_buildozer
2024-06-06T07:15:48.7575868Z bazelisk test //...
2024-06-06T07:15:48.7611914Z shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
2024-06-06T07:15:48.7612417Z env:
2024-06-06T07:15:48.7612670Z   USE_BAZEL_VERSION: 7.2.0rc3
2024-06-06T07:15:48.7613019Z ##[endgroup]
2024-06-06T07:15:59.3619476Z 2024/06/06 07:15:59 Downloading https://releases.bazel.build/7.2.0/rc3/bazel-7.2.0rc3-windows-x86_64.exe...
2024-06-06T07:16:00.9729834Z ##[error]Process completed with exit code 1.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

fmeum commented 1 month ago

@meteorcloudy

fmeum commented 1 month ago

last_green fails in a similar way.

meteorcloudy commented 1 month ago

Let me take a look now

meteorcloudy commented 1 month ago

Wow, turned out to be https://github.com/bazelbuild/bazel/commit/759fe7d7f3fdf142490e970457f72180f4d1b59a

meteorcloudy commented 1 month ago

Not sure why it's not caught in presubmit

meteorcloudy commented 1 month ago

OK, we also did a windows VM update around the same time, maybe that's the real cause.

fmeum commented 1 month ago

If https://github.com/bazelbuild/bazel/commit/759fe7d7f3fdf142490e970457f72180f4d1b59a turns out to be the culprit, I'll never consider any change safe for cherry-picking anymore :-D

meteorcloudy commented 1 month ago

The binary runs fine on the same VM image where it was built:

C:\Users\pcloudy\workdir>set USE_BAZEL_VERSION=7.2.0rc3

C:\Users\pcloudy\workdir>bazel
2024/06/06 09:04:06 Downloading https://releases.bazel.build/7.2.0/rc3/bazel-7.2.0rc3-windows-x86_64.exe...
Downloading: 63 MB out of 63 MB (100%)
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
                                                        [bazel release 7.2.0rc3]
Usage: bazel <command> <options> ...

Available commands:
meteorcloudy commented 1 month ago

I can confirm building with MSVC 2022 14.40.33807 causes this problem, but not when I switch to MSVC 2019.

meteorcloudy commented 1 month ago

I still haven't figured out a fix, but I can revert the Windows VM image first.

However, this would revert https://github.com/bazelbuild/continuous-integration/issues/1962 /cc @ahumesky @tomrenn

meteorcloudy commented 1 month ago

I've rebuilt Bazel binaries at HEAD, and hopefully that solves the issue.

@keertk We need to rerun the release pipeline for rc3, too.

meteorcloudy commented 1 month ago

OK, rc3 release have been refreshed, also fixed last_green. I'll trigger builds to fix commits on master and release-7.2.0 retrospectively.

fmeum commented 1 month ago

Could be related to https://github.com/actions/runner-images/issues/10004#issuecomment-2153091255

benjaminp commented 1 month ago

Could 8.0.0-pre.20240530.1 be refreshed, too? It seems to have been afflicted.

meteorcloudy commented 1 month ago

@benjaminp 8.0.0-pre.20240530.1 should be fixed now, can you verify?

benjaminp commented 1 month ago

https://releases.bazel.build/8.0.0/rolling/8.0.0-pre.20240530.1/bazel-8.0.0-pre.20240530.1-windows-x86_64.exe still seems affected. (The download server reports "June 6" as the last modified time.)

meteorcloudy commented 1 month ago

I see, I only refreshed 3e2aec2fdefea213cde0c0fc41eb40221995d1ca. Now reran the release pipeline, should be fixed now.