search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.23k stars 4.07k forks source link

libtool_check_unique failed to generate #14113

Open samhowes opened 3 years ago

samhowes commented 3 years ago

Description of the problem / feature request:

"/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 82, column 13, in _compile_cc_file
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout:

More context:

INFO: Repository local_config_cc instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:368:13: in <toplevel>
  /private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl:184:16: in cc_configure
Repository rule cc_autoconf defined at:
  /private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl:145:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
        configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 154, column 25, in configure_osx_toolchain
        _compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 82, column 13, in _compile_cc_file
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: 
ERROR: Error fetching repository: Traceback (most recent call last):
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
        configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 154, column 25, in configure_osx_toolchain
        _compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 82, column 13, in _compile_cc_file
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: 
ERROR: /private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/io_bazel_rules_go/BUILD.bazel:88:17: @io_bazel_rules_go//:cgo_context_data depends on @local_config_cc//:cc-compiler-darwin_x86_64 in repository @local_config_cc which failed to fetch. no such package '@local_config_cc//': libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: 
ERROR: Analysis of target '//tests/sanity:sanity' failed; build aborted: Analysis failed

build buddy link: https://app.buildbuddy.io/invocation/ee1309f6-609f-406d-92dd-eea35b849873

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No idea. This happened on my Azure Pipelines MAC OSX CI machine. Historically, this has not happened. Filing this issue because the error message said so.

This works fine locally on my own macbook pro 2015

What operating system are you running Bazel on?

Azure Pipelines macOS-latest vmImage:

Agent name: 'Azure Pipelines 2'
Agent machine name: 'Mac-1634248436964'
Current agent version: '2.193.1'
Operating System
Mac OS X
10.15.7
19H1417
Virtual Environment
Environment: macos-10.15
Version: 20211011.2
Included Software: https://github.com/actions/virtual-environments/blob/macOS-10.15/20211011.2/images/macos/macos-10.15-Readme.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/macOS-10.15%2F20211011.2
Virtual Environment Provisioner
1.0.0.0-master-20211007-1
Current image version: '20211011.2'

What's the output of bazel info release?

unkown: can't access build machine Build buddy reports this as: bazel v4.2.1 build

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

Original Azure Pipelines Build: succeeded on ubuntu-18.0.4 and windows-latest: https://dev.azure.com/samhowes/rules_msbuild/_build/results?buildId=446&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=a5d3621d-4bc4-454e-8348-6c36583ed13d

keith commented 3 years ago

Interesting. Do you know if the machine has run any other developer tools commands on macOS, like sudo xcodebuild -runFirstLaunch? A timeout to me implies macOS popped up a modal that it's expecting user input to 🙃

samhowes commented 3 years ago

Oh dear. I certainly hope it's not waiting for user input!

I don't have a way to verify if that exact machine ran that command. I have another build that just finished right after that one that appears to have succeeded, though it was scheduled on a different machine: https://dev.azure.com/samhowes/rules_msbuild/_build/results?buildId=447&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=75f7fdc6-e3bf-521e-826d-61fdbd36523a

Agent name: 'Azure Pipelines 3'
Agent machine name: 'Mac-1634248245031

These are Microsoft's standard cloud hosted, free agents though, I'd be surprised if that hadn't been run? https://github.com/actions/virtual-environments/blob/macOS-10.15/20211011.2/images/macos/macos-10.15-Readme.md

It looks like that is in one of the setup scripts from powershell to run, so unless that script didn't execute on that exact machine, it seems unlikely https://github.com/actions/virtual-environments/blob/b7f276c003aea42575b52247bdb2183e355fca2f/images/macos/helpers/Xcode.Installer.psm1#L160

keith commented 3 years ago

I would also be surprised, but it's quite suspect that another run would pass

samhowes commented 3 years ago

This popped up again today. Different workspace, but still on azure machines:

INFO: Repository local_config_cc instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:368:13: in <toplevel>
  /private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl:184:16: in cc_configure
Repository rule cc_autoconf defined at:
  /private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl:145:30: in <toplevel>
Analyzing: 49 targets (37 packages loaded, 255 targets configured)
ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
        configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 154, column 25, in configure_osx_toolchain
        _compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 82, column 13, in _compile_cc_file
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: 
ERROR: Error fetching repository: Traceback (most recent call last):
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
        configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 154, column 25, in configure_osx_toolchain
        _compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
    File "/private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 82, column 13, in _compile_cc_file
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: 
ERROR: /private/var/tmp/_bazel_runner/3d1a7f52fd47fc9a8db71770aacd13e5/external/remote_coverage_tools/BUILD:9:12: @remote_coverage_tools//:Main depends on @local_config_cc//:cc-compiler-darwin_x86_64 in repository @local_config_cc which failed to fetch. no such package '@local_config_cc//': libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout:

https://dev.azure.com/samhowes/rules_msbuild/_build/results?buildId=570&view=logs&jobId=a82d9e72-1de4-5bd6-c188-90296135479e&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=bd9c9eb7-ca76-5614-cc1d-91d849d3c79b

https://app.buildbuddy.io/invocation/ed53b9ad-5bee-41b0-a091-3d26c0751a29#details

gregestren commented 2 years ago

Issue triager here: @keith do you think this belongs here or at https://github.com/bazelbuild/rules_apple?

keith commented 2 years ago

here as it's part of the toolchain setup that lives in this repo, but I assume this is an Xcode environment issue still

phst commented 2 years ago

BTW, this also happens for me. On GitHub Actions, Bazel appears to be so flaky (especially on macOS) that I resorted to have all Bazel commands retry automatically with a one-minute sleep in between attempts. It would be nice if Bazel could generally retry non-deterministic/network-heavy actions more aggressively and increase timeouts if it detects that it's running on a CI system.

Example log in my case

ERROR: /private/var/tmp/_bazel_runner/2bf12a9d4bce6220d3e3c3743b68e3b3/external/io_bazel_rules_go/BUILD.bazel:88:17: @io_bazel_rules_go//:cgo_context_data depends on @local_config_cc//:cc-compiler-darwin_x86_64 in repository @local_config_cc which failed to fetch. no such package '@local_config_cc//': libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout: 
ERROR: Analysis of target '@com_github_bazelbuild_buildtools//buildifier:buildifier' failed; build aborted: Analysis failed
INFO: Elapsed time: 144.791s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (40 packages loaded, 7793 targets configured)
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully (40 packages loaded, 7793 targets configured)
samhowes commented 2 years ago

Not surprised this is happening on GitHub actions as well. AFAIK they are the same images that Azure Pipelines uses.

phst commented 2 years ago

BTW, is there a good workaround? Unfortunately simply retrying doesn't actually help in this case because the broken repository contents are still in the cache. Is there a way to clean out a specific external repository without having to do a full bazel clean --expunge?

keith commented 2 years ago

If you are not using Xcode (only vanilla clang builds with the Xcode command line tools), you can set BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in the environment to skip this setup.

If you know the specific path to Xcode, you can try setting DEVELOPER_DIR=/path/to/xcode in the environment which might help.

If you know the specific versions of Xcode your CI uses you can follow this approach https://www.smileykeith.com/2021/03/08/locking-xcode-in-bazel/ to hopefully reduce some of the time here.

Otherwise I would recommend adding a pre-bazel step on CI to debug compiling a single C++ file. Here is the code (for bazel 4.2.1) that seems to be taking longer than 30 seconds on those machines to run:

https://github.com/bazelbuild/bazel/blob/02ad3e3bc6970db11fe80f966da5707a6c389fdd/tools/cpp/osx_cc_configure.bzl#L56-L73

phst commented 2 years ago

Thanks, I'll try that!

BalestraPatrick commented 2 years ago

We're still seeing this issue pretty often to be annoying enough on our macOS CI. Is there any update on the investigation?

keith commented 2 years ago

Are you using GA too? AFAIUI this isn't a bazel issue

BalestraPatrick commented 2 years ago

@keith What's GA? I can reproduce this by doing the following:

  1. bazelisk clean --expunge
  2. bazelisk shutdown
  3. rm -rf our disk and repo caches.
  4. The first build will fail with this error.
keith commented 2 years ago

sorry github actions. if you can reproduce that consistently maybe you can see if it's a timeout or something else?

jayvdb commented 2 years ago

I encountered a similar error when running the https://github.com/angular/angular/ test suite, using bazel 4.2.1_1 & 5.2.0 from brew on macos 12.4 (21F79). bazel does inform me that the project requires bazel 5.0.0, but it doesnt seem that should cause this error.

angular % yarn test:ci
yarn run v1.22.17
$ bazelisk test -- //... -//devtools/... -//aio/...
INFO: Repository local_config_cc instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:487:13: in <toplevel>
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/cc_configure.bzl:184:16: in cc_configure
Repository rule cc_autoconf defined at:
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/cc_configure.bzl:145:30: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
        configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 211, column 25, in configure_osx_toolchain
        _compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 136, column 37, in _compile_cc_file
        _compile_cc_file_single_arch(repository_ctx, src_name, out_name)
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 83, column 13, in _compile_cc_file_single_arch
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout:
ERROR: /DEFAULT.WORKSPACE.SUFFIX:487:13: fetching cc_autoconf rule //external:local_config_cc: Traceback (most recent call last):
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/cc_configure.bzl", line 125, column 32, in cc_autoconf_impl
        configure_osx_toolchain(repository_ctx, cpu_value, overriden_tools)
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 211, column 25, in configure_osx_toolchain
        _compile_cc_file(repository_ctx, libtool_check_unique_src_path, "libtool_check_unique")
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 136, column 37, in _compile_cc_file
        _compile_cc_file_single_arch(repository_ctx, src_name, out_name)
    File "/private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/bazel_tools/tools/cpp/osx_cc_configure.bzl", line 83, column 13, in _compile_cc_file_single_arch
        fail(out_name + " failed to generate. Please file an issue at " +
Error in fail: libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout:
INFO: Repository org_chromium_chromium_macos_x64 instantiated at:
  /Users/jayvdb/ng/angular/WORKSPACE:128:21: in <toplevel>
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/npm/@angular/dev-infra-private/bazel/browsers/browser_repositories.bzl:14:33: in browser_repositories
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/npm/@angular/dev-infra-private/bazel/browsers/chromium/chromium.bzl:26:20: in define_chromium_repositories
Repository rule browser_archive defined at:
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/npm/@angular/dev-infra-private/bazel/browsers/browser_archive_repo.bzl:71:34: in <toplevel>
INFO: Repository org_mozilla_firefox_macos instantiated at:
  /Users/jayvdb/ng/angular/WORKSPACE:128:21: in <toplevel>
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/npm/@angular/dev-infra-private/bazel/browsers/browser_repositories.bzl:15:32: in browser_repositories
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/npm/@angular/dev-infra-private/bazel/browsers/firefox/firefox.bzl:26:20: in define_firefox_repositories
Repository rule browser_archive defined at:
  /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/npm/@angular/dev-infra-private/bazel/browsers/browser_archive_repo.bzl:71:34: in <toplevel>
ERROR: /private/var/tmp/_bazel_jayvdb/a8ba4b4a9d6093f1eb33eb3729b6a6e4/external/rules_pkg/pkg/private/BUILD:94:10: @rules_pkg//pkg/private:build_tar depends on @local_config_cc//:cc-compiler-darwin_x86_64 in repository @local_config_cc which failed to fetch. no such package '@local_config_cc//': libtool_check_unique failed to generate. Please file an issue at https://github.com/bazelbuild/bazel/issues with the following:
return code 256, stderr: Timed out, stdout:
ERROR: Analysis of target '//integration/service-worker-schema:test' failed; build aborted:
INFO: Elapsed time: 137.102s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (18 packages loaded, 26265 targets configured)
FAILED: Build did NOT complete successfully (18 packages loaded, 26265 targets configured)
    Fetching https://storage.googleapis.com/chromium-browser-snapshots/Mac/950435/chrome-mac.zip; 103,439,202B 106s
    Fetching https://ftp.mozilla.org/pub/firefox/releases/97.0/mac/en-US/Firefox%2097.0.dmg; 41,006,458B 104s
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
jayvdb@192-168-1-109 angular % bazel --help
ERROR: The project you're trying to build requires Bazel 5.0.0 (specified in /Users/jayvdb/ng/angular/.bazelversion), but it wasn't found in /usr/local/Cellar/bazel/5.2.0/libexec/bin.

Bazel binaries for all official releases can be downloaded from here:
  https://github.com/bazelbuild/bazel/releases

You can download the required version directly using this command:
  (cd "/usr/local/Cellar/bazel/5.2.0/libexec/bin" && curl -fLO https://releases.bazel.build/5.0.0/release/bazel-5.0.0-darwin-x86_64 && chmod +x bazel-5.0.0-darwin-x86_64)