search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.97k stars 4.03k forks source link

py_binary requires a C++ toolchain #8751

Open ghost opened 5 years ago

ghost commented 5 years ago

Description of the problem / feature request:

On a Linux machine without GCC installed, I am unable to build py_binary targets out of the box.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ touch WORKSPACE
$ cat > BUILD <<EOF
> py_binary(
>     name = "dummy",
>     srcs = ["dummy.py"],
> )
> EOF
$ touch dummy.py
$ bazel build :dummy
...
Auto-Configuration Error: Cannot find gcc or CC; either correct your path or set the CC environment variable

What operating system are you running Bazel on?

CentOS 7.2

What's the output of bazel info release?

release 0.26.0+vmware

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

n/a

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

n/a

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

From https://github.com/bazelbuild/bazel/issues/5133, setting BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 leads to a different, more concise error:

$ BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 bazel build :dummy
ERROR: While resolving toolchains for target //:dummy: no matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type
ERROR: Analysis of target '//:dummy' failed; build aborted: no matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type
INFO: Elapsed time: 0.092s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded, 0 targets configured)

Surprisingly, I am able to work around this by setting CC=/bin/false, confirming that a C/C++ compiler isn't actually required.

keith commented 5 years ago

I believe this was fixed by https://github.com/bazelbuild/bazel/commit/c73aa26a0807f6dd8e6c865ce67daf350e03a59b

More conversation in the bazel slack https://bazelbuild.slack.com/archives/CA306CEV6/p1560966350017900

ghost commented 5 years ago

I believe this was fixed by c73aa26

More conversation in the bazel slack https://bazelbuild.slack.com/archives/CA306CEV6/p1560966350017900

Oh, neat! I'll backport / build Bazel at HEAD and try again.

ghost commented 5 years ago

I believe this was fixed by c73aa26 More conversation in the bazel slack https://bazelbuild.slack.com/archives/CA306CEV6/p1560966350017900

Oh, neat! I'll backport / build Bazel at HEAD and try again.

I don't want to rule out PEBKAC just yet, but I'm seeing the same behavior after cherry-picking this back to our 0.26.0 branch. I'll double check my work and try building Bazel from master next week.

ishikhman commented 5 years ago

@beasleyr-vmw please update this issue ones you do the test with HEAD, thanks!

ghost commented 5 years ago

@beasleyr-vmw please update this issue ones you do the test with HEAD, thanks!

Yup, running into the same problem (but with a new reporting bug) on master (HEAD is e6ab3c41a0):

user@host dummy> bazel build :dummy
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Invocation ID: 7d1dd6a1-a73c-4f70-b762-31a50ab14a70
INFO: Call stack for the definition of repository 'local_config_cc' which is a cc_autoconf (rule definition at .../external/bazel_tools/tools/cpp/cc_configure.bzl:121:15):
 - .../external/bazel_tools/tools/cpp/cc_configure.bzl:164:5
 - /DEFAULT.WORKSPACE.SUFFIX:315:1
ERROR: An error occurred during the fetch of repository 'local_config_cc':
   Traceback (most recent call last):
        File ".../external/bazel_tools/tools/cpp/cc_configure.bzl", line 119
                configure_unix_toolchain(repository_ctx, cpu_value, overriden...)
        File ".../external/bazel_tools/tools/cpp/unix_cc_configure.bzl", line 298, in configure_unix_toolchain
                _find_generic(repository_ctx, "gcc", "CC", overriden...)
        File ".../external/bazel_tools/tools/cpp/unix_cc_configure.bzl", line 269, in _find_generic
                auto_configure_fail(msg)
        File ".../external/bazel_tools/tools/cpp/lib_cc_configure.bzl", line 112, in auto_configure_fail
                fail(("\n%sAuto-Configuration Error:%...)))

Auto-Configuration Error: Cannot find gcc or CC; either correct your path or set the CC environment variable
Internal error thrown during build. Printing stack trace: java.util.UnknownFormatConversionException: Conversion = '.'
user@host dummy> BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 bazel build :dummy                                      
Starting local Bazel server and connecting to it...                                                                                                                                                                                       
INFO: Invocation ID: 69ca4588-705b-473c-9245-9dedb7abc267                                                                                                                                                                                 
ERROR: While resolving toolchains for target //:dummy: no matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type
ERROR: Analysis of target '//:dummy' failed; build aborted: no matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type
INFO: Elapsed time: 3.016s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (10 packages loaded, 24 targets configured)

Scanning bazel query 'deps(:dummy)' output and wondering if there is any possibility this is somehow coming in through the zipper and launcher dependencies. (See https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPyBinaryRule.java#L45-L52 .)

user@host dummy> CC=/bin/false BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 bazel query 'deps(:dummy)'                        
//:dummy                                                  
@bazel_tools//tools/zip:zipper                            
@bazel_tools//tools/zip:zipper/zipper                     
@bazel_tools//third_party/ijar:zipper                     
@bazel_tools//third_party/ijar:zip_main.cc                
@bazel_tools//third_party/ijar:zip                        
@bazel_tools//third_party/ijar:zlib_client                
@bazel_tools//third_party/zlib:zlib                       
@bazel_tools//third_party/zlib:zutil.h                    
@bazel_tools//third_party/zlib:zutil.c                    
@bazel_tools//third_party/zlib:zlib.h                     
@bazel_tools//third_party/zlib:zconf.h                    
@bazel_tools//third_party/zlib:uncompr.c                  
@bazel_tools//third_party/zlib:trees.h                    
@bazel_tools//third_party/zlib:trees.c                    
@bazel_tools//third_party/zlib:inftrees.h                 
@bazel_tools//third_party/zlib:inftrees.c                 
@bazel_tools//third_party/zlib:inflate.h                  
@bazel_tools//third_party/zlib:inflate.c                  
@bazel_tools//third_party/zlib:inffixed.h                 
@bazel_tools//third_party/zlib:inffast.h                  
@bazel_tools//third_party/zlib:inffast.c                  
@bazel_tools//third_party/zlib:infback.c                  
@bazel_tools//third_party/zlib:gzwrite.c                  
@bazel_tools//third_party/zlib:gzread.c                   
@bazel_tools//third_party/zlib:gzlib.c                    
@bazel_tools//third_party/zlib:gzguts.h                   
@bazel_tools//third_party/zlib:gzclose.c                  
@bazel_tools//third_party/zlib:deflate.h                  
@bazel_tools//third_party/zlib:deflate.c                  
@bazel_tools//third_party/zlib:crc32.h                    
@bazel_tools//third_party/zlib:crc32.c                    
@bazel_tools//third_party/zlib:compress.c                 
@bazel_tools//third_party/zlib:adler32.c                  
@bazel_tools//third_party/ijar:zlib_client.h              
@bazel_tools//third_party/ijar:zlib_client.cc             
@bazel_tools//third_party/ijar:zip.h                      
@bazel_tools//third_party/ijar:zip.cc                     
@bazel_tools//third_party/ijar:platform_utils             
@bazel_tools//third_party/ijar:platform_utils.h           
@bazel_tools//third_party/ijar:platform_utils.cc          
@bazel_tools//third_party/ijar:mapped_file_windows.cc     
@bazel_tools//third_party/ijar:mapped_file_unix.cc        
@bazel_tools//third_party/ijar:mapped_file.h              
@bazel_tools//third_party/ijar:common.h                   
@bazel_tools//src:windows                                 
@bazel_tools//tools/python:2to3                           
@bazel_tools//tools/launcher:launcher                     
@bazel_tools//tools/launcher:launcher_windows             
@bazel_tools//tools/launcher:launcher.exe                 
@bazel_tools//src/tools/launcher:launcher                 
@bazel_tools//src/tools/launcher:python_launcher          
@bazel_tools//src/tools/launcher:python_launcher.h        
@bazel_tools//src/tools/launcher:python_launcher.cc       
@bazel_tools//src/tools/launcher:launcher_main.cc         
@bazel_tools//src/tools/launcher:java_launcher            
@bazel_tools//src/tools/launcher:java_launcher.h          
@bazel_tools//src/tools/launcher:java_launcher.cc         
@bazel_tools//src/tools/launcher:bash_launcher            
@bazel_tools//src/tools/launcher:launcher_base            
@bazel_tools//src/tools/launcher/util:data_parser         
@bazel_tools//src/tools/launcher/util:util                
@bazel_tools//src/tools/launcher/util:launcher_util.h     
@bazel_tools//src/tools/launcher/util:launcher_util.cc    
@bazel_tools//src/tools/launcher/util:dummy.cc            
@bazel_tools//src/tools/launcher/util:data_parser.h       
@bazel_tools//src/tools/launcher/util:data_parser.cc      
@bazel_tools//src/tools/launcher:launcher.h               
@bazel_tools//src/tools/launcher:launcher.cc              
@bazel_tools//src/main/cpp/util:filesystem                
@bazel_tools//src/main/native/windows:lib-file            
@bazel_tools//src/main/native/windows:util.h              
@bazel_tools//src/main/native/windows:util.cc             
@bazel_tools//src/main/native/windows:file.h              
@bazel_tools//src/main/native/windows:file.cc             
@bazel_tools//src/main/cpp/util:path_windows.cc           
@bazel_tools//src/main/cpp/util:path_posix.cc             
@bazel_tools//src/main/cpp/util:path_platform.h           
@bazel_tools//src/main/cpp/util:path.h                    
@bazel_tools//src/main/cpp/util:path.cc                   
@bazel_tools//src/main/cpp/util:file_windows.cc           
@bazel_tools//src/main/cpp/util:file_posix.cc             
@bazel_tools//src/main/cpp/util:file_platform.h           
@bazel_tools//src/main/cpp/util:file.h                    
@bazel_tools//src/main/cpp/util:file.cc                   
@bazel_tools//src/main/cpp/util:errors                    
@bazel_tools//src/main/cpp/util:port                      
@bazel_tools//src/main/cpp/util:port.h                    
@bazel_tools//src/main/cpp/util:port.cc                   
@bazel_tools//src/main/cpp/util:logging                   
@bazel_tools//src/main/cpp/util:strings                   
@bazel_tools//src/main/cpp/util:strings.h                 
@bazel_tools//src/main/cpp/util:strings.cc                
@bazel_tools//src/main/cpp/util:logging.h                 
@bazel_tools//src/main/cpp/util:logging.cc                
@bazel_tools//src/main/cpp/util:ijar                      
@bazel_tools//src/main/cpp/util:errors_windows.cc         
@bazel_tools//src/main/cpp/util:errors_posix.cc           
@bazel_tools//src/main/cpp/util:errors.h                  
@bazel_tools//src/main/cpp/util:blaze_exit_code           
@bazel_tools//tools/def_parser:def_parser                 
@bazel_tools//tools/def_parser:no_op.bat                  
@bazel_tools//tools/def_parser:def_parser_windows         
@bazel_tools//tools/def_parser:def_parser.exe             
@bazel_tools//third_party/def_parser:def_parser           
@bazel_tools//tools/cpp:malloc                            
@bazel_tools//third_party/def_parser:def_parser_main.cc   
@bazel_tools//third_party/def_parser:def_parser_lib       
@bazel_tools//third_party/def_parser:def_parser.h         
@bazel_tools//third_party/def_parser:def_parser.cc        
@bazel_tools//src/conditions:remote                       
@bazel_tools//src/main/cpp/util:exit_code.h               
@bazel_tools//src/tools/launcher:dummy.cc                 
@bazel_tools//src/tools/launcher:bash_launcher.h          
@bazel_tools//src/tools/launcher:bash_launcher.cc         
@bazel_tools//src/conditions:windows                      
@bazel_tools//src/conditions:host_windows                 
@bazel_tools//tools/cpp:toolchain                         
//external:cc_toolchain                                   
@local_config_cc//:toolchain                              
@local_config_cc//:local                                  
@local_config_cc//:local_config                           
@local_config_cc//:empty                                  
@bazel_tools//tools/cpp:link_dynamic_library              
@bazel_tools//tools/cpp:link_dynamic_library.sh           
@bazel_tools//tools/cpp:interface_library_builder         
@bazel_tools//tools/cpp:build_interface_so                
@bazel_tools//tools/cpp:grep-includes                     
@bazel_tools//tools/cpp:grep-includes.sh                  
//:dummy.py
brandjon commented 5 years ago

It's certainly possible we're still requiring a cpp toolchain through dependencies like the zipper, despite removing the direct implicit dependency from the py_binary rule. I don't think that's something we're likely to eliminate anytime soon. In fact, we may add more things that require cpp compilation, such as a native launcher to replace the stub script.

EdSchouten commented 4 years ago

Though I have absolutely no problems with py_binary() dragging in C++ toolchains to do all sorts of fancy things, isn't there anything that could be done to improve Bazel's analysis to defer the resolution of the toolchain to the actual point where it's being used? I've noticed that this error currently triggers, even in cases where no actual C++ code ends up being built. For example, when using rules_docker's py3_image().

I am currently working around this by declaring a bogus C++ toolchain using cc_toolchain()+cc_toolchain_config()+toolchain()+register_toolchains().

ulfjack commented 4 years ago

Not easily. The py_binary rule has to declare the dependency before it can know whether it's necessary. The built-in rule collects all transitive cc_library rules and then creates a link action (for which it needs a cc toolchain). Technically, that can only happen if the cc_library rules already have a cc toolchain dependency, but there's currently no infrastructure to inherit the toolchain in that case.

One possible option might be to add a tag or attribute to disable the dependency. That would require manual work, though.

Another option might be to find a way to forward the cc toolchain dep from the cc_library. I'm not sure if that's feasible though.

EdSchouten commented 4 years ago

Thanks for clarifying!

sgowroji commented 1 year ago

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

meastham commented 1 year ago

I believe this is still relevant. Would it be possible to use the new optional toolchain support to solve the problem?

UebelAndre commented 1 week ago

@rickeylev Should this issue be moved to rules_python? I once again ran into this and feel like the toolchain should at least be optional so if I want to cross compile a pure python zipapp (aka filegroup(name = "zip_file", output_groups = "python_zip_file")) then I'd be able to do so without setting up an entire windows C++ toolchain for windows so it can be unused.

rickeylev commented 1 week ago

I think rules_python dropped the requirement for a cc toolchain? See https://github.com/bazelbuild/rules_python/blob/main/python/private/common/py_executable.bzl#L72

It has mandatory=False in the toolchain_type() call.

UebelAndre commented 1 week ago

I think rules_python dropped the requirement for a cc toolchain? See https://github.com/bazelbuild/rules_python/blob/main/python/private/common/py_executable.bzl#L72

It has mandatory=False in the toolchain_type() call.

Ah, you're right, the issue I see for py_binary comes from https://github.com/bazelbuild/rules_python/blob/79df3c9077b12ccf7a8e73757c417c607a81f1d3/python/private/common/py_executable_bazel.bzl#L84-L93

I think these should also be configured to avoid the C++ toolchain requirement.

rickeylev commented 1 week ago

Can you expound a bit? It would help verify what's going on.

I seem to recall the launcher_maker is a prebuilt binary for windows, but for other platforms it points to a cc_binary target. If this is what's happening, then we can cut that edge for non-windows -- the launcher maker is only used on windows.

Zipper should be a prebuilt binary regardless, though? I don't recall that falling back to a non-prebuilt binary.

In any case, lets file a new issue in rules_python if the point of concern is in rules_python and not the builtin bazel rules. Most of this issue is historical in nature at this point.