Open cimes-isi opened 11 months ago
CC @Mystery-Golden-Retriever (problem originated on his laptop)
Thanks for the report!
Could you set the environment variable PY_BUILD_CMAKE_VERBOSE=1
and post the full output again, to see the environment and command line options used for all subprocesses?
Determining the architecture using CMake (regardless of what py-build-cmake does) is a bit tricky on Apple Silicon: https://cmake.org/cmake/help/latest/variable/CMAKE_HOST_SYSTEM_PROCESSOR.html#macos-platforms
Note that py-build-cmake does not enable cross-compilation on its own, except for the following:
CMAKE_OSX_ARCHITECTURES
variable based on the value of the ARCHFLAGS
environment variable on macOS, andCMAKE_GENERATOR_PLATFORM
variable and some others on Windows when the platform specified by the config file pointed to by the DIST_EXTRA_CONFIG
variable is different from the platform of the running interpreter.This is done in order to support cibuildwheel, which uses these environment variables to drive cross-compilation.
The difference between Windows and macOS is that for Windows, py-build-cmake sets CMAKE_SYSTEM_NAME
, causing CMake to enter cross-compilation mode. On macOS, only CMAKE_OSX_ARCHITECTURES
is set. This may cause the compiler to cross-compile, but does not enable the cross-compilation mode of CMake.
Just thinking out loud: are you able to build your project stand-alone (without py-build-cmake, just invoking CMake directly) when using an x86_64 version of CMake on an M2 machine?
Personally, I would treat building for x86 on an M2 as a cross-compilation situation. In other words, it's best to take control of the cross-compilation entirely, and write an appropriate CMake toolchain file, as well as a py-build-cmake cross-configuration as explained in https://tttapa.github.io/py-build-cmake/Cross-compilation.html. With the right CMake toolchain file, you can prevent CMake from picking up packages for the build system that were not intended for the target system.
For the further improvement of py-build-cmake, we would have to determine when to switch CMake into cross-compilation mode on macOS:
ARCHFLAGS
is set to any value?Could you set the environment variable
PY_BUILD_CMAKE_VERBOSE=1
and post the full output again, to see the environment and command line options used for all subprocesses?
I have an M1 mac, but don't have an Anaconda installation. I have a miniforge conda installed and would rather not try overloading it with an Anaconda unless really needed. I can try the following though.
Just thinking out loud: are you able to build your project stand-alone (without py-build-cmake, just invoking CMake directly) when using an x86_64 version of CMake on an M2 machine?
It doesn't look like CMake releases an x86_64-only build, just a universal one, so I'll try that. When forcing x64_64
, the result is essentially the same:
$ file ~/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake
/Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
/Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake (for architecture x86_64): Mach-O 64-bit executable x86_64
/Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake (for architecture arm64): Mach-O 64-bit executable arm64
$ arch -x86_64 ~/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake ..
CMake Warning (dev) at /Users/cimes/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/share/cmake-3.27/Modules/GNUInstallDirs.cmake:243 (message):
Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
target architecture is known. Please enable at least one language before
including GNUInstallDirs.
Call Stack (most recent call first):
CMakeLists.txt:3 (include)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Checking for py-build-cmake environment - not found
-- Using default install paths
-- The CXX compiler identification is AppleClang 14.0.3.14030022
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/cimes/local/fogsys/PipeEdge-Test/src-native/build
$ arch -x86_64 ~/Downloads/cmake-3.27.1-macos-universal/CMake.app/Contents/bin/cmake --build .
[ 33%] Building CXX object CMakeFiles/sched-pipeline.dir/sched-pipeline.cpp.o
[ 66%] Building CXX object CMakeFiles/sched-pipeline.dir/schedule.cpp.o
/Users/cimes/local/fogsys/PipeEdge-Test/src-native/schedule.cpp:200:10: warning: variable 'stage_num' set but not used [-Wunused-but-set-variable]
size_t stage_num = 0;
^
1 warning generated.
[100%] Linking CXX executable sched-pipeline
ld: warning: ignoring file /opt/homebrew/lib/libyaml-cpp.0.7.0.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
"YAML::InvalidNode::~InvalidNode()", referenced from:
yaml_model YAML::Node::as<yaml_model>() const in sched-pipeline.cpp.o
YAML::Node::EnsureNodeExists() const in sched-pipeline.cpp.o
YAML::Node::Scalar() const in sched-pipeline.cpp.o
YAML::Node::Type() const in sched-pipeline.cpp.o
YAML::Node::Mark() const in sched-pipeline.cpp.o
unsigned long YAML::Node::as<unsigned long>() const in sched-pipeline.cpp.o
std::__1::vector<unsigned long, std::__1::allocator<unsigned long>> YAML::Node::as<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>>() const in sched-pipeline.cpp.o
...
"YAML::BadSubscript::~BadSubscript()", referenced from:
YAML::detail::node* YAML::detail::node_data::get<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [7]>(char const (&) [7], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [14]>(char const (&) [14], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [15]>(char const (&) [15], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [8]>(char const (&) [8], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [6]>(char const (&) [6], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [11]>(char const (&) [11], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
...
"YAML::BadConversion::~BadConversion()", referenced from:
YAML::TypedBadConversion<yaml_model>::~TypedBadConversion() in sched-pipeline.cpp.o
YAML::TypedBadConversion<unsigned long>::~TypedBadConversion() in sched-pipeline.cpp.o
YAML::TypedBadConversion<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>>::~TypedBadConversion() in sched-pipeline.cpp.o
YAML::TypedBadConversion<std::__1::vector<double, std::__1::allocator<double>>>::~TypedBadConversion() in sched-pipeline.cpp.o
YAML::TypedBadConversion<double>::~TypedBadConversion() in sched-pipeline.cpp.o
YAML::TypedBadConversion<std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, yaml_device_type, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, yaml_device_type>>>>::~TypedBadConversion() in sched-pipeline.cpp.o
YAML::TypedBadConversion<yaml_device_type>::~TypedBadConversion() in sched-pipeline.cpp.o
...
"YAML::ostream_wrapper::write(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
YAML::operator<<(YAML::ostream_wrapper&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
"YAML::detail::memory::create_node()", referenced from:
YAML::detail::memory_holder::create_node() in sched-pipeline.cpp.o
"YAML::detail::node_data::empty_scalar()", referenced from:
YAML::Node::Scalar() const in sched-pipeline.cpp.o
"YAML::detail::node_data::mark_defined()", referenced from:
YAML::detail::node_ref::mark_defined() in sched-pipeline.cpp.o
"YAML::detail::node_data::end()", referenced from:
YAML::detail::node_ref::end() in sched-pipeline.cpp.o
"YAML::detail::node_data::begin()", referenced from:
YAML::detail::node_ref::begin() in sched-pipeline.cpp.o
"YAML::detail::node_data::set_null()", referenced from:
YAML::detail::node_ref::set_null() in sched-pipeline.cpp.o
"YAML::Emitter::PrepareNode(YAML::EmitterNodeType::value)", referenced from:
YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
"YAML::Emitter::SetLocalValue(YAML::EMITTER_MANIP)", referenced from:
YAML::operator<<(YAML::Emitter&, YAML::EMITTER_MANIP) in sched-pipeline.cpp.o
"YAML::Emitter::StartedScalar()", referenced from:
YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
"YAML::Emitter::Write(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
YAML::operator<<(YAML::Emitter&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
"YAML::Emitter::Emitter()", referenced from:
print_schedule(std::__1::vector<host_sched_stage, std::__1::allocator<host_sched_stage>> const&) in sched-pipeline.cpp.o
"YAML::Emitter::~Emitter()", referenced from:
print_schedule(std::__1::vector<host_sched_stage, std::__1::allocator<host_sched_stage>> const&) in sched-pipeline.cpp.o
"YAML::LoadFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)", referenced from:
load_files(schedule_ctx&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) in sched-pipeline.cpp.o
"YAML::Emitter::PrepareIntegralStream(std::__1::basic_stringstream<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) const", referenced from:
YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
"YAML::Emitter::good() const", referenced from:
YAML::Emitter& YAML::Emitter::WriteIntegralType<unsigned long>(unsigned long) in sched-pipeline.cpp.o
"YAML::Emitter::c_str() const", referenced from:
print_schedule(std::__1::vector<host_sched_stage, std::__1::allocator<host_sched_stage>> const&) in sched-pipeline.cpp.o
"typeinfo for YAML::InvalidNode", referenced from:
yaml_model YAML::Node::as<yaml_model>() const in sched-pipeline.cpp.o
YAML::Node::EnsureNodeExists() const in sched-pipeline.cpp.o
YAML::Node::Scalar() const in sched-pipeline.cpp.o
YAML::Node::Type() const in sched-pipeline.cpp.o
YAML::Node::Mark() const in sched-pipeline.cpp.o
unsigned long YAML::Node::as<unsigned long>() const in sched-pipeline.cpp.o
std::__1::vector<unsigned long, std::__1::allocator<unsigned long>> YAML::Node::as<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>>() const in sched-pipeline.cpp.o
...
"typeinfo for YAML::BadSubscript", referenced from:
YAML::detail::node* YAML::detail::node_data::get<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [7]>(char const (&) [7], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [14]>(char const (&) [14], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [15]>(char const (&) [15], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [8]>(char const (&) [8], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [6]>(char const (&) [6], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
YAML::detail::node* YAML::detail::node_data::get<char [11]>(char const (&) [11], std::__1::shared_ptr<YAML::detail::memory_holder>) const in sched-pipeline.cpp.o
...
"typeinfo for YAML::BadConversion", referenced from:
typeinfo for YAML::TypedBadConversion<yaml_model> in sched-pipeline.cpp.o
typeinfo for YAML::TypedBadConversion<unsigned long> in sched-pipeline.cpp.o
typeinfo for YAML::TypedBadConversion<std::__1::vector<unsigned long, std::__1::allocator<unsigned long>>> in sched-pipeline.cpp.o
typeinfo for YAML::TypedBadConversion<std::__1::vector<double, std::__1::allocator<double>>> in sched-pipeline.cpp.o
typeinfo for YAML::TypedBadConversion<double> in sched-pipeline.cpp.o
typeinfo for YAML::TypedBadConversion<std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, yaml_device_type, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, yaml_device_type>>>> in sched-pipeline.cpp.o
typeinfo for YAML::TypedBadConversion<yaml_device_type> in sched-pipeline.cpp.o
...
"vtable for YAML::InvalidNode", referenced from:
YAML::InvalidNode::InvalidNode(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for YAML::BadSubscript", referenced from:
YAML::BadSubscript::BadSubscript<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
YAML::BadSubscript::BadSubscript<char [7]>(YAML::Mark const&, char const (&) [7]) in sched-pipeline.cpp.o
YAML::BadSubscript::BadSubscript<char [14]>(YAML::Mark const&, char const (&) [14]) in sched-pipeline.cpp.o
YAML::BadSubscript::BadSubscript<char [15]>(YAML::Mark const&, char const (&) [15]) in sched-pipeline.cpp.o
YAML::BadSubscript::BadSubscript<char [8]>(YAML::Mark const&, char const (&) [8]) in sched-pipeline.cpp.o
YAML::BadSubscript::BadSubscript<char [6]>(YAML::Mark const&, char const (&) [6]) in sched-pipeline.cpp.o
YAML::BadSubscript::BadSubscript<char [11]>(YAML::Mark const&, char const (&) [11]) in sched-pipeline.cpp.o
...
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for YAML::BadConversion", referenced from:
YAML::BadConversion::BadConversion(YAML::Mark const&) in sched-pipeline.cpp.o
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for YAML::RepresentationException", referenced from:
YAML::RepresentationException::RepresentationException(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
"vtable for YAML::Exception", referenced from:
YAML::Exception::Exception(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [sched-pipeline] Error 1
make[1]: *** [CMakeFiles/sched-pipeline.dir/all] Error 2
make: *** [all] Error 2
We did previously check the M2 environment and verified that no ARCHFLAGS
for CMAKE_*
environment variables were set once the conda environment was activated but before running pip
to do the build.
I can test manually setting different environment variables for cmake (again forcing x64_64) to imitate what you think py-build-cmake would/should do, if you'd like.
When forcing
x64_64
, the result is essentially the same:
Thanks, that's what I expected. I'm afraid that the issue you've encountered could be a fundamental limitation of running CMake under x86_64 emulation on an M2 mac.
The problem here is that CMake does not know that the system libraries (e.g. the ones in /opt/homebrew/lib
) are for a different architecture than itself. When CMake is not in cross-compilation mode, it expects that it can use the system libraries without issues, but this assumption breaks under emulation.
There are (at least) two ways to handle this:
/usr/lib/aarch64-linux-gnu
and /usr/lib/x86_64-linux-gnu
instead of just /usr/lib
, and CMake can then select the right version.In any case, you need to provide a version of the library for the architecture you selected (an x86_64 version of yaml-cpp, in your case). And you have to make sure that CMake is able to locate that library.
I'm not familiar with Homebrew, but from what I can gather, it does not support installing the same package for different architectures: https://github.com/orgs/Homebrew/discussions/2141
So you'll have to install yaml-cpp either from source, or using a different package manager that makes a distinction between architectures. Then you should cross-compile your project, and add the path to the correct yaml-cpp version to the CMAKE_FIND_ROOT_PATH
variable in your toolchain file (list(APPEND CMAKE_FIND_ROOT_PATH "/your/x86_64/staging/directory")
).
See https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html and https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html for details about toolchain files and cross-compilation.
I don't own a macOS system myself, so you might get better suggestions over on the CMake forum or the CMake issue tracker (since the problem persists even without py-build-cmake).
Thanks, I expected that, too, after seeing the initial failure through Anaconda. And again we don't actually need a solution to this using py-build-cmake. We're not going to do anything on our end to make this work. I just thought you should be aware of the behavior and I'm happy to help debug it as long as it doesn't turn my system upside down :). In case it wasn't clear, the universal cmake binary works fine from the CLI when we don't force x86_64
mode.
It's also probably the correct behavior for cmake to compile for x86_64
in this case, esp. if any of the code compiled is supposed to link with code running under the same x86_64
Python (Anaconda). It just shouldn't be trying to link against arm64
libraries.
Thanks, I appreciate it! :)
Hi Pieter. We ran into an interesting corner case running on an Apple M2 CPU running Anaconda installed for
x86_64
(which I suppose macOS then emulates for automatically). The most correct solution to the problem is simply to install Anaconda forarm64
, then everything works as expected. I thought we should report the experience anyway, in case you think there is something that can/should be done in py-build-cmake for it, though I'm not requesting that you do so.In short, py-build-cmake appears to have cmake compile for
x86-64
but cmake still tries to link against a dependent library (libyaml-cpp) that is built forarm64
. My layman's guess is that py-build-cmake is picking up thex86_64
arch from the conda execution environment (e.g., via distlib?), but the cross-compilation configuration is somehow incomplete.I'm not entirely sure what the correct thing to do should be, but I don't think it should get into a situation where it's trying to link libraries of the wrong architecture. Perhaps at least one of:
arm64
library path should not be searched.arm64
library itself should not "found" by cmake even if the path is searched.arm64
rather thanx86-64
.Cheers.