[package] protobuf/3.17.1: Builds protoc installer for wrong architecture when cross compiling

MikeyH84 commented 3 years ago

Package and Environment Details (include every applicable attribute)

Requires package: protobuf/3.17.1

Conan profile (output of `conan profile show default` or `conan profile show <profile>` if custom profile is in use)

Configuration (profile_host):
[settings]
arch=armv8
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=5
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=5
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

Steps to reproduce (Include if Applicable)

Attempt to cross compile on build machine to host platform.

Logs (Include/Attach if Applicable)

[  1%] Running cpp protocol buffer compiler on response_data.proto. Custom options: 
Scanning dependencies of target pre-test-setup
Scanning dependencies of target contract
Scanning dependencies of target log
[  1%] Running cpp protocol buffer compiler on pre_process_data.proto. Custom options: 
[  2%] pre-test setup output directory structure
[  3%] Building CXX object _deps/shared-build/src/log/CMakeFiles/log.dir/log.cpp.o
Scanning dependencies of target network
[  4%] Building CXX object _deps/shared-build/src/contract/CMakeFiles/contract.dir/assert.cpp.o
/bin/sh: 1: /home/mike/.conan/data/protobuf/3.17.1/_/_/package/1ea143163c6ae43b7dcd78006f69dc4ed2cc53db/bin/protoc: Exec format error
src/serialise_msg/proto/CMakeFiles/proto.dir/build.make:89: recipe for target 'src/serialise_msg/proto/pre_process_data.pb.h' failed
make[2]: *** [src/serialise_msg/proto/pre_process_data.pb.h] Error 2
make[2]: *** Waiting for unfinished jobs....
/bin/sh: 1: /home/mike/.conan/data/protobuf/3.17.1/_/_/package/1ea143163c6ae43b7dcd78006f69dc4ed2cc53db/bin/protoc: Exec format error
src/serialise_msg/proto/CMakeFiles/proto.dir/build.make:81: recipe for target 'src/serialise_msg/proto/response_data.pb.h' failed
make[2]: *** [src/serialise_msg/proto/response_data.pb.h] Error 2
CMakeFiles/Makefile2:2736: recipe for target 'src/serialise_msg/proto/CMakeFiles/proto.dir/all' failed
make[1]: *** [src/serialise_msg/proto/CMakeFiles/proto.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  5%] Building CXX object src/network/CMakeFiles/network.dir/OffBoardMessage.cpp.o
[  6%] Building CXX object src/network/CMakeFiles/network.dir/AsyncClient.cpp.o
[  6%] Built target contract
[  6%] Built target pre-test-setup
[  7%] Linking CXX static library liblog.a
[  7%] Built target log
[  8%] Linking CXX static library libnetwork.a
[  8%] Built target network
Makefile:113: recipe for target 'all' failed
make: *** [all] Error 2

We are experiencing issues were the protoc compiler component gets build for the host architecture instead of the build architecture when cross compiling. This results in Conan failing to build the protobuf dependencies as the binaries are in the wrong format (fails when the protoc tool is executed).

Currently our recipes require dependencies:

protobuf/3.9.1@bincrafters/stable
protoc_installer/3.9.1@bincrafters/stable

These work no problem, but are deprecated (sunset). We attempted to move to the Conan centre, switching to protobuf/3.17.1, but now experiencing cross compilation issues.

The protoc compiler gets built for ARM architecture (our host platform) but the build system architecture is x86-64 Linux. Resulting in error:

/bin/sh: 1: /root/.conan/data/protobuf/3.17.1/_/_/package/1ea143163c6ae43b7dcd78006f69dc4ed2cc53db/bin/protoc: Exec format error

Not sure the reason why the libraries got combined back into a single Conan recipe. Maybe we are missing something? Some additional options perhaps?

But essentially we would like either additional options exposed in the recipe for cross compilation or the libraries to be split again and function in a similar manner to how they were at version 3.9.1.

Possible Hack

We hacked a local copy of the recipe to point to an already pre-existing x86-64 Linux build of the protobuf library (containing the protoc tool). However, we do not want to modify/maintain our own custom version of the recipe. This also adds unwanted dependencies in the CI/CD and local builds. eg. x86-64 Linux version must already exist in the systems local Conan cache with matching protocol versions.

We can host the deprecated libraries ourselves, but we would like future support.

Thank you for your time :)

ericriff commented 3 years ago

You can use protobuf as both, a requirement and as a build_requirment. You end up using the libraries from the requirement, crosscompiled for your target arch and protoc from the build_requirement package. That is how I do it.

MikeyH84 commented 3 years ago

@ericriff Thanks for the reply. Do you have an example? I am not sure how to correctly implement your idea. Not sure how to specify in my conanfile to use our x86-64 arch protoc.

zuowanbushiwo commented 3 years ago

@ericriff @MikeyH84 Did you solve it? Can you give an example? thanks！

MikeyH84 commented 2 years ago

@ericriff This situation is a Canadian Cross. So we ended up following this.

We created both ARM and x86-64 profiles. Our ARM profile also used a Yocto SDK toolchain so we included the necessary ENV VARS for that in the profile.

Then installed using the Conan experimental feature:

conan install \
  --build=missing \
  --profile:host ./conan-profile-host \
  --profile:build ./conan-profile-build \
  "${PROJECT_DIR}"

PeteAudinate commented 2 years ago

I'm seeing this same issue. Could someone provide some insight into how this is intended to work when cross-compiling with the CMakeDeps generator?

conanfile.py:

class MyConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"

    requires = "protobuf/3.19.1"
    tool_requires = "protobuf/3.19.1"

CMakeLists.txt:

cmake_minimum_required(VERSION 3.18)
project (MyProject)

find_package(protobuf REQUIRED)

add_library(my_target my_proto.proto)
target_link_libraries(my_target protobuf::protobuf)
protobuf_generate_cpp(PROTO_SRCS PROTO_HDRS TARGET my_target)
protobuf_generate(LANGUAGE cpp TARGET my_target PROTOS my_proto.proto)

If I build this, CMake tries to execute the host platform's protoc binary when building:

$ conan install .. --profile linux-armv8 --profile:build linux-x64
$ cmake .. -G Ninja -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake
$ cmake --build .
...
/bin/sh: 1: /home/username/.conan/data/protobuf/3.19.1/_/_/package/e59c9b7d983a28031f7cda229595ac22e2116e8b/bin/protoc: Exec format error

PeteAudinate commented 2 years ago

If I add the following, it finds the correct protoc binary in the build context, and everything is fine: conanfile.py:

  def generate(self):
        cmake = CMakeDeps(self)
        cmake.build_context_activated = ["protobuf"]
        cmake.build_context_suffix = {"protobuf": "_BUILD"}
        cmake.generate()

        toolchain = CMakeToolchain(self)
        toolchain.generate()

CMakeLists.txt:

...
find_package(protobuf_BUILD REQUIRED)
find_package(protobuf REQUIRED)
...

Is this the best way to do this?

The find_package(protobuf_BUILD) line must come before find_package(protobuf), or it won't work.

The CMake configure step emits these warnings:

 CMake Warning at build/protobuf_BUILDTargets.cmake:15 (message):
  Component target name 'protobuf::libprotobuf' already exists.
Call Stack (most recent call first):
  build/protobuf_BUILDConfig.cmake:11 (include)
  CMakeLists.txt:9 (find_package)

CMake Warning at build/protobuf_BUILDTargets.cmake:15 (message):
  Component target name 'protobuf::libprotoc' already exists.
Call Stack (most recent call first):
  build/protobuf_BUILDConfig.cmake:11 (include)
  CMakeLists.txt:9 (find_package)

Although this works, it feels a lot more awkward than it should be. Am I missing something?

eirikb commented 2 years ago

Hi. Just wanted to mention I've successfully cross compiled protobuf with Yocto SDK using two profiles. I have a docker image including a minimal Yocto SDK for armv7hf. How the docker image and SDK was made can be seen in this gist.

A proof of building can be seen here. This job runs

conan create . protobuf/3.19.2@ -pr:b=buildpr -pr:h=hostpr --build=outdated

with docker image ghcr.io/eirikb/yocto-sdk-armv7hf:3.1.14 against CCI of protobuf. This proves it works because test_package will generate code from a .proto file.

Running

docker run --rm -it ghcr.io/eirikb/yocto-sdk-armv7hf:3.1.14 conan install protobuf/3.19.2@ -pr:b=buildpr -pr:h=hostpr --build=outdated

directly is also possible, but it won't run the test_package.

The docker image by itself is not enough as it is, the reason it works now is because the test_package will add protobuf to [build_requires] as seen in the code. This must be either done by the consumer, or it could be set in the hostpr. I tried setting it in the profile and that worked fine.

I'm not sure if the last message from @MikeyH84 is the same as my gist, but it might be very similar. I had to do some experimentation before it worked out of the box. I also discussed the approach in https://github.com/conan-io/conan/issues/7431 . I believe setting [build_requires], as in test_package, is exactly what @ericriff talked about. For me this was not enough, I also had to make some changes to my image for it to work with Yocto.

rob-baily commented 7 months ago

@ericriff @MikeyH84 Did you solve it? Can you give an example? thanks！

I have it resolved using his comment for a build that supports cross compile to an RPi 4.

Create the profile for your host system (aarch64 in our case) with the appropriate buildenv settings for your cross compiler.

In the conanfile.py add protobuf as both a requirement and build requirement as shown below.

requires = "protobuf/3.21.12"
...
def build_requirements(self):
  self.tool_requires("protobuf/3.21.12")

When doing conan install specify your host and build profiles like this: conan install . --output-folder $BUILD_FOLDER --build=missing -pr:h=${HOST_PROFILE} During this step both versions of protobuf are compiled.
When doing the build specify the host profile as well: conan build . --output-folder $BUILD_FOLDER -pr:h=${HOST_PROFILE}

The way to confirm this is to look for PROTOBUF_PROTOC_EXECUTABLE in the CMakeCache.txt and verify it points to the build version. NOTE: If you change the conanfile you will need to remove your build folder for it to work correctly.

MikeyH84 commented 7 months ago

@rob-baily

I ressolved the cross-compile issue by manually pointing to the correct Protoc binary in the build context like so:

    requires = (
    ...,
        "protobuf/3.21.1",
    )

   def build_requirements(self):
        self.tool_requires("protobuf/3.21.1")

   def build(self):      
    cmake = CMake(self)     
    cmake.definitions["Protobuf_PROTOC_EXECUTABLE"] = os.path.join(self.deps_env_info["protobuf"].PATH[0], "protoc")
    ...

I didn't do anything special with the install/build/create commands. I did use the newer Conan build/host profiles for the build contexts. An example would look like the following:

conan install \
  --build=missing \
  --profile:build="${BUILD_PROFILE}"  \
  --profile:host="${HOST_PROFILE}"  \
  "${CONAN_RECIPE}"

Note handling Protobuf_PROTOC_EXECUTABLE in the conanfile.py means we didn't need to do anything to manage Protobuf in CMake.

rob-baily commented 7 months ago

I didn't do anything special with the install/build/create commands. I did use the newer Conan build/host profiles for the build contexts.

Note handling Protobuf_PROTOC_EXECUTABLE in the conanfile.py means we didn't need to do anything to manage Protobuf in CMake.

I did not need to do anything extra in the CMake files with my configuration above. I use find_package(Protobuf REQUIRED) and protobuf_generate_cpp so perhaps your setting for the executable is extra now. I am using conan 2.0.17.

MikeyH84 commented 7 months ago

We still need that, but we using Conan 1.59.0. A lot of the build/host context issues got sorted with the 2.0 migration

wuziq commented 1 month ago

You can use protobuf as both, a requirement and as a build_requirment. You end up using the libraries from the requirement, crosscompiled for your target arch and protoc from the build_requirement package. That is how I do it.

Just confirming that this works. No need to set Protobuf_PROTOC_EXECUTABLE, either, just did the usual find_package(Protobuf REQUIRED).

conan-io / conan-center-index