conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.09k stars 959 forks source link

[question] Unable to build crc32c: c1xx: fatal error C1083: Cannot open source file: '': No such file or directory #14598

Open cocoza4 opened 1 year ago

cocoza4 commented 1 year ago

What is your question?

Hi,

I'm unable to cross-compile https://github.com/google/crc32c using MSVC. However, normal build without Conan works

Steps

  1. Create conanfile.py from below
  2. run conan source .
  3. run conan install . -pr:h /workspace/conan/profiles/msvc-x64 -pr:b default
  4. conan build .

Output

-- Using Conan toolchain: /root/crc32c/build/Release/generators/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 17 with extensions OFF
-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Searching 16 bit integer - Using unsigned short
-- Check if the system is big endian - little endian
-- Performing Test CRC32C_HAVE_NO_DEPRECATED
-- Performing Test CRC32C_HAVE_NO_DEPRECATED - Failed
-- Performing Test CRC32C_HAVE_NO_SIGN_COMPARE
-- Performing Test CRC32C_HAVE_NO_SIGN_COMPARE - Failed
-- Performing Test CRC32C_HAVE_NO_UNUSED_PARAMETER
-- Performing Test CRC32C_HAVE_NO_UNUSED_PARAMETER - Failed
-- Performing Test CRC32C_HAVE_NO_MISSING_FIELD_INITIALIZERS
-- Performing Test CRC32C_HAVE_NO_MISSING_FIELD_INITIALIZERS - Failed
-- Performing Test HAVE_BUILTIN_PREFETCH
-- Performing Test HAVE_BUILTIN_PREFETCH - Failed
-- Performing Test HAVE_MM_PREFETCH
-- Performing Test HAVE_MM_PREFETCH - Failed
-- Performing Test HAVE_SSE42
-- Performing Test HAVE_SSE42 - Failed
-- Performing Test HAVE_ARM64_CRC32C
-- Performing Test HAVE_ARM64_CRC32C - Failed
-- Performing Test HAVE_STRONG_GETAUXVAL
-- Performing Test HAVE_STRONG_GETAUXVAL - Failed
-- Performing Test HAVE_WEAK_GETAUXVAL
-- Performing Test HAVE_WEAK_GETAUXVAL - Failed
-- Configuring done
-- Generating done
-- Build files have been written to: /root/crc32c/build/Release
conanfile.py (crc32c/1.1.2): CMake command: cmake --build "/root/crc32c/build/Release" '--' '-j12'
[ 20%] Building CXX object CMakeFiles/crc32c_arm64.dir/src/crc32c_arm64.cc.obj
[ 40%] Building CXX object CMakeFiles/crc32c_sse42.dir/src/crc32c_sse42.cc.obj

c1xx: fatal error C1083: Cannot open source file: '': No such file or directory
crc32c_sse42.cc
Generating Code...
make[2]: *** [CMakeFiles/crc32c_sse42.dir/build.make:76: CMakeFiles/crc32c_sse42.dir/src/crc32c_sse42.cc.obj] Error 2
make[2]: *** Deleting file 'CMakeFiles/crc32c_sse42.dir/src/crc32c_sse42.cc.obj'
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/crc32c_sse42.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

c1xx: fatal error C1083: Cannot open source file: '': No such file or directory
crc32c_arm64.cc
Generating Code...
make[2]: *** [CMakeFiles/crc32c_arm64.dir/build.make:76: CMakeFiles/crc32c_arm64.dir/src/crc32c_arm64.cc.obj] Error 2
make[2]: *** Deleting file 'CMakeFiles/crc32c_arm64.dir/src/crc32c_arm64.cc.obj'
make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/crc32c_arm64.dir/all] Error 2
make: *** [Makefile:124: all] Error 2
ERROR: conanfile.py (crc32c/1.1.2): Error in build() method, line 36
    cmake.build()
    ConanException: Error 2 while executing cmake --build "/root/crc32c/build/Release" '--' '-j12'

The problem comes from HAVE_MM_PREFETCH - Failed and HAVE_SSE42 - Failed. When building without Conan, they were both Success.

Conan file

from conan import ConanFile
from conan.tools.cmake import CMake, CMakeToolchain, cmake_layout
from conan.tools.scm import Git

class crc32cConan(ConanFile):
    name = "crc32c"
    description = "CRC32C implementation with support for CPU-specific acceleration instructions"
    topics = ("crc32c", "crc")
    homepage = "https://github.com/google/crc32c"
    license = "BSD-3-Clause"

    version = "1.1.2"

    package_type = "library"
    settings = "os", "arch", "compiler", "build_type"

    def layout(self):
        cmake_layout(self)

    def source(self):
        git = Git(self)
        git.clone(url="https://github.com/google/crc32c", args=[f"--branch {self.version}", "--depth 1"])

    def generate(self):
        tc = CMakeToolchain(self)
        tc.variables["CRC32C_BUILD_TESTS"] = False
        tc.variables["CRC32C_BUILD_BENCHMARKS"] = False
        tc.variables["CRC32C_INSTALL"] = True
        tc.variables["CRC32C_USE_GLOG"] = False
        tc.generate()

    def build(self):
        cmake = CMake(self)
        cmake.configure(build_script_folder="crc32c")
        cmake.build()

    def package(self):
        cmake = CMake(self)
        cmake.install()

    def package_info(self):
        self.cpp_info.set_property("cmake_file_name", "Crc32c")
        self.cpp_info.set_property("cmake_target_name", "Crc32c::crc32c")

msvc-x64

[settings]
os=Windows
arch=x86_64
# Clang is only used as a placeholder, since we are hacking for cross-compiling
compiler=clang
compiler.libcxx=libstdc++11
compiler.cppstd=17
compiler.version=10
build_type=Release

[options]

[tool_requires]
toolchain_msvc-x64/0.1.0

[conf]
tools.env.virtualenv:auto_use=True

[buildenv]
CXXFLAGS="/EHsc"

toolchain toolchain_msvc-x64

set(CMAKE_SYSTEM_NAME Windows)
set(CMAKE_SYSTEM_VERSION 10.0)
set(CMAKE_SYSTEM_PROCESSOR AMD64)

set(MSVC_BASE "/opt/msvc/vc/tools/msvc")
set(MSVC_VER "14.29.30133")
set(WINSDK_BASE "/opt/msvc/kits/10")
set(WINSDK_VER "10.0.19041.0")
set(WINSDK_ARCH "x64")

set(MSVC_INCLUDE "${MSVC_BASE}/${MSVC_VER}/include")
set(MSVC_LIB "${MSVC_BASE}/${MSVC_VER}/lib")
set(WINSDK_INCLUDE "${WINSDK_BASE}/include/${WINSDK_VER}")
set(WINSDK_LIB "${WINSDK_BASE}/lib/${WINSDK_VER}")

if(NOT EXISTS "${MSVC_BASE}" OR
   NOT EXISTS "${MSVC_INCLUDE}" OR
   NOT EXISTS "${MSVC_LIB}")
  message(SEND_ERROR
          "CMake variable MSVC_BASE must point to a folder containing MSVC "
          "system headers and libraries")
endif()

if(NOT EXISTS "${WINSDK_BASE}" OR
   NOT EXISTS "${WINSDK_INCLUDE}" OR
   NOT EXISTS "${WINSDK_LIB}")
  message(SEND_ERROR
          "CMake variable WINSDK_BASE and WINSDK_VER must resolve to a valid "
          "Windows SDK installation")
endif()

if(NOT EXISTS "${WINSDK_INCLUDE}/um/windows.h")
  message(SEND_ERROR "Cannot find Windows.h")
endif()

set(CMAKE_C_COMPILER "/opt/msvc/bin/${WINSDK_ARCH}/cl" CACHE FILEPATH "")
set(CMAKE_CXX_COMPILER "/opt/msvc/bin/${WINSDK_ARCH}/cl" CACHE FILEPATH "")
set(CMAKE_LINKER "/opt/msvc/bin/${WINSDK_ARCH}/link" CACHE FILEPATH "")
set(CMAKE_RC_COMPILER "/opt/msvc/bin/${WINSDK_ARCH}/rc" CACHE FILEPATH "")

set(CMAKE_C_COMPILER_FORCED TRUE) # forces CMake to skip compiler checks
set(CMAKE_CXX_COMPILER_FORCED TRUE)
set(CMAKE_SIZEOF_VOID_P 8) # sets CMake bitness to 64bit

set(COMPILE_FLAGS
    /I "${MSVC_INCLUDE}"
    /I "${WINSDK_INCLUDE}/ucrt"
    /I "${WINSDK_INCLUDE}/shared"
    /I "${WINSDK_INCLUDE}/um"
    /I "${WINSDK_INCLUDE}/winrt")

string(REPLACE ";" " " COMPILE_FLAGS "${COMPILE_FLAGS}")

# We need to preserve any flags that were passed in by the user. However, we
# can't append to CMAKE_C_FLAGS and friends directly, because toolchain files
# will be re-invoked on each reconfigure and therefore need to be idempotent.
# The assignments to the _INITIAL cache variables don't use FORCE, so they'll
# only be populated on the initial configure, and their values won't change
# afterward.
set(_CMAKE_C_FLAGS_INITIAL "${CMAKE_C_FLAGS}" CACHE STRING "")
set(CMAKE_C_FLAGS "${_CMAKE_C_FLAGS_INITIAL} ${COMPILE_FLAGS} $ENV{CFLAGS}" CACHE STRING "" FORCE)
set(_CMAKE_CXX_FLAGS_INITIAL "${CMAKE_CXX_FLAGS}" CACHE STRING "")
set(CMAKE_CXX_FLAGS "${_CMAKE_CXX_FLAGS_INITIAL} ${COMPILE_FLAGS} $ENV{CXXFLAGS}" CACHE STRING "" FORCE)

set(LINK_FLAGS
    # Prevent CMake from attempting to invoke mt.exe. It only recognizes the slashed form and not the dashed form.
    /manifest:no

    /LIBPATH:"${MSVC_LIB}/${WINSDK_ARCH}"
    /LIBPATH:"${WINSDK_LIB}/ucrt/${WINSDK_ARCH}"
    /LIBPATH:"${WINSDK_LIB}/um/${WINSDK_ARCH}"
    /DEFAULTLIB:user32)

string(REPLACE ";" " " LINK_FLAGS "${LINK_FLAGS}")

# See explanation for compiler flags above for the _INITIAL variables.
set(_CMAKE_EXE_LINKER_FLAGS_INITIAL "${CMAKE_EXE_LINKER_FLAGS}" CACHE STRING "")
set(CMAKE_EXE_LINKER_FLAGS "${_CMAKE_EXE_LINKER_FLAGS_INITIAL} ${LINK_FLAGS}" CACHE STRING "" FORCE)

set(_CMAKE_MODULE_LINKER_FLAGS_INITIAL "${CMAKE_MODULE_LINKER_FLAGS}" CACHE STRING "")
set(CMAKE_MODULE_LINKER_FLAGS "${_CMAKE_MODULE_LINKER_FLAGS_INITIAL} ${LINK_FLAGS}" CACHE STRING "" FORCE)

set(_CMAKE_SHARED_LINKER_FLAGS_INITIAL "${CMAKE_SHARED_LINKER_FLAGS}" CACHE STRING "")
set(CMAKE_SHARED_LINKER_FLAGS "${_CMAKE_SHARED_LINKER_FLAGS_INITIAL} ${LINK_FLAGS}" CACHE STRING "" FORCE)

Note that this toolchain works when building other libraries.

Conan 1.5.8

Have you read the CONTRIBUTING guide?

memsharded commented 1 year ago

Hi @cocoza4

Thanks for your question.

Some quick feedback:

cocoza4 commented 1 year ago

@memsharded thanks for your response, I updated conan to 1.60.2 and it still didn't work.

memsharded commented 1 year ago

Thanks for sharing it!

The conan build command in Conan 1.X might have some limitations. I guess that you have tried with conan create . instead the full install + source + build flow, and it fails in the same way? If you haven't, could you please try that?

Maybe sharing the full output, not only the build part, could have some extra hints. Thanks!

cocoza4 commented 1 year ago
root@peeranat:/workspace/3rdparty/crc32c# conan create . -pr:h /workspace/conan/profiles/msvc-x64 -pr:b default
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'url'. It is recommended to add it as attribute
Exporting package recipe
crc32c/1.1.2: The stored package has not changed
crc32c/1.1.2: Exported revision: 9a7e165e7628a58a72fcd16e6f5749db
Configuration (profile_host):
[settings]
arch=x86_64
build_type=Release
compiler=clang
compiler.cppstd=17
compiler.libcxx=libstdc++11
compiler.version=10
os=Windows
[options]
[build_requires]
*: toolchain_msvc-x64/0.1.0
[env]
[conf]
tools.env.virtualenv:auto_use=True
[buildenv]
CXXFLAGS="/EHsc"

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

crc32c/1.1.2: Forced build from source
Installing package: crc32c/1.1.2
Requirements
    crc32c/1.1.2 from 'artifactory' - Cache
Packages
    crc32c/1.1.2:0246896ed8e77a58a3985dffb2e82bf25f71ed9e - Build
Build requirements
    toolchain_msvc-x64/0.1.0 from 'artifactory' - Cache
Build requirements packages
    toolchain_msvc-x64/0.1.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Cache

Cross-build from 'Linux:x86_64' to 'Windows:x86_64'
Installing (downloading, building) binaries...
toolchain_msvc-x64/0.1.0: Already installed!
crc32c/1.1.2: Applying build-requirement: toolchain_msvc-x64/0.1.0
crc32c/1.1.2: WARN: Build folder is dirty, removing it: /root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e
crc32c/1.1.2: Copying sources to build folder
crc32c/1.1.2: Building your package in /root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e
crc32c/1.1.2: Generator txt created conanbuildinfo.txt
crc32c/1.1.2: Calling generate()
crc32c/1.1.2: Preset 'release' added to CMakePresets.json. Invoke it manually using 'cmake --preset release'
crc32c/1.1.2: If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release/generators/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
crc32c/1.1.2: Aggregating env generators
crc32c/1.1.2: Calling build()
crc32c/1.1.2: CMake command: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/.conan/data/crc32c/1.1.2/_/_/package/0246896ed8e77a58a3985dffb2e82bf25f71ed9e" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/./crc32c"
-- Using Conan toolchain: /root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release/generators/conan_toolchain.cmake
-- Conan toolchain: C++ Standard 17 with extensions OFF
-- The C compiler identification is MSVC 19.29.30133.0
-- The CXX compiler identification is MSVC 19.29.30133.0
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Searching 16 bit integer - Using unsigned short
-- Check if the system is big endian - little endian
-- Performing Test CRC32C_HAVE_NO_DEPRECATED
-- Performing Test CRC32C_HAVE_NO_DEPRECATED - Failed
-- Performing Test CRC32C_HAVE_NO_SIGN_COMPARE
-- Performing Test CRC32C_HAVE_NO_SIGN_COMPARE - Failed
-- Performing Test CRC32C_HAVE_NO_UNUSED_PARAMETER
-- Performing Test CRC32C_HAVE_NO_UNUSED_PARAMETER - Failed
-- Performing Test CRC32C_HAVE_NO_MISSING_FIELD_INITIALIZERS
-- Performing Test CRC32C_HAVE_NO_MISSING_FIELD_INITIALIZERS - Failed
-- Performing Test HAVE_BUILTIN_PREFETCH
-- Performing Test HAVE_BUILTIN_PREFETCH - Failed
-- Performing Test HAVE_MM_PREFETCH
-- Performing Test HAVE_MM_PREFETCH - Failed
-- Performing Test HAVE_SSE42
-- Performing Test HAVE_SSE42 - Failed
-- Performing Test HAVE_ARM64_CRC32C
-- Performing Test HAVE_ARM64_CRC32C - Failed
-- Performing Test HAVE_STRONG_GETAUXVAL
-- Performing Test HAVE_STRONG_GETAUXVAL - Failed
-- Performing Test HAVE_WEAK_GETAUXVAL
-- Performing Test HAVE_WEAK_GETAUXVAL - Failed
-- Configuring done
-- Generating done
-- Build files have been written to: /root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release
crc32c/1.1.2: CMake command: cmake --build "/root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release" '--' '-j12'
[ 20%] Building CXX object CMakeFiles/crc32c_arm64.dir/src/crc32c_arm64.cc.obj
[ 40%] Building CXX object CMakeFiles/crc32c_sse42.dir/src/crc32c_sse42.cc.obj

c1xx: fatal error C1083: Cannot open source file: '': No such file or directory
crc32c_sse42.cc
Generating Code...
make[2]: *** [CMakeFiles/crc32c_sse42.dir/build.make:76: CMakeFiles/crc32c_sse42.dir/src/crc32c_sse42.cc.obj] Error 2
make[2]: *** Deleting file 'CMakeFiles/crc32c_sse42.dir/src/crc32c_sse42.cc.obj'
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/crc32c_sse42.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

c1xx: fatal error C1083: Cannot open source file: '': No such file or directory
crc32c_arm64.cc
Generating Code...
make[2]: *** [CMakeFiles/crc32c_arm64.dir/build.make:76: CMakeFiles/crc32c_arm64.dir/src/crc32c_arm64.cc.obj] Error 2
make[2]: *** Deleting file 'CMakeFiles/crc32c_arm64.dir/src/crc32c_arm64.cc.obj'
make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/crc32c_arm64.dir/all] Error 2
make: *** [Makefile:124: all] Error 2
crc32c/1.1.2: 
crc32c/1.1.2: ERROR: Package '0246896ed8e77a58a3985dffb2e82bf25f71ed9e' build failed
crc32c/1.1.2: WARN: Build folder /root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release
ERROR: crc32c/1.1.2: Error in build() method, line 36
    cmake.build()
    ConanException: Error 2 while executing cmake --build "/root/.conan/data/crc32c/1.1.2/_/_/build/0246896ed8e77a58a3985dffb2e82bf25f71ed9e/build/Release" '--' '-j12'
root@peeranat:/workspace/3rdparty/crc32c#
memsharded commented 12 months ago

Checking the output I can't see anything relevant. The next step would be to reproduce it locally, but I am afraid that setting up that cross-compiler might be not simple.

Does the cross-build of other projects work without problems with that setup? You might try with an out-of-the box conan new hello/0.1 -m=cmake_lib recipe, to see what happens.

I think this could be the use of check_cxx_source_compiles() and other similar CMake utilities. For some reason sometimes they don't play well when cross-building with Conan, but we need to further investigate it.

cocoza4 commented 12 months ago

I confirm that the cross compiling works on other project. And you are right, check_cxx_source_compiles() caused the problem, this command should set HAVE_MM_PREFETCH and HAVE_SSE42 to ON instead of OFF.

memsharded commented 12 months ago

I have tried to make this fail with some simpler cross-build scenarios (Linux86->LinuxArm and Linux86->WindowsMingw64), but without success, your recipe above with something like this cross-builds fine:

[settings]
os=Windows
arch=x86_64
compiler=gcc
compiler.libcxx=libstdc++11
compiler.cppstd=14
compiler.version=9
build_type=Release

[conf]
tools.build:compiler_executables = {"c": "x86_64-w64-mingw32-gcc", "cpp": "x86_64-w64-mingw32-g++"}
cocoza4 commented 12 months ago

can you try with MSVC?

cocoza4 commented 12 months ago

seems like the problem only occurs with MSVC

memsharded commented 12 months ago

Setting up MSVC in Linux seems way more complicated and that it will require more time. Is there any docker image or something like that with things pre-installed we could use to try thing?

cocoza4 commented 12 months ago

@memsharded MSVC on linux can be found here: https://drive.google.com/file/d/1PPALM6jzUMQQ-5IofAGsL5-WOH1-oifn/view?usp=sharing

cocoza4 commented 12 months ago

@memsharded is everything ok, anything else you need from my side?

memsharded commented 12 months ago

I am afraid that I might not be allowed to use such a binary in my computer, this is why I was asking for some docker image, I was expecting that the installation of MSVC on linux could be done from some public official sources, and the dockerfile would help in setting it up too. I will try to check it, but I might not be able, I will keep you posted.

cocoza4 commented 12 months ago

i managed to find the root cause, when I commented the following code, the build passed. Any idea why's that?

[conf]
tools.env.virtualenv:auto_use=True

[buildenv]
CXXFLAGS="/EHsc"
memsharded commented 12 months ago

That code is creating conanbuild.sh or conanbuild.bat that are loaded before executing commands, and they contain information about the environment, like things in tool_requires and such.

I don't know exactly why they would be affecting. Does it work if you only comment the CXXFLAGS but leave the [conf] section enabled?

cocoza4 commented 12 months ago

yes it worked when I commented out CXXFLAGS. Is there a way I can remove CXXFLAGS in conanfile.py?

cocoza4 commented 11 months ago

@memsharded any update on this?