pytket build fails because m4 cannot build its binary

[dlyongemallo]

I am trying to following the instructions here to build pytket.

When I run the conan create tket --user tket --channel stable --build=missing -o boost/*:header_only=True -o tklog/*:shared=True -o tket/*:shared=True -tf "" command (at the top of the page), the build process stops with this error:

  *********************************************************
  Recipe 'm4/1.4.19' cannot build its binary
  It is possible that this recipe is not Conan 2.0 ready
  If the recipe comes from ConanCenter, report it at https://github.com/conan-io/conan-center-index/issues
  If it is your recipe, check if it is updated to 2.0
  *********************************************************

ERROR: m4/1.4.19: Error in build() method, line 99
        autotools.configure()
        ConanException: Error 77 while executing

My conan version is 2.0.14.

[cqc-alec]

This sounds like an issue building boost. What is your conan profile? You can run the command

cat $(conan config home)/profiles/default

to see it.

[dlyongemallo]

Here's the output:

[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.version=193
os=Windows

[cqc-alec]

That aligns exactly with what we use for the builds on the CI.

Did you add the remote:

conan remote add tket-libs https://quantinuumsw.jfrog.io/artifactory/api/conan/tket1-libs --index 0

?

[dlyongemallo]

Yes, I ran the commands on this page before doing the above.

(If I run this command now, it results in: ERROR: Remote 'tket-libs' already exists in remotes (use --force to continue))

[cqc-alec]

Maybe you could post the full output of the conan create command?

[dlyongemallo]

Okay, here's the entire output:

$ conan create tket --user tket --channel stable --build=missing -o boost/*:header_only=True -o tklog/*:shared=True -o tket/*:shared=True -tf ""

======== Exporting recipe to the cache ========
tket/1.2.72@tket/stable: Exporting package recipe: C:\Users\davin\workspace\CQCL\tket\tket\conanfile.py
tket/1.2.72@tket/stable: Copied 2 '.py' files: conanfile.py, std_clifford_decomp.py
tket/1.2.72@tket/stable: Copied 3 '.txt' files: CMakeLists.txt, CMakeLists.txt, CMakeLists.txt
tket/1.2.72@tket/stable: Copied 1 '.in' file: tketConfig.cmake.in
tket/1.2.72@tket/stable: Copied 172 '.hpp' files
tket/1.2.72@tket/stable: Copied 274 '.cpp' files
tket/1.2.72@tket/stable: Copied 4 '.json' files: ibm_montreal.json, bug777_circuit.json, lexiroute_circuit.json, lexiroute_circuit_relabel_to_ancilla.json
tket/1.2.72@tket/stable: Exported to cache folder: C:\Users\davin\.conan2\p\tketa4972c3a0c464\e
tket/1.2.72@tket/stable: Exported: tket/1.2.72@tket/stable#6ec7e375e3af3a17c7c9b25c923c4ddf (2023-12-13 04:23:32 UTC)

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows
[options]
boost/*:header_only=True
tket/*:shared=True
tklog/*:shared=True
[conf]
tools.cmake.cmaketoolchain:generator=Ninja

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=msvc
compiler.cppstd=14
compiler.runtime=dynamic
compiler.runtime_type=Release
compiler.version=193
os=Windows
[conf]
tools.cmake.cmaketoolchain:generator=Ninja

======== Computing dependency graph ========
Graph root
    cli
Requirements
    boost/1.83.0#e948ecbe69e6e240658ba2eda46ba8e8 - Cache
    eigen/3.4.0#1bc43ded5116678f202a75aa887ef142 - Cache
    gmp/6.2.1#83f403337b343ce4530d1e9c5fb3dd68 - Cache
    nlohmann_json/3.11.3#45828be26eb619a2e04ca517bb7b828d - Cache
    symengine/0.11.1#8c0efc27eaf2bddd1971bd281bf8eac7 - Cache
    tkassert/0.3.4@tket/stable#8bb12ecc8f32da6f8910ce243881d4a6 - Cache
    tket/1.2.72@tket/stable#6ec7e375e3af3a17c7c9b25c923c4ddf - Cache
    tklog/0.3.3@tket/stable#d2e2c1ed51aeaab48be50f553ce3fc48 - Cache
    tkrng/0.3.3@tket/stable#22f29285e71ec0286d09e8e3b3ed39c6 - Cache
    tktokenswap/0.3.6@tket/stable#3c49cf1643fe4654ab7689bbdd757852 - Cache
    tkwsm/0.3.6@tket/stable#8fbb760c19bba3f184d2838ba1c73f35 - Cache
Build requirements
    autoconf/2.71#53be95d228b2dcb30dc199cb84262d8f - Cache
    automake/1.16.5#058bda3e21c36c9aa8425daf3c1faf50 - Cache
    m4/1.4.19#ec4002e19cde0802f89204cab65f28f9 - Cache
    msys2/cci.latest#5a31efa2bde593541fd5ac3bcc50c01c - Cache
    yasm/1.3.0#fb800a15413dca19bfaef9e4b5d50694 - Cache

======== Computing necessary packages ========
m4/1.4.19: Compatible package ID 723257509aee8a72faf021920c2874abc738e029 equal to the default package ID: Skipping it.
tket/1.2.72@tket/stable: Checking 3 compatible configurations
tket/1.2.72@tket/stable: Compatible configurations not found in cache, checking servers
tket/1.2.72@tket/stable: '98e77a22b519e30ed94cd42ab49cb326bdcfe1e4': compiler.cppstd=17
tket/1.2.72@tket/stable: 'a0fe9cb1c6ed6155bb2bc4efc7c8ae5e9128305d': compiler.cppstd=20
tket/1.2.72@tket/stable: '5da56ac487ac0fedd09f16903a4bbcefc3db2543': compiler.cppstd=23
Requirements
    boost/1.83.0#e948ecbe69e6e240658ba2eda46ba8e8:da39a3ee5e6b4b0d3255bfef95601890afd80709#eb10a41c296a4acc9392570ae8d2cee6 - Cache
    eigen/3.4.0#1bc43ded5116678f202a75aa887ef142:da39a3ee5e6b4b0d3255bfef95601890afd80709#3d4cb80b19cf7de2d5fd0a2413dab746 - Cache
    gmp/6.2.1#83f403337b343ce4530d1e9c5fb3dd68:deeb86d617ccfdc956de72096fc3225aebe5b266#8017e7392f2b0a55a3522a16ff0834e1 - Cache
    nlohmann_json/3.11.3#45828be26eb619a2e04ca517bb7b828d:da39a3ee5e6b4b0d3255bfef95601890afd80709#552828a2560333bcbeff9d087ce0af0f - Cache
    symengine/0.11.1#8c0efc27eaf2bddd1971bd281bf8eac7:f614ab276bd7964e2d2628e3ee9eae9ea2f1641b#0eb12b7af5f1e66a591b64835d79c9b1 - Cache
    tkassert/0.3.4@tket/stable#8bb12ecc8f32da6f8910ce243881d4a6:02b00640c6827e70545e3270a1306f12f7cbaf09#dd55a139d9f4076204c5cf5a0e4bff4c - Cache
    tket/1.2.72@tket/stable#6ec7e375e3af3a17c7c9b25c923c4ddf:cc836a9151f3253864f34c246a2119cefe9bbb79 - Build
    tklog/0.3.3@tket/stable#d2e2c1ed51aeaab48be50f553ce3fc48:fc63140a903c0dd7b21eae859df9e8a710d8e8aa#958112191c83cf3fea7a6191960aa1b9 - Download (tket-libs)
    tkrng/0.3.3@tket/stable#22f29285e71ec0286d09e8e3b3ed39c6:a0de755b62b50efcb25b16ca1f8e7841d86f7d97#68bfeb6abf03d6baab27d6e8bc0fbb41 - Cache
    tktokenswap/0.3.6@tket/stable#3c49cf1643fe4654ab7689bbdd757852:d79541512f046c62a16d6d0708d1606b56c9696c#f876c5d7c697decdb8feaaae106bca8c - Cache
    tkwsm/0.3.6@tket/stable#8fbb760c19bba3f184d2838ba1c73f35:d79541512f046c62a16d6d0708d1606b56c9696c#1b6865a47e8b21b58864211f6b275e60 - Cache
Build requirements
    m4/1.4.19#ec4002e19cde0802f89204cab65f28f9:723257509aee8a72faf021920c2874abc738e029 - Build
    msys2/cci.latest#5a31efa2bde593541fd5ac3bcc50c01c:956a88975bda9dfcc485e2861d71e74bd7e2b9a5#c99a611f14016fba4a51b85204f62fa3 - Cache
Skipped binaries
    autoconf/2.71, automake/1.16.5, yasm/1.3.0

======== Installing packages ========

-------- Downloading 1 package --------
tklog/0.3.3@tket/stable: Retrieving package fc63140a903c0dd7b21eae859df9e8a710d8e8aa from remote 'tket-libs'
tklog/0.3.3@tket/stable: Package installed fc63140a903c0dd7b21eae859df9e8a710d8e8aa
tklog/0.3.3@tket/stable: Downloaded package revision 958112191c83cf3fea7a6191960aa1b9
boost/1.83.0: Already installed! (1 of 13)
eigen/3.4.0: Already installed! (2 of 13)
msys2/cci.latest: Already installed! (3 of 13)
nlohmann_json/3.11.3: Already installed! (4 of 13)
tkrng/0.3.3@tket/stable: Already installed! (6 of 13)
gmp/6.2.1: Already installed! (7 of 13)

-------- Installing package m4/1.4.19 (8 of 13) --------
m4/1.4.19: Building from source
m4/1.4.19: Package m4/1.4.19:723257509aee8a72faf021920c2874abc738e029
m4/1.4.19: Copying sources to build folder
m4/1.4.19: Building your package in C:\Users\davin\.conan2\p\b\m41a8773afc564a\b
m4/1.4.19: Calling generate()
m4/1.4.19: Generators folder: C:\Users\davin\.conan2\p\b\m41a8773afc564a\b\build-release\conan
m4/1.4.19: Generating aggregated env files
m4/1.4.19: Generated aggregated env files: ['conanbuild.sh', 'conanbuild.bat', 'conanrun.bat']
m4/1.4.19: Calling build()
m4/1.4.19: Apply patch (file): patches/1.4.19-0001-open-files-in-binary-mode.patch
m4/1.4.19: Apply patch (file): patches/1.4.19-0002-ar-lib.patch
m4/1.4.19: Apply patch (file): patches/1.4.19-0003-msvc-debug-assertion.patch
m4/1.4.19: Apply patch (file): patches/1.4.19-0004-disable-hardening-in-source.patch
m4/1.4.19: Calling:
 > "/c/users/davin/.conan2/p/b/m41a8773afc564a/b/src/configure" --prefix=/ --bindir=${prefix}/bin --sbindir=${prefix}/bin --libdir=${prefix}/lib --includedir=${prefix}/include --oldincludedir=${prefix}/include gl_cv_func_printf_directive_n=no gl_cv_func_snprintf_directive_n=no gl_cv_func_snprintf_directive_n=no ac_cv_func__set_invalid_parameter_handler=yes
m4/1.4.19: RUN: "/c/users/davin/.conan2/p/b/m41a8773afc564a/b/src/configure" --prefix=/ --bindir=${prefix}/bin --sbindir=${prefix}/bin --libdir=${prefix}/lib --includedir=${prefix}/include --oldincludedir=${prefix}/include gl_cv_func_printf_directive_n=no gl_cv_func_snprintf_directive_n=no gl_cv_func_snprintf_directive_n=no ac_cv_func__set_invalid_parameter_handler=yes
conanvcvars.bat: Activating environment Visual Studio 17 - amd64 - vcvars_ver=14.3
[vcvarsall.bat] Environment initialized for: 'x64'
mkdir: cannot create directory ‘/dev/shm’: Read-only file system

Creating /dev/shm directory failed.
POSIX semaphores and POSIX shared memory will not work

mkdir: cannot create directory ‘/dev/mqueue’: Read-only file system

Creating /dev/mqueue directory failed.
POSIX message queues will not work

configure: loading site script /etc/config.site
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a race-free mkdir -p... /c/users/davin/.conan2/p/msys27f2f094a41efb/p/bin/msys64/usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking for gcc... cl -nologo
checking whether the C compiler works... no
configure: error: in `/c/Users/davin/.conan2/p/b/m41a8773afc564a/b/build-release':
configure: error: C compiler cannot create executables
See `config.log' for more details

m4/1.4.19: ERROR:
Package '723257509aee8a72faf021920c2874abc738e029' build failed
m4/1.4.19: WARN: Build folder C:\Users\davin\.conan2\p\b\m41a8773afc564a\b\build-release
*********************************************************
Recipe 'm4/1.4.19' cannot build its binary
It is possible that this recipe is not Conan 2.0 ready
If the recipe comes from ConanCenter, report it at https://github.com/conan-io/conan-center-index/issues
If it is your recipe, check if it is updated to 2.0
*********************************************************

ERROR: m4/1.4.19: Error in build() method, line 99
        autotools.configure()
        ConanException: Error 77 while executing

It looks like maybe the prefix is set incorrectly to /? Should it be set to the value of ${HOME} instead somehow? But I didn't see any place in the instructions where this is set to something else.

Running the tket "one-off build" command from this page, namely, conan create tket --user=tket --channel=stable --build=missing -o "boost/*":header_only=True, results in the same error.

However, I was able to successfully build the utility libraries as described here, using conan create libs/tkxxx --build=missing -o boost/*:header_only=True for tkxxx = tklog, tkassert, tkrng, tktokenswap, and tkwsm.

[cqc-alec]

Can you just try this:

conan install --requires m4/1.4.19@ --build=missing

Also can you just check that

conan remote list

prints

tket-libs: https://quantinuumsw.jfrog.io/artifactory/api/conan/tket1-libs [Verify SSL: True, Enabled: True]
conancenter: https://center.conan.io [Verify SSL: True, Enabled: True]

?

[dlyongemallo]

I figured out the cause of the above problem, but now I ran into another one.

I normally develop on Linux (where the instructions are working), and I'm trying to set up a development environment on Windows. I'm using Git BASH, and vcvarsall.bat seems not to set up the environment correctly, in particular the paths. Most things work, but while building m4, configure was unable to locate the cl.exe file. This is what causes the above breakage.

I switched to using VS 2022's own Developer Command Prompt and was able to proceed further. Now it is failing with a different error message: LINK : fatal error LNK1189: library limit of 65535 objects exceeded. Here's the relevant part at the end (I can post the entire output if you want but it's mostly a repeat of the above up to here):

tket/1.2.72@tket/stable: Running CMake.build()
tket/1.2.72@tket/stable: RUN: cmake --build "C:\Users\davin\.conan2\p\b\tket1720a250aa5ab\b\build\Release" -- -j16
conanvcvars.bat: Activating environment Visual Studio 17 - amd64 - vcvars_ver=14.3
[vcvarsall.bat] Environment initialized for: 'x64'
[158/158] Linking CXX shared library tket.dll
FAILED: tket.dll tket.lib
C:\WINDOWS\system32\cmd.exe /C "C:\WINDOWS\system32\cmd.exe /C ""C:\Program Files\CMake\bin\cmake.exe" -E __create_def C:\Users\davin\.conan2\p\b\tket1720a250aa5ab\b\build\Release\CMakeFiles\tket.dir\.\exports.def C:\Users\davin\.conan2\p\b\tket1720a250aa5ab\b\build\Release\CMakeFiles\tket.dir\.\exports.def.objs && cd C:\Users\davin\.conan2\p\b\tket1720a250aa5ab\b\build\Release" && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_dll --intdir=CMakeFiles\tket.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MICROS~4\2022\COMMUN~1\VC\Tools\MSVC\1438~1.331\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\tket.rsp  /out:tket.dll /implib:tket.lib /pdb:tket.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO  /DEF:CMakeFiles\tket.dir\.\exports.def  && cd ."
LINK: command "C:\PROGRA~1\MICROS~4\2022\COMMUN~1\VC\Tools\MSVC\1438~1.331\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\tket.rsp /out:tket.dll /implib:tket.lib /pdb:tket.pdb /dll /version:0.0 /machine:x64 /INCREMENTAL:NO /DEF:CMakeFiles\tket.dir\.\exports.def /MANIFEST:EMBED,ID=2" failed (exit code 1189) with the following output:
LINK : fatal error LNK1189: library limit of 65535 objects exceeded
ninja: build stopped: subcommand failed.

tket/1.2.72@tket/stable: ERROR:
Package 'cc836a9151f3253864f34c246a2119cefe9bbb79' build failed
tket/1.2.72@tket/stable: WARN: Build folder C:\Users\davin\.conan2\p\b\tket1720a250aa5ab\b\build\Release
*********************************************************
Recipe 'tket/1.2.72@tket/stable' cannot build its binary
It is possible that this recipe is not Conan 2.0 ready
If the recipe comes from ConanCenter, report it at https://github.com/conan-io/conan-center-index/issues
If it is your recipe, check if it is updated to 2.0
*********************************************************

ERROR: tket/1.2.72@tket/stable: Error in build() method, line 98
        cmake.build()
        ConanException: Error 1 while executing

Actually, let me back up for a bit. The reason I am trying to set up a development environment in Windows is because I'm trying to debug the pytket calls into tket which use pybind11, i.e, I want to step from the python code into the C++ code. I was sort of able to do this in VS Code in Linux by attaching gdb to the C++ process (using something like this), but this is very tedious. Based on this page, it seems that Visual Studio (which requires Windows) has integrated mixed-mode debugging. This is what I'm trying to set up.

So the higher-level question I have is: what environment do tket's own developers normally use to debug pytket? My goal is to set up something as close to this as possible on my machine.

The more immediate question is: what is causing the link object limit to be exceeded?

[cqc-alec]

Regarding the immediate issue, I'm not sure, but you could try replicating the steps from the CI workflow here, though I don't think they're very different from what you're doing.

It is also possible to build with cmake only (no conan), but I don't think we've tested this at all on Windows and it may need some modification.

[cqc-alec]

Regarding the higher-level question: I'm not aware of good tools for debugging across the pybind11 boundary. I would tend to just write some C++ code that reproduces the issue and link it statically with the tket library (for example by adding it to the C++ tests), to get around this issue. Looks like VSCode does have some good tooling for this though...

[github-actions[bot]]

This issue has been automatically marked as stale.