search

ska-sa / spead2

Library for the Streaming Protocol for Exchange of Astronomical Data (SPEAD)
http://spead2.readthedocs.io/en/latest/
GNU Lesser General Public License v3.0
23 stars 14 forks source link

Ninja requirement #269

Closed james-smith-za closed 1 year ago

james-smith-za commented 1 year ago

I see here: https://github.com/ska-sa/spead2/pull/246#discussion_r1308085640 and here: https://github.com/ska-sa/spead2/commit/231c3b5fdd5f3e871f03cf34ccd65e548f54c409 that ninja was specifically removed from the requirements.

I'm running into an issue: https://github.com/ska-sa/katgpucbf/blob/1c50f97f977b74206dc1f3c26642e18de35bcf46/Dockerfile#L79

The output from docker build ends up giving me (among other things)

2023-09-08 09:11:42 (63.9 MB/s) - 'spead2-4.0.0.tar.gz' saved [2430189/2430189]

The Meson build system
Version: 1.2.1
Source dir: /tmp/spead2-4.0.0
Build dir: /tmp/spead2-4.0.0/build
Build type: native build
Project name: spead2
Project version: 4.0.0
C++ compiler for the host machine: c++ (gcc 11.3.0 "c++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0")
C++ linker for the host machine: c++ ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 (jinja2, pycparser, packaging) found: YES (/venv/bin/python) modules: jinja2, pycparser, packaging
Found pkg-config: /usr/bin/pkg-config (0.29.2)
Run-time dependency Boost found: YES 1.74.0 (/usr/include)
Run-time dependency dl found: YES
Run-time dependency threads found: YES
Check usable header "libdivide.h" : YES 
Run-time dependency Boost (found: program_options) found: YES 1.74.0 (/usr)
Run-time dependency Boost (missing: unit_test_framework) found: NO (tried system)
Cuda compiler for the host machine: nvcc (nvcc 12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0)
Cuda linker for the host machine: nvcc nvlink 12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0
Checking if "gdrapi library" : links: NO 
Run-time dependency libibverbs found: YES 1.14.39.0
Run-time dependency librdmacm found: YES 1.3.39.0
Run-time dependency libmlx5 found: YES 1.22.39.0
Run-time dependency pcap found: YES 1.10.1
Run-time dependency libcap found: YES 2.44
Checking for function "ibv_modify_qp_rate_limit" with dependencies libibverbs, librdmacm: YES 
Checking for function "recvmmsg" : YES 
Checking for function "sendmmsg" : YES 
Checking for function "eventfd" : YES 
Checking for function "sem_init" with dependency threads: YES 
Checking for function "pthread_setaffinity_np" with dependency threads: YES 
Checking if "function multi-versioning" compiles: YES 
Checking if "MOVNTDQ intrinsic" compiles: YES 
Configuring common_features.h using configuration
Build targets in project: 19

ERROR: Could not detect Ninja v1.8.2 or newer

Earlier up I get this:

Step 10/17 : RUN pip install pip==22.3.1 setuptools==65.6.3 wheel==0.38.4 jinja2 packaging pycparser meson
 ---> Running in e3f4784d2db2

Collecting pip==22.3.1
  Downloading pip-22.3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 37.4 MB/s eta 0:00:00
Collecting setuptools==65.6.3
  Downloading setuptools-65.6.3-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 42.3 MB/s eta 0:00:00
Collecting wheel==0.38.4
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 KB 25.1 MB/s eta 0:00:00
Collecting packaging
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 KB 15.1 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 22.8 MB/s eta 0:00:00
Collecting meson
  Downloading meson-1.2.1-py3-none-any.whl (962 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 962.5/962.5 KB 39.8 MB/s eta 0:00:00
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Installing collected packages: wheel, setuptools, pycparser, pip, packaging, meson, MarkupSafe, jinja2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 59.6.0
    Uninstalling setuptools-59.6.0:
      Successfully uninstalled setuptools-59.6.0

  Attempting uninstall: pip
    Found existing installation: pip 22.0.2
    Uninstalling pip-22.0.2:
      Successfully uninstalled pip-22.0.2

Successfully installed MarkupSafe-2.1.3 jinja2-3.1.2 meson-1.2.1 packaging-23.1 pip-22.3.1 pycparser-2.21 setuptools-65.6.3 wheel-0.38.4

So ... it didn't look as though ninja got installed?

If I add meson-python and ninja to the line in question, then the container image builds. (meson-python by itself doesn't solve the issue.) I'm not quite clear on the difference between meson and meson-python but neither of them appear to have pulled in ninja.

Unless I misunderstood the instructions completely.

bmerry commented 1 year ago

meson-python is the glue that lets Meson be used as a PEP 517 backend. When used in that manner, it will detect whether ninja is already installed, and if not, try installing a ninja from pip (which is just a packaged binary uploaded to pip, since ninja is not Python at all). The recommendation from one of the Meson developers is to install ninja from the OS.

So to bump spead2, I would add ninja to the list of packets to apt-get install in the Dockerfile.

Incidentally, you can remove libboost-system-dev from the installation list, and I think you'll be able to remove libboost-program-options-dev too (it'll mean the binary tools like mcdump and spead2-bench don't get installed, but I don't think they go into the final image anyway).

james-smith-za commented 1 year ago

Interesting. That makes a bit more sense.

Incidentally, when I do remove the boost things as you suggest, I get this:

2.777 The Meson build system
2.777 Version: 1.2.1
2.777 Source dir: /tmp/spead2-4.0.0
2.777 Build dir: /tmp/spead2-4.0.0/build
2.777 Build type: native build
2.777 Project name: spead2
2.777 Project version: 4.0.0
2.777 C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
2.777 C++ linker for the host machine: c++ ld.bfd 2.38
2.777 Host machine cpu family: x86_64
2.777 Host machine cpu: x86_64
2.777 Program python3 (jinja2, pycparser, packaging) found: YES (/venv/bin/python) modules: jinja2, pycparser, packaging
2.777 Found pkg-config: /usr/bin/pkg-config (0.29.2)
2.777 Run-time dependency Boost found: YES 1.74.0 (/usr/include)
2.777 Run-time dependency dl found: YES
2.777 Run-time dependency threads found: YES
2.777 Check usable header "libdivide.h" : YES 
2.777 Run-time dependency Boost (missing: program_options) found: NO (tried system)
2.777 
2.777 meson.build:45:28: ERROR: Dependency "boost" not found, tried system
2.777 
2.777 A full log can be found at /tmp/spead2-4.0.0/build/meson-logs/meson-log.txt
------
ERROR: failed to solve: executor failed running [/bin/sh -c SPEAD2_VERSION=$(grep ^spead2== katgpucbf/requirements.txt | cut -d= -f3) &&     wget "https://github.com/ska-sa/spead2/releases/download/v$SPEAD2_VERSION/spead2-$SPEAD2_VERSION.tar.gz" &&     tar -zxf "spead2-$SPEAD2_VERSION.tar.gz" &&     cd "spead2-$SPEAD2_VERSION" &&     meson setup build &&     cd build &&     meson compile &&     meson install]: exit code: 1

I put back program-options-dev and it seems to have worked now.

bmerry commented 1 year ago

Incidentally, when I do remove the boost things as you suggest, I get this:

Ok, that'll be because spead2 defaults to requiring to build the tools (so that people don't get confused about why they're missing if they aren't installed), but if you pass -Dtools=disabled to meson setup then it'll skip building them and should be fine without libboost-program-options-dev (build times will also be slightly shorter).