search

canonical / craft-parts

https://canonical-craft-parts.readthedocs-hosted.com
GNU Lesser General Public License v3.0
11 stars 37 forks source link

Premature parts environment variable expansion #541

Closed NucciTheBoss closed 1 year ago

NucciTheBoss commented 1 year ago

Bug Description

I am trying to create a ROCK for High-Performance Linpack (HPL), a Fortran-based toolkit used to benchmark the Top 500 supercomputers in the world. HPL uses the OpenBLAS library under the hood. Since there are multiple OpenBLAS implementations available in the Ubuntu universe package repository, the Debian alternatives mechanism is used in the OpenBLAS package's postinst script to configure the system-wide OpenBLAS installation. However, since *craft's do not execute Debian package postinst scripts in part staging, I need to create the symlinks that would be created by update-alternatives manually in override-stage. Same situation for libpsm-infinipath1, another dependency of HPL.

HPL expects libopenblas.so.0 and _libpsminfinipath.so.1 to be available under _usr/lib/x8664-linux-gnu so I want to use the CRAFT_ARCH_TRIPLET environment variable, but it is deprecated. I am using $CRAFT_ARCH_TRIPLET_BUILD_FOR instead, but it looks like craft-parts is instead expanding $CRAFT_ARCH_TRIPLET_BUILD_FOR to x86_64-linux-gnu_BUILD_FOR instead. My ROCK packing is failing since x86_64-linux-gnu_BUILD_FOR does not exist under stage/.

To Reproduce

  1. sudo snap install rockcraft --classic --edge
  2. Copy & paste attached rockcraft.yaml file to local directory
  3. wget https://netlib.org/benchmark/hpl/hpl-2.3.tar.gz
  4. rockcraft -v
  5. See error after craftctl default completes in the override-stage step.

part yaml

---
name: linpack
base: ubuntu:22.04
version: "0.1.0"
summary: A ROCK for the High-Performance LINPACK benchmark
description: |
  High-Performance LINPACK (HPL) is a benchmark used to measure the
  performance of the Top 500 supercomputers. This ROCK contains
  the necessary MPI libraries and LINPACK binaries to run the
  benchmark on a Charmed HPC cluster. Libfabric is included
  to assist with MPI bindings to the underlying machine host.
license: Apache-2.0
platforms:
  amd64:

parts:
  linpack:
    source: ./hpl-2.3.tar.gz
    plugin: autotools
    build-packages:
      - gfortran
      - libmpich-dev
      - libatlas-base-dev
      - libopenblas-dev
    stage-packages:
      - libmpich-dev
      - libatlas-base-dev
      - libopenblas-dev
      - libfabric1
    override-stage: |
      set -eux

      craftctl default
      ln --force --verbose \
        usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR/openblas-pthread/libopenblas.so.0 \
        usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR/libopenblas.so.0
      ln --force --verbose \
        usr/lib/libpsm1/libpsm_infinipath.so.1.16 \
        usr/lib/$CRAFT_ARCH_TRIPLET_BUILD_FOR/libpsm_infinipath.so.1

Relevant log output

Starting Rockcraft 0.0.1.dev1                                                                                                                                                                                      
Logging execution to '/tmp/rockcraft.log'                                                                                                                                                                          
CRAFT_ARCH_TRIPLET is deprecated, use CRAFT_ARCH_TRIPLET_BUILD_{ON|FOR}

...

2023-09-06 14:53:44.691 :: 2023-09-06 14:53:38.849 :: + set -eux
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:38.850 :: + craftctl default
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.868 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/openblas-pthread/pkgconfig/blas-openblas.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.869 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/openblas-pthread/pkgconfig/lapack-openblas.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.939 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/pkgconfig/blas-atlas.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.940 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/pkgconfig/lapack-atlas.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.941 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/pkgconfig/libnsl.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.941 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/pkgconfig/libtirpc.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.942 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/pkgconfig/libxcrypt.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:42.943 For pkg-config file /root/stage/usr/lib/x86_64-linux-gnu/pkgconfig/mpich.pc, prefix was changed from prefix=/root/parts/linpack/install/usr
2023-09-06 14:53:44.691 ::  to prefix=/root/stage/usr
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:43.271 :: + ln --force --verbose usr/lib/x86_64-linux-gnu_BUILD_FOR/openblas-pthread/libopenblas.so.0 usr/lib/x86_64-linux-gnu_BUILD_FOR/libopenblas.so.0
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:43.275 :: ln: failed to access 'usr/lib/x86_64-linux-gnu_BUILD_FOR/openblas-pthread/libopenblas.so.0': No such file or directory
2023-09-06 14:53:44.691 :: 2023-09-06 14:53:43.428 'override-stage' in part 'linpack' failed with code 1.
mr-cal commented 1 year ago

Oh no, it looks like we have an overlap between the new CRAFT_ARCH_TRIPLET_BUILD_FOR and the now-deprecated CRAFT_ARCH_TRIPLET project variables (see #514).

As a workaround, you should be able to wrap the variable in brackets (${CRAFT_ARCH_TRIPLET_BUILD_FOR}).

I'm not sure how we should go about solving this in craft-parts.

mr-cal commented 1 year ago

This dictionary needs to be reordered and commented upon. We also need a regression test.