search

wolfi-dev / os

Main package repository for production Wolfi images
Other
828 stars 257 forks source link

pkgconf test pipeline failures #32554

Open dustinkirkland opened 3 weeks ago

dustinkirkland commented 3 weeks ago

We have a handful (~135) packages in wolfi that should be able to successfully use our pkgconf pipeline, but for various reason are not passing those tests. See the attached branch, which adds pkgconf tests to packages which ship a pkgconf/*.pc file: https://github.com/dustinkirkland/wolfi-os/tree/pkgconf.failure

Most of these appear to be "real" test failures, in that various package dependencies are missing. There's perhaps some missing or faulty logic in our SCA?

xnox commented 3 weeks ago

unexpected items in failure basket. will check; in theory given the rolling openssf rebuilds these all should have fallen into place, will dig into what's wrong.

dustinkirkland commented 3 weeks ago

Here's a log of "make package/%" for each of the failing tests: https://gist.github.com/dustinkirkland/6f15a129fd8c2b75216dda6f487907b7

murraybd commented 3 weeks ago

@dustinkirkland This should fix it for some packages.

diff --git a/pipelines/test/pkgconf.yaml b/pipelines/test/pkgconf.yaml
index d36b0393c..6a1258265 100644
--- a/pipelines/test/pkgconf.yaml
+++ b/pipelines/test/pkgconf.yaml
@@ -16,7 +16,7 @@ pipeline:
         if grep -q "^Name:" $pc_file; then
           lib_name=$(basename "$pc_file" ".pc")
           # Ensure .pc file is installed in usr/lib/pkgconfig/*.pc
-          echo "$pc_file" | grep -q "^usr/lib/pkgconfig/${lib_name}.pc$"
+          echo "$pc_file" | grep -q "^usr/lib\|share/pkgconfig/${lib_name}.pc$"
           # Try to get the library name from the pc filename
           # Run some package config tests, which will likely be used by
           # other packages that have build dependencies on this dev package
murraybd commented 3 weeks ago

PR here https://github.com/wolfi-dev/os/pull/32701

dustinkirkland commented 3 weeks ago

Ooh....let me test that....

dustinkirkland commented 3 weeks ago

Okay, with that patch, these package tests now succeed:

murraybd commented 3 weeks ago

Some debugging of one of the failures:

/home/build # pkgconf --print-errors --exists glew; echo $?
Package osmesa was not found in the pkg-config search path.
Perhaps you should add the directory containing `osmesa.pc'
to the PKG_CONFIG_PATH environment variable
Package 'osmesa', required by 'glu', not found
1

An apk add of mesa-dev sorted that package at least.

murraybd commented 3 weeks ago

This PR adds some more output (--print-errors from above) to the pipeline so should be helpful.

https://github.com/wolfi-dev/os/pull/32715

murraybd commented 3 weeks ago

Here are some updated test results using the 2 changes to the pkgconf pipeline:

https://gist.github.com/murraybd/6608af12ad1d0010b446a52eb36a385e

murraybd commented 3 weeks ago

The following are the failing packages from the above gist:

kaniini commented 3 weeks ago

Would it be useful if pkgconf had a machine readable version of —print-errors?

dustinkirkland commented 1 week ago

Machine readable is fine....but I'd find more useful, a handy, definitive error message that explains what exactly is wrong and how to fix it, @kaniini

kaniini commented 1 week ago

I can look into improving the error messages. Originally, we used the same error messages as the original freedesktop.org pkg-config for compatibility, but have deviated in the name of usability in other error reporting areas already.

sanjayk0508 commented 6 days ago

The following are the failing packages from the above gist:

  • [ ] apr-util
  • [ ] at-spi2-core
  • [ ] audit - installs .pc files in /usr/local/lib
  • [ ] brew
  • [ ] checkov
  • [ ] flac
  • [ ] freerdp
  • [ ] gcc-6
  • [ ] gi-docgen - installs .pc files in /usr/local/share
  • [ ] glew
  • [ ] glu
  • [ ] graphene
  • [ ] gtk-2.0
  • [ ] highway
  • [ ] kubeflow-katib
  • [ ] MORE

Hi @murraybd, I’m planning to work on fixing a few failing packages from the recent issues. I noticed that several packages are failing due to missing dependencies or .pc files for required libraries like libcrypto, libxml-2.0, and libgsf-1. Could you provide some guidance on how to address these missing dependencies in the test environment? Specifically, are we supposed to update the apk dependencies, adjust the PKG_CONFIG_PATH, or modify the pipeline to handle missing .pc files more gracefully? Any examples or steps to properly configure the environment and resolve these issues would be really helpful for getting started.

murraybd commented 6 days ago

Hi @murraybd, I’m planning to work on fixing a few failing packages from the recent issues. I noticed that several packages are failing due to missing dependencies or .pc files for required libraries like libcrypto, libxml-2.0, and libgsf-1. Could you provide some guidance on how to address these missing dependencies in the test environment? Specifically, are we supposed to update the apk dependencies, adjust the PKG_CONFIG_PATH, or modify the pipeline to handle missing .pc files more gracefully? Any examples or steps to properly configure the environment and resolve these issues would be really helpful for getting started.

The missing dependencies should be tracked down and added to the apk dependencies, so for example I looked at apr-util and added libapr-dev to the runtime dependencies of apr-util-dev and rebuilt the package ran the test and then pkgconf was happy.

https://github.com/wolfi-dev/os/pull/34655

However, I expect there to be some dependency chains here e.g. libxrandr-dev fails due to xrender being missing and then make test/xrender with the pkgconf pipeline added fails.