search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
963 stars 1.77k forks source link

[package] libpq/15.4: broken symlinks in include/catalog/ #24895

Open scandyna opened 3 months ago

scandyna commented 3 months ago

Description

Problem

Uploading libpq/15.4 to a conan remote fails, because the package contains broken symlinks in the include/catalog/ subfolder.

Expected

There are no broken symlinks in the package and upload works.

Workaround

Set the CONAN_SKIP_BROKEN_SYMLINKS_CHECK environment variable to True fixes the upload.

I think that those headers are not used for my particular use case (building Qt5), but it could be a problem for other cases, meaning the check has sense. Also, as far as I understand, this will not work anymore with Conan 2.

Some details

In the generated package, under include/catalog/, there are symlinks, like pg_conversion_d.h, pg_foreign_data_wrapper_d.h and others.

For the particular case of my build, pg_foreign_data_wrapper_d.h points to /root/.conan/data/libpq/15.4/_/_/build/f35a0224260f0d753acb9d90beb1f1c501d0d02b/src/src/backend/catalog/pg_foreign_data_wrapper_d.h.

If the package is just downloaded, the build folder does not exist, or could be located in another place.

I also did look into a libpq/14.5 I installed last year on my machine, and those symlinks do not exist, they are files.

Package and Environment Details

Conan profile

Note: also reproducible with the default profile on Ubuntu 22.04.

In the CI, 2 profiles are used. The host profile is named linux_ubuntu-24.04_gcc13_x86_64_qt_and_more The build profile is named linux_gcc13_x86_64

Configuration (profile_host): [settings] arch=x86_64 build_type=Debug compiler=gcc compiler.cppstd=17 compiler.libcxx=libstdc++11 compiler.version=13 os=Linux os.distribution=ubuntu-24.04 [options] boost:header_only=True icu:shared=False libalsa:shared=True qt:gui=True qt:qtserialport=True qt:shared=True qt:widgets=True qt:with_glib=False qt:with_mysql=False qt:with_odbc=True qt:with_pq=True qt:with_sqlite3=True qt:with_vulkan=False zlib:shared=False [build_requires] [env] [buildenv] CC=gcc-13 CXX=g++-13 Configuration (profile_build): [settings] arch=x86_64 build_type=Release compiler=gcc compiler.cppstd=17 compiler.libcxx=libstdc++11 compiler.version=13 os=Linux [options] [build_requires] [env] [buildenv] CC=gcc-13 CXX=g++-13

Steps to reproduce

To see the broken symlinks, on an Ubuntu 22.04:

conan install libpq/15.4@

Here are the commands in the CI:

conan remote add gitlab_test https://gitlab.com/api/v4/projects/16325699/packages/conan
conan config install --source-folder conan https://gitlab.com/api/v4/projects/16304913/packages/generic/conan1_config/0.1.8/conan_config.zip
conan install libpq/15.4@ --profile:build linux_gcc13_x86_64 --profile:host linux_ubuntu-24.04_gcc13_x86_64_qt_and_more --settings:build build_type=Release --settings:host build_type=Debug --build missing --update
conan user --password ${CI_JOB_TOKEN} --remote gitlab_test ci_user
conan upload "*" --remote gitlab_test --all -c

Logs

Click to expand log ``` Running with gitlab-runner 17.0.0~pre.88.g761ae5dd (761ae5dd) on green-2.saas-linux-small-amd64.runners-manager.gitlab.com/default ns46NMmJ, system ID: s_85d7af184313 Resolving secrets Preparing the "docker+machine" executor 00:21 Using Docker executor with image registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest ... Authenticating with credentials from job payload (GitLab Registry) Pulling docker image registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest ... Using docker image sha256:9500ee95e2e02672198e82ef8f494f55aa34684fdfdc5f1d3dce09eec3345581 for registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest with digest registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13@sha256:c4a1519fdb497e23c944c6162e0179a8483233c9daebe5e3475633438dcc5cfb ... Preparing environment 00:06 Running on runner-ns46nmmj-project-16325699-concurrent-0 via runner-ns46nmmj-s-l-s-amd64-1723369727-4c40f6db... Getting source from Git repository 00:02 Fetching changes with git depth set to 50... Initialized empty Git repository in /builds/scandyna/conan-qt-builds/.git/ Created fresh repository. Checking out eaf500b1 as detached HEAD (ref is testing)... Skipping Git submodules setup $ git remote set-url origin "${CI_REPOSITORY_URL}" Executing "step_script" stage of the job script 00:06 Using docker image sha256:9500ee95e2e02672198e82ef8f494f55aa34684fdfdc5f1d3dce09eec3345581 for registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13:latest with digest registry.gitlab.com/scandyna/docker-images-ubuntu/ubuntu-24.04-build-qt-gcc-13@sha256:c4a1519fdb497e23c944c6162e0179a8483233c9daebe5e3475633438dcc5cfb ... $ conan --version Conan version 1.64.1 $ conan remote add gitlab_test https://gitlab.com/api/v4/projects/16325699/packages/conan WARN: Remotes registry file missing, creating default one in /root/.conan/remotes.json $ conan config install --source-folder conan https://gitlab.com/api/v4/projects/16304913/packages/generic/conan1_config/0.1.8/conan_config.zip Trying to download https://gitlab.com/api/v4/projects/16304913/packages/generic/conan1_config/0.1.8/conan_config.zip Unzipping 19.3KB Installing settings.yml Copying file windows_gcc8_x86_64_qt_widgets_modules_boost to /root/.conan/profiles Copying file windows_gcc8_x86_64_qt_widgets_modules to /root/.conan/profiles Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_tsan_qt_widgets_modules to /root/.conan/profiles Copying file linux_gcc8_x86_64_tsan_qt_core to /root/.conan/profiles Copying file windows_gcc7_x86_64 to /root/.conan/profiles Copying file linux_clang10_x86_64_libc++ to /root/.conan/profiles Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_tsan_qt_widgets_modules to /root/.conan/profiles Copying file linux_gcc8_x86_64_tsan to /root/.conan/profiles Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_qt_widgets_modules to /root/.conan/profiles Copying file linux_gcc7_x86_64_tsan to /root/.conan/profiles Copying file linux_ubuntu-20.04_clang11_x86_64_libc++ to /root/.conan/profiles Copying file linux_ubuntu-24.04_gcc13_x86_64 to /root/.conan/profiles Copying file windows_msvc16_x86 to /root/.conan/profiles Copying file windows_msvc15_x86_64 to /root/.conan/profiles Copying file boost to /root/.conan/profiles Copying file windows_gcc8_x86_64 to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_qt_widgets_modules_boost to /root/.conan/profiles Copying file windows_msvc16_x86_64 to /root/.conan/profiles Copying file linux_gcc7_x86_64_qt_core to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++ to /root/.conan/profiles Copying file linux_gcc7_x86_64 to /root/.conan/profiles Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_qt_widgets_modules_boost to /root/.conan/profiles Copying file qt_core to /root/.conan/profiles Copying file linux_ubuntu-18.04_gcc8_x86_64_qt_widgets_modules_boost to /root/.conan/profiles Copying file linux_to_windows_gcc7_x86 to /root/.conan/profiles Copying file linux_ubuntu-18.04_gcc8_x86_64 to /root/.conan/profiles Copying file windows_msvc16_x86_qt_widgets_modules to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_tsan_qt_core to /root/.conan/profiles Copying file qt_gui_modules to /root/.conan/profiles Copying file linux_ubuntu-24.04_gcc13_x86_64_qt_and_more to /root/.conan/profiles Copying file qt_widgets_modules to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_tsan_qt_gui_modules to /root/.conan/profiles Copying file linux_to_windows_gcc7_x86_64 to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_tsan to /root/.conan/profiles Copying file windows_gcc7_x86_qt_widgets_modules to /root/.conan/profiles Copying file linux_gcc7_x86_64_qt_core_modules to /root/.conan/profiles Copying file windows_gcc7_x86 to /root/.conan/profiles Copying file windows_msvc15_x86_64_qt_widgets_modules to /root/.conan/profiles Copying file linux_clang11_x86_64_libc++ to /root/.conan/profiles Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_tsan to /root/.conan/profiles Copying file windows_msvc16_x86_64_qt_widgets_modules_boost to /root/.conan/profiles Copying file windows_gcc7_x86_64_qt_widgets_modules to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_tsan_qt_core_modules to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libstdc++11 to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_qt_widgets_modules to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_qt_gui_modules to /root/.conan/profiles Copying file linux_gcc8_x86_64 to /root/.conan/profiles Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_qt_widgets_modules to /root/.conan/profiles Copying file linux_gcc7_x86_64_qt_gui_modules to /root/.conan/profiles Copying file qt_core_modules to /root/.conan/profiles Copying file linux_gcc7_x86_64_qt_widgets_modules to /root/.conan/profiles Copying file common_libraries_options to /root/.conan/profiles Copying file linux_ubuntu-18.04_gcc8_x86_64_qt_widgets_modules to /root/.conan/profiles Copying file linux_ubuntu-20.04_clang11_x86_64_libc++_tsan to /root/.conan/profiles Copying file linux_ubuntu-18.04_clang10_x86_64_libc++ to /root/.conan/profiles Copying file linux_gcc13_x86_64 to /root/.conan/profiles Copying file windows_msvc16_x86_64_qt_widgets_modules to /root/.conan/profiles Copying file linux_clang6.0_x86_64_libc++_tsan_qt_widgets_modules to /root/.conan/profiles Copying file linux_ubuntu-18.04_clang10_x86_64_libc++_qt_widgets_modules_boost to /root/.conan/profiles $ conan remote list conancenter: https://center.conan.io/ [Verify SSL: True] gitlab_test: https://gitlab.com/api/v4/projects/16325699/packages/conan [Verify SSL: True] $ conan install ${PACKAGE_NAME}/${PACKAGE_VERSION}@ --profile:build $CONAN_PROFILE_BUILD --profile:host $CONAN_PROFILE_HOST --settings:build build_type=Release --settings:host build_type=$BUILD_TYPE --build missing --update WARN: *** Conan 1 is legacy and on a deprecation path *** WARN: *** Please upgrade to Conan 2 *** Auto detecting your dev setup to initialize the default profile (/root/.conan/profiles/default) Found gcc 13 gcc>=5, using the major as version ************************* WARNING: GCC OLD ABI COMPATIBILITY *********************** Conan detected a GCC version > 5 but has adjusted the 'compiler.libcxx' setting to 'libstdc++' for backwards compatibility. Your compiler is likely using the new CXX11 ABI by default (libstdc++11). If you want Conan to use the new ABI for the default profile, run: $ conan profile update settings.compiler.libcxx=libstdc++11 default Or edit '/root/.conan/profiles/default' and set compiler.libcxx=libstdc++11 ************************************************************************************ Default settings os=Linux os_build=Linux arch=x86_64 arch_build=x86_64 compiler=gcc compiler.version=13 compiler.libcxx=libstdc++ build_type=Release *** You can change them in /root/.conan/profiles/default *** *** Or override with -s compiler='other' -s ...s*** Configuration (profile_host): [settings] arch=x86_64 build_type=Debug compiler=gcc compiler.cppstd=17 compiler.libcxx=libstdc++11 compiler.version=13 os=Linux os.distribution=ubuntu-24.04 [options] boost:header_only=True icu:shared=False libalsa:shared=True qt:gui=True qt:qtserialport=True qt:shared=True qt:widgets=True qt:with_glib=False qt:with_mysql=False qt:with_odbc=True qt:with_pq=True qt:with_sqlite3=True qt:with_vulkan=False zlib:shared=False [build_requires] [env] [buildenv] CC=gcc-13 CXX=g++-13 Configuration (profile_build): [settings] arch=x86_64 build_type=Release compiler=gcc compiler.cppstd=17 compiler.libcxx=libstdc++11 compiler.version=13 os=Linux [options] [build_requires] [env] [buildenv] CC=gcc-13 CXX=g++-13 libpq/15.4: Not found in local cache, looking in remotes... libpq/15.4: Trying with 'conancenter'... Downloading conanmanifest.txt Downloading conanfile.py Downloading conan_export.tgz libpq/15.4: Downloaded recipe revision a3354d556287c3d091c9f4d578b66ec8 Installing package: libpq/15.4 Requirements libpq/15.4 from 'conancenter' - Downloaded Packages libpq/15.4:f35a0224260f0d753acb9d90beb1f1c501d0d02b - Download Installing (downloading, building) binaries... libpq/15.4: Retrieving package f35a0224260f0d753acb9d90beb1f1c501d0d02b from remote 'gitlab_test' Downloading conanmanifest.txt Downloading conaninfo.txt Downloading conan_package.tgz libpq/15.4: Package installed f35a0224260f0d753acb9d90beb1f1c501d0d02b libpq/15.4: Downloaded package revision 0 Aggregating env generators $ conan user --password ${CI_JOB_TOKEN} --remote gitlab_test ci_user Changed user of remote 'gitlab_test' from 'None' (anonymous) to 'ci_user' $ conan upload "*" --remote gitlab_test --all -c Uploading to remote 'gitlab_test': Uploading libpq/15.4 to remote 'gitlab_test' Downloading conan_sources.tgz Recipe is up to date, upload skipped Uploading package 1/1: f35a0224260f0d753acb9d90beb1f1c501d0d02b to 'gitlab_test' ERROR: libpq/15.4:f35a0224260f0d753acb9d90beb1f1c501d0d02b: Upload package to 'gitlab_test' failed: The file is a broken symlink, verify that you are packaging the needed destination files: '/root/.conan/data/libpq/15.4/_/_/package/f35a0224260f0d753acb9d90beb1f1c501d0d02b/include/catalog/pg_foreign_data_wrapper_d.h'.You can skip this check adjusting the 'general.skip_broken_symlinks_check' at the conan.conf file. ERROR: Errors uploading some packages Cleaning up project directory and file based variables 00:00 ERROR: Job failed: exit code 1 ```
scandyna commented 1 month ago

I have downloaded the postgresql sources that the recipe takes, and tried to build it by steps on an Ubuntu-18.04 with gcc. Note: I did out of source build, like documented here: https://www.postgresql.org/docs/15/install-procedure.html

The various make install that are done in the conanfile.py does not bring symlinks in the install path. It also not installs the catalog header files.

In the source tree, in src/include/catalog, we have headers like pg_aggregate.h.

In the build tree, in src/include/catalog, there are symlinks, like pg_aggregate_d.h, with an absolute path to the source tree.

In PR 24755, line 247-248 have been fixed to copy the catalog from the source tree:

copy(self, "*.h", src=os.path.join(self.source_folder, "src", "include", "catalog"),
                  dst=os.path.join(self.package_folder, "include", "catalog"))

Because we do in source build, I think that source_folder and build_folder are the same. This will bring the broken symlinks to the package.

In the build() method, configure and make should be called from the build tree:

def build(self):
  ...
  autotools = Autotools(self)
    with chdir(self, os.path.join(self.build_folder)):
      autotools.configure()
    with chdir(self, os.path.join(self.build_folder, "src", "backend")):
      autotools.make(target="generated-headers")
  ...

The same should be done for the package() method:

def package(self):
  ...
  autotools = Autotools(self)
    with chdir(self, os.path.join(self.build_folder, "src", "common")):
      autotools.install()
  ...

Now, the package will not have the broken symlinks anymore. The headers like pg_aggregate.h will be present in include/catalog, but headers like pg_aggregate_d.h are missing.

This is because they are not in the build tree in src/backend/catalog (only a symlink to a Makefile is present in my case).

In line 254-255, the catalog should also be copied from the source tree:

copy(self, "*.h", src=os.path.join(self.source_folder, "src", "backend", "catalog"),
                            dst=os.path.join(self.package_folder, "include", "catalog"))

I did try to update the recipe locally, and it works (builds, no symlinks and catalog files present, test package works). Because this seems to be an important change, I think I should test it with various configurations. I also completely ignored the MSVC part.

scandyna commented 1 month ago

I work on this on my fork.

Tests are here.