[bug] searching virtual repository does not search component local repository

samuel-emrys commented 2 years ago

This is an issue where when a virtual repository contains both a remote repository and a local repository, only the remote repository will be searched when invoking conan search, despite the local repository containing non-default configured binaries that will install successfully.

Environment Details (include every applicable attribute)

Operating System+version: Arch Linux
Compiler+version: clang 13
Conan version: 1.46.0
Python version: 3.10.0
Artifactory Community Edition (Self Hosted): Conan edition license 7.31.13 rev 73113900

Steps to reproduce (Include if Applicable)

Set up a virtual repository, similarly to as follows, where conan-center-builds is the local repository deployment target of conan-center-virtual and conan-center-remote is the remote repository pointing at https://center.conan.io

Build a package with a non standard configuration:

$ conan install armadillo/10.7.3@ -o use_lapack=openblas --build=missing

Upload non-standard binary to virtual:

$ conan upload armadillo/10.7.3@:28678c86e642de6430f76466384a432cc9b8bd4d --all -r conan-center-virtual

Wipe this from cache to ensure it has uploaded properly and will download:

$ conan remove armadillo
$ conan install armadillo/10.7.3@ -o use_lapack=openblas

Search for this package in the remote
```
$ conan search armadillo/10.7.3@ -r conan-center-virtual | grep 28678c86e642de6430f76466384a432cc9b8bd4d
```
Note that this will return no results, because conan-center-builds is not searched. It will however return results for conan-center-remote. I get 72 configurations without grepping the above command.

Logs (Executed commands with output) (Include/Attach if Applicable)

$ conan remove armadillo
Are you sure you want to delete from 'armadillo/10.7.3' (yes/no): yes
$ conan install armadillo/10.7.3@ -o use_lapack=openblas
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=clang
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux
os_build=Linux
[options]
use_lapack=openblas
[build_requires]
[env]
CC=/usr/bin/clang
CXX=/usr/bin/clang++
armadillo/10.7.3: Not found in local cache, looking in remotes...
armadillo/10.7.3: Trying with 'conan-center-virtual'...
Downloading conanmanifest.txt completed [0.26k]
Downloading conanfile.py completed [13.06k]
Downloading conan_export.tgz completed [0.31k]
Decompressing conan_export.tgz completed [0.00k]
armadillo/10.7.3: Downloaded recipe revision 722293f6c1dcf90989e8f7ff41561f2c
Installing package: armadillo/10.7.3
Requirements
    armadillo/10.7.3 from 'conan-center-virtual' - Downloaded
    hdf5/1.12.0 from 'conan-center-virtual' - Cache
    openblas/0.3.15 from 'conan-center-virtual' - Cache
    zlib/1.2.11 from 'conan-center-virtual' - Cache
Packages
    armadillo/10.7.3:28678c86e642de6430f76466384a432cc9b8bd4d - Download
    hdf5/1.12.0:99f071c4d0b78b591a6ca63057f44e0e6c95701b - Cache
    openblas/0.3.15:9d6665a9a4afaf7838ce20fb44ef94df8bde8e0f - Cache
    zlib/1.2.11:63c2a85d57849e261f98f935b93ecac31ba71b84 - Cache

Installing (downloading, building) binaries...
armadillo/10.7.3: Retrieving package 28678c86e642de6430f76466384a432cc9b8bd4d from remote 'conan-center-virtual'
Downloading conanmanifest.txt completed [44.06k]
Downloading conaninfo.txt completed [1.33k]
Downloading conan_package.tgz completed [565.96k]
Decompressing conan_package.tgz completed [0.00k]
armadillo/10.7.3: Package installed 28678c86e642de6430f76466384a432cc9b8bd4d
armadillo/10.7.3: Downloaded package revision 97a813db630d8de50911bf8282d648dc
openblas/0.3.15: Already installed!
openblas/0.3.15: Setting OpenBLAS_HOME environment variable: /home/user/.conan/data/openblas/0.3.15/_/_/package/9d6665a9a4afaf7838ce20fb44ef94df8bde8e0f
[HOOK - conan-center.py] post_package_info(): [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] OK
[HOOK - conan-center.py] post_package_info(): [LIBRARY DOES NOT EXIST (KB-H054)] OK
zlib/1.2.11: Already installed!
[HOOK - conan-center.py] post_package_info(): [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] OK
[HOOK - conan-center.py] post_package_info(): [LIBRARY DOES NOT EXIST (KB-H054)] OK
hdf5/1.12.0: Already installed!
[HOOK - conan-center.py] post_package_info(): [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] OK
[HOOK - conan-center.py] post_package_info(): [LIBRARY DOES NOT EXIST (KB-H054)] OK
[HOOK - conan-center.py] post_package_info(): [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] OK
[HOOK - conan-center.py] post_package_info(): [LIBRARY DOES NOT EXIST (KB-H054)] OK
Aggregating env generators
$ conan search armadillo/10.7.3@ -r conan-center-virtual | grep 28678c86e642de6430f76466384a432cc9b8bd4d
$

To demonstrate that this does actually return results:

$ conan search armadillo/10.7.3@ -r conan-center-virtual | wc -l
1664

memsharded commented 2 years ago

This doesn't seem a Conan issue, but an Artifactory one.

It seems that the default behavior of Artifactory virtual repository is to cache the remote that was used in the resolution of the recipe and forward search requests to that other repo. But I am no expert in Artifactory.

So I would recommend the following:

Submit a ticket to the Artifactory tracker: https://www.jfrog.com/jira/secure/Dashboard.jspa
If you are a customer, push it via JFrog support
Paste the information here so we can track it, and try to push it (this is out of the scope of the Conan team, we can only try to help, but we cannot solve this ourselves)

samuel-emrys commented 2 years ago

@memsharded no worries, thanks for the direction. I wasn't sure where the lines blurred between the conan team and the artifactory team when it came to conan. I'll post a ticket there when I get a chance.

samuel-emrys commented 2 years ago

https://www.jfrog.com/jira/browse/RTFACT-26879 for tracking

conan-io / conan