[question] Is there a possibility to search packages/recipes by topics?

[stefanruf]

Hi, at Conan version 1.9 the attribute 'topics' was introduced. I know there is a way to use the --query parameter for search, but for that the reference has to be known. That isn't applicable for me. I want to search for a list of packages or recipes via a single topic or a list of topic-values.

Is there a possibility to search for packages/recipes which has the same topic?

To help us debug your issue please explain:

• [x] I've read the CONTRIBUTING guide.
• [ ] I've specified the Conan version, operating system version and any tool that can be relevant.
• [ ] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[uilianries]

Hi @stefanruf !

Not by an unique command, but it's possible using a small script:

import argparse
from conans.client import conan_api

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('topic', type=str)
    args = parser.parse_args()
    conan, _, _ = conan_api.Conan.factory()
    recipes = conan.search_recipes(None)
    for recipe in recipes["results"][0]["items"]:
        recipe = recipe["recipe"]["id"]
        attributes = conan.inspect(recipe, attributes=["topics"])
        if attributes["topics"] is not None and args.topic in attributes["topics"]:
            print(recipe})

Conan search only inspect options and settings for now, it's a good feature request adding topics.

[stefanruf]

Hi @uilianries,

Thanks for the answer. That helps me for first. Is there already a ticket for the feature request to search via topics, which I can track?

[jgsogo]

Depending on the server you are using to store the packages, you can benefit from other features. For example, for Artifactory, you can transform those topics into properties (using a hook) and then use the Artifactoy API (or JFrog CLI) to query for packages matching really complex search expressions.

Right now topics are written only in the recipe, in order to get them the recipe has to be loaded which can take a lot of time (it needs to resolve all the dependency graph). The other option would be to store the information associated with the packages, but it is a type of information that has nothing to do with a package manager (topics are not like settings or options).

I might be missing some relevant use case, but IMO it is more like a feature to request for the server UI, than something to implement in the client.

[jmpatri1]

I would sure love to search for all packages matching a specific topic. Would allow me to set all my internal packages to use a common namespace theme, then find what's available on the servers via a conan search --topic topicA command

[memsharded]

Hi @jmpatri1

Being able to search by topics in servers too, would require to do server side changes (and all server implementers should do it, and that is complicated, we can only try to suggest it to Artifactory team and even that is complicated, which server are you using?)

Would allow me to set all my internal packages to use a common namespace theme, then find what's available on the servers via a conan search --topic topicA command

How many packages are you using at your organization to require this?

I am afraid this will not be possible at the moment, and very unlikely to be prioritized into the roadmap.

[jmpatri1]

We have a couple of enterprise deployments of Artifactory. At present, I don't think there is a huge number of conan packages in use, however we're pretty early in our conan adoption. We do have something like 20k git repo's around the org and I would say that code discovery is a major issue that a search feature like this could help address.

[amamory]

this script does something similar, it searches for a specific attribute among the packages. In this case, i use it to find all packages with a certain arch name: x86_64, armv8, etc. https://github.com/amamory/conan-scripts/blob/main/search_arch.py any feedback is appreciated

[memsharded]

Hi @amamory

Thanks for sharing this, indeed these kind of scripts can be interesting for other users. Now in Conan 2.0 we have new mechanisms to integrate them, maybe if you want to contribute it as a custom command to https://github.com/conan-io/conan-extensions, that would be great.

Otherwise, at the moment we are not planning to implement a search based on topics. It is implemented in ConanCenter (conan.io/center) web app, but for other servers, we would need to create new client-server APIs, and that is not planned at the moment, too costly and complex, involving servers, etc. This can be probably solved with other parts of existing infrastructure, like searching in source repos, company knowledge bases, etc.

So I am closing this ticket as not planned, thanks for the feedback!

[tim-hilt]

import argparse
from conans.client import conan_api

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('topic', type=str)
    args = parser.parse_args()
    conan, _, _ = conan_api.Conan.factory()
    recipes = conan.search_recipes(None)
    for recipe in recipes["results"][0]["items"]:
        recipe = recipe["recipe"]["id"]
        attributes = conan.inspect(recipe, attributes=["topics"])
        if attributes["topics"] is not None and args.topic in attributes["topics"]:
            print(recipe})

Can something like this also be achieved with the current Conan Python API? I got this far:

import argparse
from conan.api.conan_api import ConanAPI, SearchAPI

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("topic", type=str)
    args = parser.parse_args()

    conan = ConanAPI()
    search = SearchAPI(conan)
    recipes = search.recipes(None)

    # ... Missing inspect API

[memsharded]

Hi @tim-hilt

Sure, or even better, create a custom command: https://docs.conan.io/2/reference/extensions/custom_commands.html#reference-commands-custom-commands

This way your script will be managed, can be shared and conan config install and many more. I strongly recommend creating one instead of using the raw api

[tim-hilt]

@memsharded thanks for the reply! Even if I did create a custom command (which is a good recommendation) I wouldn’t know how to query the found recipes for their topics. Can you help me out there?

[memsharded]

We don't have public API for this specific use case, but you can check the implementation of inspect() in the LocalAPI subapi:

    def inspect(self, conanfile_path, remotes, lockfile):
        app = ConanApp(self._conan_api)
        conanfile = app.loader.load_named(conanfile_path, name=None, version=None,
                                          user=None, channel=None, remotes=remotes, graph_lock=lockfile)
        return conanfile

Once you have the conanfile_path it should be relatively doable. The path can be obtained with the CacheAPI.export_path for every given recipe already in the cache.

We are in the process now of documenting and stabilizing the PythonAPI, we already did the RemotesAPI last Conan 2.1 and we will document some more for 2.2.

However, I'd like to suggest that this topics searching might not be optimal. If you are searching in a repo with many recipes, like conan-center-index, it is much simpler and faster to just text-search with your IDE in the source of the repo including all recipes.

If the packages creating is decentralized, for your own packages created from multitude of source repos, querying all server side to collect all recipes/versions to parse the topics to search will probably be very slow. If there is an information/sharing/discoverability issue within the organization, then, search by topics would be just a small value here?

[tim-hilt]

Unfortunately my use-case involves searching all conan-packages that are deployed on a certain remote. So I don't have all the conanfiles available at search-time.

I understand, that the Conan-Center webapp text-searches the conan-center-index repository, when the user filters by topic?

[memsharded]

I understand, that the Conan-Center webapp text-searches the conan-center-index repository, when the user filters by topic?

This is not done on the fly against the server, recipes are previously processed and indexed in a dedicated DB.

Unfortunately my use-case involves searching all conan-packages that are deployed on a certain remote. So I don't have all the conanfiles available at search-time.

If the repo is large, this will take a lot of time, so you might also want to run it once, and generate some DB or file to query later.

[tim-hilt]

Yes, that's exactly what I'm planning for my use-case: Caching the search every 15 minutes or so