[feature] Generate a post-install report after "conan install"

[unfor19]

Hi,

I'm using the command conan install to install external packages and generate an --installer-folder.

I need a way to determine how many packages were Downloaded/Cached/Built during the conan install process.

Maybe if there was a new flag --packages-report or --packages-report-json, like:

conan install --packages-report --install-folder ./install_folder  ...
# Generates the JSON file ./install_folder/conanpackagesreport.json

And the expected content would be something like this

./install_folder/conanpackagesreport.json

{
    "summary": {
        "built": 2,
        "from_cache": 4,
        "downloaded": 2
    }
    ... // Add additional meaningful data here
}

@memsharded I'm totally in for developing this feature by myself, though I want to make sure you'll approve it when the day comes ... What do you think?

• Conan version - 1.49.0
• OS: Any
• [x] I've read the CONTRIBUTING guide.

[memsharded]

Hi @unfor19

Quick question, isn't the conan install --json output what you are looking for? Please have a look and let me know.

Otherwise, for new features, they should be targeted already to Conan 2.0, to the develop2 branch. If looking into 2.0, there are a couple of things that you need to check first:

• Conan user custom commands. It is possible to create and manage custom commands, using the Conan new PythonAPI. This is probably a recommended way for this feature, because every user might want different reports, in different ways and formats, and this can achieve it.
• You also want to have a look to the new deployers with --deploy=mydeployer. Not documented yet, but you can have a look to the test suite in the develop2 branch. They can also be used to collect data about the installed dependencies, and generate reports.
• Also, most commands will have a full --format=json output with quite complete reports.

Please have a look at these features, try them for your goal and let us know.

[unfor19]

@memsharded Wow Thank you for the detailed answer, I'll check it out.

I've just tested the conan---json .release.json flag and the output is not exactly what I'm looking for. I think that it's best if I share my use-case.

In the CICD:

• conan install
• conan upload "*" (uploads EVERYTHING)
• conan build my-package
• conan package my-package
• conan export-pkg my-package
• conan upload my-package

The reason for upload "*" is that if something was built in the CI, I want it to be uploaded to save time for our developers; We've come to a point where we consume many packages, so I want to be able to know what happened after conan install, so if everything was used from Cached then I would like skip "upload EVERYTHING" step.

Main considerations -

1. Avoid hitting my private JFrog Conan Artifactory with unnecessary uploads
2. Avoid Wasting 1 minute per CI process for each build
3. Avoid running something that shouldn't run as it may lead to undesired output

Regarding Conan v2, I'm still scared of it, need to read more about the migration requirements :)

[memsharded]

The reason for upload "*" is that if something was built in the CI, I want it to be uploaded to save time for our developers; We've come to a point where we consume many packages, so I want to be able to know what happened after conan install, so if everything was used from Cached then I would like skip "upload EVERYTHING" step.

Some implementation details: Conan is able to skip a ton of upload, because it has file de-duplication (against Artifactory), but also some checking of checksums to avoid unnecessary transfers. You might want to give it a try, compare, and if necessary report, but it might not have a huge impact on wasted time in practice.

Conan 2.0 will further use the revisions feature, and avoid more unnecessary uploads and downloads if the revisions already exist in the server.

Is the 1 minute some real data of wasted time or just an estimation? Do you have some accurate measurements of the time? I am curious where 1 minute will go if nothing needs to be uploaded, in my experience it is not an expensive operation. For example, conan upload * -r=myremote --all -c --parallel with 12 recipes and their 12 binaries will take just around 3 seconds if it is a no-op, nothing actually uploaded.

Regarding Conan v2, I'm still scared of it, need to read more about the migration requirements :)

There is a useful guide in https://docs.conan.io/en/latest/conan_v2.html, to start to get your recipes ready for 2.0, in 1.X. There will be a subset of recipe syntax that will be 1.X-2.0 compatible, so the recommended approach is to start working on the recipes while in 1.X, then later introduce a parallel 2.0 pipeline in your CI that builds with 2.0, and when it is mostly green (red wouldn't block production), then do the switch over to 2.0.

Regarding the actual issue, I am reading https://docs.conan.io/en/latest/reference/commands/output/install.html, and it seems the remote, download and cache fields contain enough information to see what is "new" and should be uploaded, isn't it? Please let me know

[unfor19]

@memsharded I'll measure it and get back with an answer, we're running multiple CI/CD processes concurrently, so I'm trying to avoid a server overload, especially because the server is not HA (using OSS for now) so if it crashes due to high load of requests, it just crashes :\

Regarding the real issue - I know why I thought the --json filename doesn't answer my requirements, I searched for the word cache in the output file and haven't found anything, so I figured it doesn't work. As an example I got this (filtered so it'll include only Boost):

TL;DR Output -

        "downloaded": false,
        "exported": false,
        "error": null,
        "remote": null,

Raw Output:

{
  "error": false,
  "installed": [
    {
      "recipe": {
        "id": "boost/1.78.0#912b8b9ea38e6b85d815a9efecb5f689",
        "downloaded": false,
        "exported": false,
        "error": null,
        "remote": null,
        "time": "2022-07-25T11:28:46.812723",
        "dependency": true,
        "name": "boost",
        "version": "1.78.0",
        "user": null,
        "channel": null
      },
      "packages": [
        {
          "id": "1d65a000d13da3296f3a499e53c72b11c93926bf",
          "downloaded": false,
          "exported": false,
          "error": null,
          "remote": null,
          ...
          }
        }
      ]
    }
  ]
}

@memsharded As you can see from the above output, I got downloaded = false and the cache key does not exist (reminding you I'm on Conan v1.49.0); Hence I can't really tell if it's installed from cached, or built from cache. I would love to hear your thoughts about it. Can I assume that if cached and downloaded keys do not exist, means that everything was installed from cache?

And thanks a ton for all the links that you're sharing here, super userful

[memsharded]

I'll measure it and get back with an answer, we're running multiple CI/CD processes concurrently, so I'm trying to avoid a server overload, especially because the server is not HA (using OSS for now)

You mean ArtifactoryCE? Or the open-source conan_server? Because ArtifactoryCE is completely free, and it can be used in production, easy to maintain, and much faster than conan_server.

Now I see about the keys. Apparently the cache key is not there, and the state is a combination of downloaded, exported and built. If neither of these is True, then it can be assumed that the artifact was already in the cache. If you are focusing on the packages built from sources, maybe checking the built field is enough.

The above still holds, it is unlikely that we will be doing changes to this output, but the Conan 2.0 output for --format=json now in beta is the best time to give feedback, as it will be way easier to change if necessary.

[unfor19]

I guess I'm using Artifactory CE -

Regarding the use in production, I assume you refer to "make it Highly available" (as I mentioned earlier) though I can't find a way to make it highly available "for free", see the docs:

Licensing Artifactory HA is supported with an Enterprise License. Each node in the cluster must be activated with a different license. Before adding any additional node, please add each additional node's license using the node that is already running. source

But that's another topic and I don't want to spam you with questions about it over here :)

So getting back to the original issue, I'll play with the existing --json report a bit more, I think it'll answer my needs, closing this ticket.

@memsharded Thank you for your support, I'm using Conan at my workplace all over the place, and building for almost any platform - macOS, iOS, android-armv8, Linux, Windows (was a major challenge), Web Assembly (WASM, not even sure it's possible with Conan). I would love to have a chat with you about it, to give you the perspective of a company that dropped VCPKG and moved to Conan, I have a feeling it will be fruitful for both of us. Let me know if you want to peer about it; Either way, thank you!

[memsharded]

Regarding the use in production, I assume you refer to "make it Highly available" (as I mentioned earlier) though I can't find a way to make it highly available "for free", see the docs:

Yes, you are totally right, the HA feature of Artifactory is a paid, one, I was confused by the OSS, and I thought it could be the conan_server

So getting back to the original issue, I'll play with the existing --json report a bit more, I think it'll answer my needs, closing this ticket.

Great. Let us know how it goes, I hope you can make it work.

@memsharded Thank you for your support, I'm using Conan at my workplace all over the place, and building for almost any platform - macOS, iOS, android-armv8, Linux, Windows (was a major challenge), Web Assembly (WASM, not even sure it's possible with Conan). I would love to have a chat with you about it, to give you the perspective of a company that dropped VCPKG and moved to Conan, I have a feeling it will be fruitful for both of us. Let me know if you want to peer about it; Either way, thank you!

Sure, I am definitely willing to learn from your experience, please feel free to send an email to info@conan.io, and we can take it from there. Cheers!