Use same artifactory credential to download sources in conan recipes

[derived-coder]

Conan stores the credentials of artifactory in order to download recipes. My use case is, that I have source code in artifactory that the conan recipes is downloading via tools.get The artifactory needs also credentials for downloading the stuff. It would be nice, if conan can reuse these credentials in order to get these artifacts.

[uilianries]

Hi @derived-coder !

I'm curious, how are you dealing with this?

[memsharded]

Yes, this seems to be a valid use case.

It might be a difficult feature to implement, because what conan stores is not the credentials, but stores the user and a JWT token that works with the Conan API. I have no idea if this token could also work for other repository types, so this deserves some investigation first.

[uilianries]

I think it can be implemented by Conan client ...

The tools.get has the parameter auth which can receive CONAN_LOGIN_USERNAME and CONAN_PASSWORD. However, we could pass the conan user instance to re-use both username and password IF the remote address matches, e.g. using the same Artifactory instance for source downloading and package uploading.

We could add a new parameter:

tools.get(url, sha256, ..., remote_credentials="bincrafters")

I'm saying, to download this source, if an authentication is required, then use the same that for the remote "bincrafters", which was computed by conan user command early.

The current option is not far from this approach, but requires a few steps more:

username = os.getenv("CONAN_LOGIN_USERNAME", "user")
password = os.getenv("CONAN_PASSWORD", None)
tools.get(url, sha256, auth=(username, password))

[memsharded]

The problem I see with this approach is that there is no guarantee that the access tokens of the Artifactory Conan repo would be valid for another repo, specially if the other repo is another type, like a generic repo. Yes, in many cases the permissions might be good, but security policies in companies might differ. What the solution would look like for this case?

[uilianries]

The problem I see with this approach is that there is no guarantee that the access tokens of the Artifactory Conan repo would be valid for another repo, specially if the other repo is another type, like a generic repo.

I see. As generic repo can't be added as Conan remote, it won't work for this approach.

in many cases the permissions might be good, but security policies in companies might differ

Do you mean sniffing passwords? Even auth is not safe, which is the regular approach.

What the solution would look like for this case?

New repo category? Add support for generic repo, for Artifactory and Bintray only, by a new argument:

conan remote add sources --generic http://192.168.0.24/api/generic/company/sources
conan user -r sources -p password user

[memsharded]

New repo category? Add support for generic repo, for Artifactory and Bintray only, by a new argument

That is an interesting idea worth exploring.

[derived-coder]

Any update here? It is quite important

[memsharded]

No, this is not going to be fast. We know about the use case, but a correct solution will require a bunch of big changes, like defining new infrastructure, a new table in the credentials DB? How to pass and inject that into the tools without breaking? Implementing this correctly will take some time, and right now the team is over capacity, so this cannot be prioritize.

At the moment there is the above workaround:

username = os.getenv("CONAN_LOGIN_USERNAME", "user")
password = os.getenv("CONAN_PASSWORD", None)
tools.get(url, sha256, auth=(username, password))

This works fine, and it is not that much different than neededing to specify on the command line:

conan remote add sources --generic http://192.168.0.24/api/generic/company/sources
conan user -r sources -p password user

Please note that what it is NOT possible is to use the current credentials for a Conan Artifactory remote for other type of repos (like generic repos), because these credentials are not the same in the general case, and this will easily break and block users that don't share the credentials. So in any case, it will be necessary to provide credentials for those other repos. Providing them as env-vars is still a best practice even for conan repos, because we don't want a text password in the scripts for conan user command. So it will be not much different than just providing env-vars for all cases.

If you want to have a nice message for developers (not CI) you can always do:

try:
     username = os.environ["MY_SOURCES_USER"]
except KeyError:
     raise ConanException("MY_SOURCES_USER env-var not defined. You need to define it to access sources")
...
tools.get(url, sha256, auth=(username, password))

We will work on this when it is possible, sorry that at this moment it needs to wait a bit.

[derived-coder]

The problem is, in some shells (e.g. oh my zshell) everything is stored what you input, i.e. the password is stored when I do an export.... It would help when I input the password and store it somewhere.

[derived-coder]

Any update here? Can you not implement at least the same credentials support you have when uploading the package to artifactory? So that the credentials are need to input from the user and stored somewhere on disk and then reused?

[Nekto89]

So what is the best approach for this problem currently? I thought that having sources in separate generic repository is a good idea but now I have problem with authentication on Artifactory.

[memsharded]

Hi @Nekto89

I think the above might still be the best:

username = os.getenv("MY_REPO_USER", "user")
password = os.getenv("MY_REPO_PASSWORD", None)
tools.get(url, sha256, auth=(username, password))

Is the same approach as when credentials are for git repos, basically treat them as different credentials than the ones of the Conan repository. Reasons:

• Conan does not store user/pass to Artifactory, it uses it to obtain a token, but never stores the password. We won't store the password to Artifactory, and we don't have a protocol for generic repos, as they would put specialized code in the Conan client for this server (Conan client is mostly unaware of Artifactory, it is the server side that implement the Conan protocol).
• The level of authorization/authentication might be very different, there are users out there that they want developers to have access to recipes and packages, but not to the original source code. Recipes with this approach work, and will only fail when try to build from source.

[Nekto89]

Hi @memsharded,

I've noticed that it is possible to upload files to conan repository in the same way as to generic repository (completely custom folder structure). Is it possible to reuse token that is already stored by conan for downloading sources that are stored in such way?

I don't like an idea of injecting separate credentials every time - thirdparties from different channels/repositories might be used in the same build and it would be very inconvenient to use this for developers.

One more bad idea is to store sources as another package via build requirements.

[memsharded]

I've noticed that it is possible to upload files to conan repository in the same way as to generic repository (completely custom folder structure). Is it possible to reuse token that is already stored by conan for downloading sources that are stored in such way?

This might be dangerous, definitely not documented and not supported behavior. Conan repo might at some point become confused by files it is not expecting the its folders. Yes, Artifactory uses a folder-like storage approach and a generic api that can put files there, but not a supported thing and can break anytime, I wouldn't recommend it.

Maybe this needs to step back a little bit about the use case:

• Are the downloads tarballs of third party, open source libraries? Are they your own private source code?
• If it is your own source code, why creating specific zip and putting them in an Artifactory repo?
• If it is just a copy of third party open source libraries, why not leave the repo read-only with anonymous access?

[Nekto89]

Currently the idea is to have: 1) conan repo for open source libraries 2) generic repo for mirroring sources of open source libraries 3) conan repo for libraries that were bought and can be used only by specific team 4) generic repo for storing sources of non-open source libraries

1+2 will have anonymous access, but 3+4 won't. Also Artifactory doesn't allow setting different access to different files/folders inside one repo (or is there some magic option that I've missed?) so there will be multiple of 3+4 repos for different teams inside company. Building from sources will be usual behavior for non-Windows platforms.

[memsharded]

The idea is to move this with a POC to try to see rough edges:

• Add a special "remote" information (can use the remotes api)
• That information should be propagated to the downloads functions in recipes
• Credentials can be stored in the remotes json if that simplifies the POC

Furthermore, lets move this to 2.0 scope, no sense to try to introduce it here.

[memsharded]

Downloading sources got a new "credentials" configuration file in 2.0.X. Please read: https://docs.conan.io/2/reference/config_files/source_credentials.html

It is not planned that a generic repo will share the same credentials (token) than a Conan repo, so reusing a Conan repo token is not a feature that seems feasible.

I think this feature satisfies this request, so closing this ticket. Please test it and report new tickets as necessary, feedback very welcome. Thanks!