search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.29k stars 984 forks source link

Can't upload a package to artifactory using CONAN_LOGIN_USERNAME and CONAN_PASSWORD #5434

Closed tylerholzer closed 6 months ago

tylerholzer commented 5 years ago

Issue

We're unable to upload a package into a artifactory hosted conan server using the CONAN_LOGIN_USERNAME and CONAN_PASSWORD method of authentication. Rather than authenticating and uploading, the conan command errors with ERROR: 404: Not Found. [Remote: conan_remote]

Environment

Current Conan Version: 1.16.1 Previous Working Conan Version: 1.12.3 Artifactory Version: 6.10.2

Expected Behavior

The conan upload command should see the CONAN_LOGIN_USERNAME and CONAN_PASSWORD environment variables on the command line and use these to authenticate

Releated Issues

I believe a similar issue was briefly explored in #4930 however the fact that the environment variables weren't respected was not addressed and it focused on the 404 status

Command and output with logging level 10

$ CONAN_LOGGING_LEVEL=10 CONAN_LOGIN_USERNAME=conan_username CONAN_PASSWORD=conan_password conan upload -r conan_remote --all boost-python36/1.59.0@build/prod
DEBUG :conan_api.py   [160]: INIT: Using config '/home/user/tholzer/.conan/conan.conf' [2019-07-02 11:33:53,934]
DEBUG :tracer.py      [152]: CONAN_API: upload(confirm=False,package=None,remote_name=conan_remote,pattern=boost-python36/1.59.0@build/prod,policy=None,retry_wait=None,all_packages=True,integrity_check=False,query=None,retry=None) [2019-07-02 11:33:53,966]
Uploading to remote 'conan_remote':
Uploading boost-python36/1.59.0@build/prod to remote 'conan_remote'
DEBUG :rest_client_common.py[110]: REST: ping: http://artifactory.company.com/artifactory/api/conan/conan-repo/v1/ping [2019-07-02 11:34:03,342]
DEBUG :rest_client_common.py[136]: REST: get: http://artifactory.company.com/artifactory/api/conan/conan-repo/v1/conans/boost-python36/1.59.0/build/prod/digest [2019-07-02 11:34:03,364]
DEBUG :rest_client_common.py[38]: REST ERROR: <class 'conans.errors.NotFoundException'> [2019-07-02 11:34:03,376]
DEBUG :rest_client_common.py[102]: REST: Check credentials: http://artifactory.company.com/artifactory/api/conan/conan-repo/v1/users/check_credentials [2019-07-02 11:34:03,427]
DEBUG :rest_client_common.py[136]: REST: get: http://artifactory.company.com/artifactory/api/conan/conan-repo/v1/conans/boost-python36/1.59.0/build/prod [2019-07-02 11:34:03,491]
DEBUG :rest_client_common.py[38]: REST ERROR: <class 'conans.errors.NotFoundException'> [2019-07-02 11:34:03,504]
DEBUG :rest_client_common.py[130]: REST: post: http://artifactory.company.com/artifactory/api/conan/conan-repo/v1/conans/boost-python36/1.59.0/build/prod/upload_urls [2019-07-02 11:34:03,556]
DEBUG :rest_client_common.py[38]: REST ERROR: <class 'conans.errors.NotFoundException'> [2019-07-02 11:34:03,568]
ERROR: 404: Not Found. [Remote: conan_remote]
memsharded commented 5 years ago

Hi @tylerholzer !

I have tried a quick test and it worked, couldn't reproduce.

Could you please verify, that if you do this it works?:

$ conan user --clean
$ conan user conan_username -p conan_password
$ conan upload ...

Just in case, to rule out other possible causes. Many thanks!

tylerholzer commented 5 years ago

Yes using the provided steps, I'm able to authenticate and upload without issue. We're trying to use the environment variable based authentication method in our CI job so that we don't have to expose the password on the CLI.

I did make a discovery that this seems to only happen with new packages and not when attempting to reupload a package. I have not tested with a new version of an existing package. Running the following commands, the second upload command successfully authenticates and states Package is up to date, upload skipped

$ conan user --clean
$ conan user conan_username -p conan_password
$ conan upload -r conan_remote --all newpack/1.0.0@user/channel
$ conan user --clean
$ export CONAN_LOGIN_USERNAME=conan_username
$ export CONAN_PASSWORD=conan_password
$ conan upload -r conan_remote --all newpack/1.0.0@user/channel

If I then delete newpack via the artifactory interface and attempt to run the following, I still see the ERROR: 404: Not Found. [Remote: conan_remote]

$ conan user --clean
$ export CONAN_LOGIN_USERNAME=conan_username
$ export CONAN_PASSWORD=conan_password
$ conan upload -r conan_remote --all newpack/1.0.0@user/channel
memsharded commented 5 years ago

Thanks for the feedback, still can't reproduce. If the user doesn't have the right privileges, lets say the password is incorrect, I get something like:

DEBUG :rest_client_common.py[38]: REST ERROR: <class 'conans.errors.AuthenticationException'> [2019-07-02 16:47:16,503]
Please log in to "art" to perform this action. Execute "conan user" command.
Remote 'art' username: Got username 'admin' from environment
Please enter a password for "admin" account: Got password '******' from environment
DEBUG :rest_client_common.py[86]: REST: Authenticate: http://localhost:8081/artifactory/api/conan/conan-local/v1/users/authenticate [2019-07-02 16:47:16,517]
ERROR: Wrong user or password

So the message Got password '******' from environment should be there, not sure why you are not seeing it. Can you please try that later example with a wrong password, to see what you get? Thanks.

tylerholzer commented 5 years ago

I guess I read too much into the Package is up to date, upload skipped. It happens even when CONAN_PASSWORD is set to an incorrect password. If I modify the package, then try to upload, I get the 404 error again.

Not sure if it could be triggering the behavior, our artifactory server is set to allow anonymous downloads and browsing, however it requires authentication to push artifacts.

memsharded commented 5 years ago

It happens even when CONAN_PASSWORD is set to an incorrect password

setting the CONAN_PASSWORD to a wrong password, won't have effect until it is needed. Unless a conan user --clean is done, the auth tokens are already cached, and the password is not needed.

I would think of some permissions configuration in Artifactory, but as it works when the password is given in conan user -p (I guess it also works interactively), it seems difficult to be an Artifactory permissions issue. Maybe a quick test would be to grab an Artifactory CE free, run it locally in your machine, and try there with the default admin-password user, to try to reduce the scope of our search.

The thing is that CONAN_PASSWORD is basically intercepted and replace the user typed password, I am thinking, but I cannot find where the difference could be...

tylerholzer commented 5 years ago

I downloaded and setup the Artifactory CE 6.10.2 and was able to create a reproducer. By setting the Hide Existence of Unauthorized Resources under the Security Configuration page on my Artifactory CE server, conan will no longer prompt for the username or password.

memsharded commented 5 years ago

Hi @tylerholzer

Ok, good that you were able to reproduce it.

Now I wonder, isn't it exactly the intended behavior? If you set Hide Existence of Unauthorized Resources, that is exactly what it means, it won't return any kind of AuthenticationException or ForbiddenException, and the client will not authenticate against the server. For these scenarios, what it seems to make sense is to do an explicit login before the upload command, with something like conan user -p command. This command will take the env-var from environment so something like:

$ conan user --clean
$ set CONAN_PASSWORD=mypassword
$ conan user mysyusername -p -r=myremote
Please enter a password for "mysusername" account: Got password '******' from environment
Changed user of remote 'myremote' from 'None' (anonymous) to 'mysusername'
$ conan upload zlib* -r=myremote --all --confirm

The above works, and I think it is what makes sense for that server security configuration, and avoids having the password codified in plain text in the scripts, which was the original concern, wasn't it?. Please check and tell me. Thanks for the feedback!

tylerholzer commented 5 years ago

The above example works as described and does allow us not to have to pass the password on the CLI. I would suggest adding that interaction to the documentation. Before creating this issue, I did try and find alternative ways to login, and I didn't find any obvious way to have conan user respect CONAN_PASSWORD.

That being said, in my opinion, if CONAN_LOGIN_USERNAME and CONAN_PASSWORD are in the environment, then conan should try and authenticate with the server before trying any connections. In my mind, the fact that I provided a username and password to conan via documented methods means that when it makes the requests, it will already be authorized and won't receive a 404. This is also desirable from an Artifactory auditing and management standpoint, as unless I am mistaken, conan will operate anonymously for as long as possible, before logging in for the first authenticated action. This could lead to confusing logs and make it difficult to determine who performed what action. These are just my thoughts on the matter, the provided commands do provide a workflow.

memsharded commented 5 years ago

The above example works as described and does allow us not to have to pass the password on the CLI. I would suggest adding that interaction to the documentation. Before creating this issue, I did try and find alternative ways to login, and I didn't find any obvious way to have conan user respect CONAN_PASSWORD.

Sure, lets add it to the docs, you are totally right, it would be useful.

That being said, in my opinion, if CONAN_LOGIN_USERNAME and CONAN_PASSWORD are in the environment, then conan should try and authenticate with the server before trying any connections.

The thing is that conan works with tokens. It will not store the password, but it gets an authentication token from the server, which is cached and later reused. That makes the process much more efficient. If an extra auth request has to be done for every Conan request hitting the server, that would be much slower, which can be very painful for large projects with large dependency graphs.

This is also desirable from an Artifactory auditing and management standpoint, as unless I am mistaken, conan will operate anonymously for as long as possible, before logging in for the first authenticated action. This could lead to confusing logs and make it difficult to determine who performed what action.

That is the point, there are users that prefer to use anonymous operations as much as possible and only use authenticated one when it is really needed. For example, having this as the default policy in CI guarantees that anonymous users (e.g. some developers) will be able to work locally, because they are guaranteed to have access to dependencies. Conan tries to not be opinionated on the flows, in this case it means that the only way to enable different users to have different policies is a explicit call of conan user to proactively perform the authentication, and use the current token/anonymous credentials until the server really requires authentication.

We will clarify all these aspects in the docs, thanks very much for all the feedback!

tylerholzer commented 5 years ago

Thanks for the explanation of the reasoning, I can see why the tool works like it does. I think the clarification of these aspects in the docs will help avoid the confusion I ran into.

I want to thank you overall as well for your help and quick responses on this issue. I'm happy to close this issue as the initial issue was solved with the conan user conan_user -p and CONAN_PASSWORD, however if you would like to keep it open for tracking the documentation changes, I'll leave that up to you.

memsharded commented 5 years ago

Thanks very much for your understanding :)

Yes, sure, I have contributed this PR to the docs: https://github.com/conan-io/docs/pull/1349, we will close this issue when it is merged and docs are published.

memsharded commented 6 months ago

This should have been closed by https://github.com/conan-io/docs/pull/1349 time ago