Compatibility with keyring (libsecret) for user login credentials

[Lawrencemm]

It would be nice if I could store my credentials for a remote into a keychain such as libsecret like I can do with git.

[memsharded]

I have changed the title, as keychain is more related to iOS apps, which does not relate here.

There is a python interface to libsecret thanks to Gnome: https://developer.gnome.org/libsecret/0.16/py-examples.html, which might be useful for this.

It sounds a bit very specific solution for one platform, I don't know if that will work in other Linuxes, for example, this should be checked.

Also, what would be the interface? Users would be able to opt-in to this? I don't think it should be enabled by default. How the interface would look like?

[Lawrencemm]

A section in Conan config credential with an entry helper which points to an executable that can receive commands to store and retrieve credentials. Absent by default which leads to no attempt by Conan to store/retrieve credentials.

It seems that if the command line interface is well defined for Conan credential helpers, it could be multiplatform and not even specific to keyring implementations of credential management.

[memsharded]

It seems that if the command line interface is well defined for Conan credential helpers, it could be multiplatform and not even specific to keyring implementations of credential management.

No, I don't mean the interface to the library :)

I meant the user interface. How the user should tell conan, in the command line, in conan.conf, with an env-var... how to opt-in to store the credentials in the keyring. What would be the process to remove them from the keyring, if any (also, from the user point of view, not from the code)

[Lawrencemm]

Opt in is just specifying a credential helper.

I would say using the same interface as Git would be a good starting point: https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage

[memsharded]

Ok, thanks for the feedback. That would be the conan.conf file then (which can be set with conan config too).

But I have checked it and bad news:

• There is no python package in PyPI for it: https://pypi.org/search/?q=libsecret. Not having it is almost a blocker, we cannot add dependencies to Conan that are not python packages, otherwise redistribution, creation of the installers, etc., becomes a nightmare.
• The python binding that comes with the linux system package in https://developer.gnome.org/libsecret/0.16/py-examples.html, seems to come licensed by a mixture of licenses, but including GPL terms. So conan (MIT) cannot use it.

Lets wait for feedback, to check if someone comes up with another solution that makes implementation reasonable.

Without these, I think it would be a lot of work for the value, and my recommendation would be to not implement this feature at this moment.

[alacasta]

Hi colleagues, Have you checked if keyring module fits your needs? (https://pypi.org/project/keyring/) The use case that @Lawrencemm mentions can be covered up since it is able to use many different backends (depending on the OS but, from MacOS Keychain to Windows credential manager) or even adapt a customized one (https://pypi.org/project/keyring/#write-your-own-keyring-backend). BTW This module is licensed under MIT License terms, so may works.

It is not that clear to me if he was really asking to provide also external connectivity to one developed by him. If so, i don't either see how to provide an interface for it.

[memsharded]

Hi @alacasta

Thanks for that pointer, very interesting, and it could facilitate things. Still, I am seeing that in order to use it in mainstream systems as Ubuntu, the following is needed:

$ sudo apt install python3-venv libdbus-glib-1-dev
...
$ pip install secretstorage dbus-python
$ pip install keyring

So it would be very difficult to provide a fully automated pip install, but the users would still need to install those things from apt.

Maybe this would be a good candidate for a conan plugin (or maybe a hook? cc @jgsogo @danimtb)

[alacasta]

Hi @memsharded,

Not really sure if all this stuff is needed. As far as I remember from my experience using it on Ubuntu 16 was that worked out of the box. I think that all these recommendations were suggested for using it on Virutual Environments. Isn't it? I think that is enough for this year so, happy new year Conan colleagues, you are doing an incredible job!!

[memsharded]

In Windows it worked ok with the Windows Vault backend. In Linux:

>>> import keyring
>>> keyring.set_password("conan-test", "myuser", "mypassword")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/memsharded/venvs/conan27/local/lib/python2.7/site-packages/keyring/core.py", line 63, in set_password
    _keyring_backend.set_password(service_name, username, password)
  File "/home/memsharded/venvs/conan27/local/lib/python2.7/site-packages/keyring/backends/fail.py", line 23, in get_password
    raise RuntimeError(msg)
RuntimeError: No recommended backend was available. Install the keyrings.alt package if you want to use the non-recommended backends. See README.rst for details.

Also, I see there is no interface to get a user-password pair, only to query the password for a given user, but the user still needs to be defined. So not possible to automatically get the username, only a cached password.

With the above, if this will be considered, I now balance 99% towards doing this in a plugin/hook

[memsharded]

And happy new year too @alacasta ! :) :)

[jgsogo]

Hi! Happy New Year! 🎆

[IMO] If there are some packages/libraries/applications that need to be installed (apt, pip, msi,... or licenses) we should think about a plugin, not a hook. At this moment hooks are functions that get executed associated with different stages or steps related to the recipe workflow, and getting the credentials for a remote it is not about the package workflow (although it is needed).

Thinking about plugins, we haven't decided anything yet (#3778), this one could be an interesting use-case to experiment with plugins as it won't interfere with the Conan model, neither with the recipes. But I'm not sure if this functionality will get enough priority in our queue...

[alacasta]

Hi all, I am not very experienced in Conan (but I'd like to be though) and even less in the plugins/hooks technology. I am not aware about how the future come for increasing Conan capabilities by means of them but for the time being, I have tried to propose a (hook) solution at https://github.com/alacasta/hooks/blob/master/hooks/credentials_manager.py

Of course this is just a proof of concept, but I'd really like to contribute with a solution if you decide to go in some direction adding official support to this functionality.

Thanks!

[tl;dr]

The idea behind is

• If someone wants to use the Credential Manager for his password, a user is needed to be previously set to that remote

  conan user username -r foo

  This allows to discriminate between those cases where anonymous login is allowed.

• Once the user is linked to that repo, then the hook makes the magic but some preconditions are previously checked:
  • keyring and getpass are needed. They can both being installed using pip
  • keyring needs to have a backend already set. Otherwise, an error will be prompted asking to set it up.

Some considerations:

• I know that the issue is still under triaging phase. I made this attemt just for fun and for helping about the possibilities behind. I didn't open PR since no tests have been added either.
• IMO is clear that is not the way since i needed to somehow access to many Conan internals by means of the Clients' API. Probably if exporting additional info to Hooks, accessing to Conan API will not be required.

@memsharded @jgsogo I found a typo in the Hook doc (https://docs.conan.io/en/latest/extending/hooks.html)

• In the Storage, activation and sharing section, there is a hook.conf example where it is indicated this as valid

 ...
 [hooks]
 attribute_checker
 conan-center
 my_custom_hook/hook.py

however, there is a note claiming that .py extension should not be used, so probably it should be

 ...
 [hooks]
 attribute_checker
 conan-center
 my_custom_hook/hook

[jgsogo]

Thanks a lot for the effort and POC, @alacasta! It looks like connecting this functionality is easier that I was expecting.

As you said, there is too much access to Conan internals, and also there isn't a clear hook to call the _ _register_using_nativekeyring function. It is a smell pointing that hooks (at least how they are implemented now, or the ones that are available right now) are not the way to go because they are too tight to the recipe workflow.

The hard part of this issue is opening the door to an architecture for plugins that allow extending some parts of Conan without recipes depending on those plugins to work (a plugin about credentials won't break recipes, so it is a good example to make an experiment).

I feel like that plugins are the way to go (it is just my opinion), I agree that another approach would be to add more triggers that let the user to connect hooks to more functionalities, but I feel we would not be doing the right thing.

[Lawrencemm]

I don't really like the idea of plugins myself, it's a vague concept, you can't have arbitrary functionality added to something and expect it to compose and live well in the long term, just my two cents.

@memsharded I'm not sure why the conclusion of your last point is that plugins should be used. Why not just catch that exception and issue a warning from Conan if there's no backend installed.

[lasote]

@alacasta fixed docs :heart: @Lawrencemm I understand your concern about the plugins. We need to discuss how we want the plugins mechanism but I agree it should provide a way to guarantee easy and complete installation, like the hooks with the conan config install. Even if we decide to install plugins with pip, the error message when the plugin is not there should be clear enough to let you know how to install it.

[memsharded]

@memsharded I'm not sure why the conclusion of your last point is that plugins should be used. Why not just catch that exception and issue a warning from Conan if there's no backend installed.

A couple of reasons:

• Every third party dependency is a potential problem. Yes, they sometimes fail to install in some systems, with different python versions. We have had this issue at least twice with conan dependencies. Even if the issues are reported and addressed to the library maintainers, it takes time to fix them, time that users are blocked and we cannot do anything. All that is at install time, even before conan is launched, so capturing imports errors is not enough.
• Implementing this feature as built-in won't allow the flexibility that the library itself provides. This is an advance feature, users wanting to use it, will probably have different needs, backends, etc. Implementing it as a plugin/hook allows users to very easily customize it. Built-in makes that very difficult.

[memsharded]

In order for this feature to move forward, we should discuss first about the approach, if we are considering the "plugins" one. I am setting the "whiteboard" label to put it in the discussion queue.

[nejch]

I hope the discussion around system-level dependencies like libsecret hasn't derailed this issue unnecessarily. I thought it was common for people to have to add a keyring backend on their OS (usually Mac and Windows have them pre-installed) if one is not present for these things to work.

For reference:

• git: as stated above it's assumed with git that some backends may need to be installed or provided as custom helpers. E.g. see https://blog.scottlowe.org/2016/11/21/gnome-keyring-git-credential-helper/
• pip: the docs are quite happy just telling people to install even keyring separately: https://github.com/pypa/pip/pull/8636/files so they will also have to provide the native OS backend if not present.
• random example (a Jira python CLI): https://github.com/alisaifee/jira-cli/pull/12 also assumes the user has provided a backend.

I'm sure there's more out there. The reason I'm bumping this is that Artifactory, GitLab and Nexus all now support Conan packages, and with private registries setting up all kinds of package manager config can get quite messy.

Edit: personally the pip approach seems quite reasonable, and I'm wondering if following their approach would allow people to set 1 keyring entry for a single host/instance like GitLab serving different repos/packages (e.g. git, pip, conan etc).

[DamDsj]

Hello, Thank you for all the job done. Is there any update concerning password storage on conan.

Gitlab has provided for some time the possibility to push conan packages. For authentication, you can generate access token. The value is only accessible when you create it.

When using pip, everything goes smoothly as it is stored by keyring on your system.

When using conan, I didn't find a way to save it encrypted anywhere. The conan user ... command is requiring often (after restart or after some time) to re-enter the password. As you need to generate access token from gitlab every time you use such command, either you regenerate manually a token from gitlab's settings or you save the credentials into a plain text. Saving the credentials in plain text is not very good security-wise. With such information, you could access to all the recipies (thus the source code) and other package system such as python package. Do you have an idea how I could store safely the conan credentials on my system

Thank you.

[nejch]

In python-gitlab, a recent PR added a very naive helper mechanism that simply expects a credential helper (can be anything that prints out tokens to stdout - keyring, pass, vault, bitwarden, etc). This behaves a bit more like git and docker credential helpers.

https://github.com/python-gitlab/python-gitlab/pull/1359/files

Just thought I'd add this here if it's any use - this would require no additional python dependencies, the user would bring their own helpers.

[memsharded]

We have recently added credentials.json file (with jinja templates too), and made some refactors, and we have read about the new helpers in other platforms, so we think it should be relatively doable to add a plugin/helper solution (in the form of a python extension that users provide, and can be managed with conan config install too) that receives the remote name and URL and returns the credentials to Conan. If you are still interested in this, please let me know.