search

pimalaya / himalaya

CLI to manage emails
https://pimalaya.org
MIT License
3.17k stars 95 forks source link

Keyring not working on MacOS #484

Closed ttiurani closed 2 weeks ago

ttiurani commented 1 month ago

Hi!

Thank you for a great project, I'm very much interested in starting using himalaya, and cloned the latest git.

However, I can't seem to get my SMTP to work with IMAP. The first problem I run into is:

https://github.com/pimalaya/himalaya/blob/master/config.sample.toml#L309

and

https://github.com/pimalaya/himalaya/blob/master/config.sample.toml#L474

which fails when I try to run it with invalid TOML. After I comment out the "smtp" backend, I can read my emails via IMAP but SMTP does not seem to want to connect.

Is the master branch just in too much flux right now to be usable?

p.s. Also, on OSX, I can't get any prompt to to access the keyring, it just fails. When I try to reset the configuration, it doesn't prompt for a new password, but just exists. Also, doing a fresh configuration jams completely in the automatic searching for servers step without creating a config.toml, so I can't start from scratch either.

soywod commented 1 month ago

Thank you for a great project, I'm very much interested in starting using himalaya, and cloned the latest git.

Awesome, feel free to join the Matrix room if you have any question. Or use GitHub Discussions.

However, I can't seem to get my SMTP to work with IMAP.

There was a typo in the config sample. I pushed a fix, the SMTP backend config should look like this: https://github.com/pimalaya/himalaya/blob/24c9e3b384cfa627222f30e6b3e09a0266b0f297/config.sample.toml#L470

To summarize, backend sets up the backend to use by default for all features. Then you can customize backends for every feature. For sending message it is message.send.backend.

soywod commented 1 month ago

Also, on OSX, I can't get any prompt to to access the keyring, it just fails.

I am not sure to understand your issue. Could you share your IMAP config?

ttiurani commented 1 month ago

I actually forgot to enable the keyring feature, so after that I was able to run:

himalaya account configure --reset myaccount

which then prompted for both IMAP and SMPT passwords correctly. This is what I have in config.toml:

imap.auth.keyring = "myimap"
smtp.auth.keyring = "mysmtp"

However still after that when I run himalaya I get:

himalaya --trace
2024-09-20T06:52:50.970018Z DEBUG keyring::service: setting global keyring service name `himalaya-cli`
2024-09-20T06:52:50.971258Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-09-20T06:52:50.971337Z DEBUG keyring: getting keyring secret for key `myimap`
Error:
   0: cannot get imap password from global keyring
   1: error while using secret from keyring
   2: cannot get secret from keyring matching `myimap`
   3: No matching entry found in secure storage

Location:
   [snip]/.cargo/git/checkouts/himalaya-1e70121402046f0a/24c9e3b/src/backend/mod.rs:149

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                ⋮ 5 frames hidden ⋮
   6: himalaya::backend::BackendContextBuilder::new::{{closure}}::h0d6564af9944b291
      at <unknown source file>:<unknown line>
   7: himalaya::email::envelope::command::list::ListEnvelopesCommand::execute::{{closure}}::h67ec32dc4ef7fedd
      at <unknown source file>:<unknown line>
   8: tokio::runtime::park::CachedParkThread::block_on::h28065f25dbdf7743
      at <unknown source file>:<unknown line>
   9: himalaya::main::he701a6acf6211af2
      at <unknown source file>:<unknown line>
  10: std::sys_common::backtrace::__rust_begin_short_backtrace::hf3ce97749171bdf8
      at <unknown source file>:<unknown line>
  11: std::rt::lang_start::{{closure}}::h5fe9430594e3a994
      at <unknown source file>:<unknown line>
  12: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h28f55c80744bada5
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284
  13: std::panicking::try::do_call::h2a7711271ca00d3c
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552
  14: std::panicking::try::h8cfa57ed24255592
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516
  15: std::panic::catch_unwind::h4ba1cfc637be3672
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142
  16: std::rt::lang_start_internal::{{closure}}::he9dd7d8444a0863e
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148
  17: std::panicking::try::do_call::h4f92b6155a985b32
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552
  18: std::panicking::try::h1810f6567656aa0f
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516
  19: std::panic::catch_unwind::h979a0e4ea0734504
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142
  20: std::rt::lang_start_internal::hba631f1493ca29ca
      at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148
  21: _main<unknown>
      at <unknown source file>:<unknown line>

and for some reason the keyring doesn't work. I wonder if I need to on OSX somehow give permissions to the himalaya binary? It did however (perhaps) correctly set the password so I would assume it could read it too.

soywod commented 1 month ago

I actually forgot to enable the keyring feature

Yes indeed, the keyring feature is off by default.

No matching entry found in secure storage

Mmh I cannot reproduce your issue. Here what I tried:

  1. Add imap.auth.type = "password" and imap.auth.keyring = "myimap" in my config
  2. Run himalaya account config --reset, give a wrong password when prompted

  3. Run himalaya envelope list:

    2024-09-20T07:10:23.154239Z DEBUG keyring::service: setting global keyring service name `himalaya-cli`    
2024-09-20T07:10:23.156828Z  INFO himalaya::email::envelope::command::list: executing list envelopes command
2024-09-20T07:10:23.156989Z DEBUG keyring: getting keyring secret for key `myimap`    
2024-09-20T07:10:23.436755Z DEBUG client::build: rustls::client::hs: No cached session for DnsName("posteo.de")    
2024-09-20T07:10:23.437039Z DEBUG client::build: rustls::client::hs: Not resuming any session    
2024-09-20T07:10:23.490023Z DEBUG client::build: rustls::client::hs: Using ciphersuite TLS13_AES_256_GCM_SHA384    
2024-09-20T07:10:23.490179Z DEBUG client::build: rustls::client::tls13: Not resuming    
2024-09-20T07:10:23.493173Z DEBUG client::build: rustls::client::tls13: TLS1.3 encrypted extensions: [ServerNameAck]    
2024-09-20T07:10:23.493240Z DEBUG client::build: rustls::client::hs: ALPN protocol is None    
2024-09-20T07:10:23.554343Z DEBUG client::build: email::imap: using password authentication
2024-09-20T07:10:23.554466Z DEBUG client::build: email::imap: supported auth mechanisms mechanisms=[Plain, Login]
2024-09-20T07:10:23.554500Z DEBUG client::build: email::imap: trying auth mechanism… mechanism=Plain
2024-09-20T07:10:25.397993Z  WARN client::build: email::imap: authentication failed mechanism=Plain err=ResolveTask(UnexpectedNoResponse(StatusBody { kind: No, code: Some(Other(CodeOther(b"AUTHENTICATIONFAILED"))), text: Text("Authentication failed.") }))
2024-09-20T07:10:25.398156Z DEBUG client::build: email::imap: trying auth mechanism… mechanism=Login
2024-09-20T07:10:25.398194Z DEBUG client::build: email::imap: trying login…
Error: 
   0: cannot authenticate to IMAP server using LOGIN mechanism
   1: cannot resolve IMAP task
   2: unexpected NO response: Authentication failed.

The secret is correctly taken from the keyring system. Do you confirm that you did the same steps, in the same order?

To check if the issue does not come from OSX, could you try to run the following test?

git clone https://github.com/pimalaya/core.git pimalaya
cd pimalaya
cargo test -p secret-lib --test secret
ttiurani commented 1 month ago

Running that test works.

running 1 test
test test_secret_variants ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
soywod commented 1 month ago

Then I don't get what is happening, it should work. Could you share your whole config?

soywod commented 1 month ago

Also, on which version and/or master commit hash do you run Himalaya CLI?

ttiurani commented 1 month ago

I looked at the Keychain in OSX and it seems reseting isn't creating any entries there. This shows also when I try to run the reset twice in a row: the second run doesn't find anything:

$ himalaya account configure --reset myaccount --trace
2024-09-20T07:49:06.512730Z DEBUG keyring::service: setting global keyring service name `himalaya-cli`
2024-09-20T07:49:06.513929Z  INFO himalaya::account::command::configure: executing configure account command
2024-09-20T07:49:06.513959Z DEBUG keyring: deleting keyring secret for key `myimap`
2024-09-20T07:49:06.514052Z  WARN himalaya::account::command::configure: error while resetting imap secrets: cannot delete password from global keyring
2024-09-20T07:49:06.514063Z DEBUG himalaya::account::command::configure: error while resetting imap secrets: DeletePasswordFromKeyringError(KeyringError(DeleteSecretError(NoEntry, "myimap")))
2024-09-20T07:49:06.514086Z DEBUG keyring: deleting keyring secret for key `mysmtp`
2024-09-20T07:49:06.514105Z  WARN himalaya::account::command::configure: error while resetting smtp secrets: cannot delete password from global keyring
2024-09-20T07:49:06.514108Z DEBUG himalaya::account::command::configure: error while resetting smtp secrets: DeletePasswordFromKeyringError(KeyringError(DeleteSecretError(NoEntry, "mysmtp")))
2024-09-20T07:49:06.514114Z DEBUG keyring: finding keyring secret for key `myimap`
2024-09-20T07:49:06.514133Z DEBUG email::account::config::passwd: cannot find imap password from keyring, setting it
? IMAP password 2024-09-20T07:49:06.514289Z TRACE mio::poll: registering event source with poller: token=Token(0), interests=READABLE
                                                                                                                                         2024-09-20T07:49:06.514324Z TRACE mio::poll: registering event source with poller: token=Token(1), interests=READABLE
> IMAP password ********
2024-09-20T07:49:21.949817Z DEBUG keyring: setting keyring secret for key `myimap`
2024-09-20T07:49:21.949913Z DEBUG keyring: finding keyring secret for key `mysmtp`
2024-09-20T07:49:21.949948Z DEBUG email::account::config::passwd: cannot find imap password from keyring, setting it
> SMTP password ********
2024-09-20T07:49:53.471435Z DEBUG keyring: setting keyring secret for key `mysmtp`
Account myaccount successfully reconfigured!

$ himalaya account configure --reset myaccount --trace
2024-09-20T07:51:03.704386Z DEBUG keyring::service: setting global keyring service name `himalaya-cli`
2024-09-20T07:51:03.706062Z  INFO himalaya::account::command::configure: executing configure account command
2024-09-20T07:51:03.706104Z DEBUG keyring: deleting keyring secret for key `myimap`
2024-09-20T07:51:03.706218Z  WARN himalaya::account::command::configure: error while resetting imap secrets: cannot delete password from global keyring
2024-09-20T07:51:03.706235Z DEBUG himalaya::account::command::configure: error while resetting imap secrets: DeletePasswordFromKeyringError(KeyringError(DeleteSecretError(NoEntry, "myimap")))
2024-09-20T07:51:03.706267Z DEBUG keyring: deleting keyring secret for key `mysmtp`
2024-09-20T07:51:03.706295Z  WARN himalaya::account::command::configure: error while resetting smtp secrets: cannot delete password from global keyring
2024-09-20T07:51:03.706300Z DEBUG himalaya::account::command::configure: error while resetting smtp secrets: DeletePasswordFromKeyringError(KeyringError(DeleteSecretError(NoEntry, "mysmtp")))
2024-09-20T07:51:03.706308Z DEBUG keyring: finding keyring secret for key `myimap`
....

I installed himalaya like this, do I need more features by any chance?

cargo install --git https://github.com/pimalaya/himalaya.git --features keyring --force himalaya
ttiurani commented 1 month ago

Oh, and in March 2024 I ran an older version of himalaya and I could now see that Keyring had those entries correctly there. So this has worked on this machine before.

I also tried to give access to those entries to this new self-built himalaya, but it didn't work: it can't find the entries.

soywod commented 1 month ago

I installed himalaya like this, do I need more features by any chance?

Nop, it should definitely work like this.

Oh, and in March 2024 I ran an older version of himalaya and I could now see that Keyring had those entries correctly there. So this has worked on this machine before.

Interesting, I will investigate the changes.

But I don't understand how the test can pass but not Himalaya CLI. They both do the same thing :thinking:

Could you also try the keyring tests?

cargo test -p keyring-lib --test keyring
ttiurani commented 1 month ago

That passes fine too:

$ cargo test -p keyring-lib --test keyring
warning: unused import: `trace`
  --> keyring/src/lib.rs:19:18
   |
19 | use log::{debug, trace};
   |                  ^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

warning: `keyring-lib` (lib) generated 1 warning (run `cargo fix --lib -p keyring-lib` to apply 1 suggestion)
   Compiling keyring-lib v0.4.3 (/Users/ttiurani/devel/misc/pimalaya/keyring)
    Finished test [unoptimized + debuginfo] target(s) in 2.36s
     Running tests/keyring.rs (target/debug/deps/keyring-37d7f6f70a22a545)

running 1 test
test test_keyring_entry ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
soywod commented 1 month ago

I suspect Himalaya CLI not to take the latest version of secret-lib and keyring-lib. I pushed a fix, could you try to install it again and see?

cargo install --git https://github.com/pimalaya/himalaya.git --features keyring --force himalaya
ttiurani commented 1 month ago

Well that's odd, I ran the above but it didn't pick up the latest commit:

Installing himalaya v1.0.0 (https://github.com/pimalaya/himalaya.git#24c9e3b3)

I then ran:

cargo install --git https://github.com/pimalaya/himalaya.git\#55ecb547c1d2a5fc23acaedbcfd126f3fa98eee7 --features keyring --force himalaya

and that picked up the latest and built fine.

Compiling secret-lib v0.4.6 (https://github.com/pimalaya/core#2767c389)
Compiling keyring-lib v0.4.3 (https://github.com/pimalaya/core#2767c389)

Unfortunately didn't work, no difference.

soywod commented 1 month ago

I propose you a last thing, instead of using cargo install could you use git directly?

cargo uninstall himalaya
git clone https://github.com/pimalaya/himalaya.git
cd himalaya
cargo run --features keyring -- account config --reset
cargo run --features keyring -- envelope list

If it still does not work, then I have no clue on what's going on, and it's gonna be hard to debug since I cannot reproduce. Tests pass so technically keyring works on your machine. I will write down a serie of tests, using different level of libs from low to high (keyring-lib > secret-lib > email-lib > himalaya), and ask you to run it on your machine.

Meanwhile, you can use the OSX security CLI instead:

imap.auth.cmd = "security find-internet-password -s 'myimap'"
ttiurani commented 1 month ago

I tried to compile from sources, but to no avail. Same problem.

However, I did get this to work:

imap.auth.cmd = "security find-generic-password -a 'myimap' -w"
smtp.auth.cmd = "security find-generic-password -a 'mysmtp' -w"

It uses the values that I saved in March with the old Himalaya version. So all is good at least for now!

So I can continue using himalaya for emails now, which is great. Thank you!

I'll close this issue.

ttiurani commented 1 month ago

One more thing: it might be a security issue with different himalaya binaries. When I ran that security, it asked for my password to access the keychain. It might have originally also asked the same of the himalaya binary in March.

However this time I didn't get a popup once about Keychain access with my self-compiled himalaya binary. Don't really know why, but that might be the reason it can't edit/access it.

soywod commented 1 month ago

However this time I didn't get a popup once about Keychain access with my self-compiled himalaya binary. Don't really know why, but that might be the reason it can't edit/access it.

Interesting. Now that keychain access has been asked once for this particular binary, could you try again with the keyring variant? It maybe unlocked things, somehow. If it still does not work then I will keep the issue open till I find the root cause.

soywod commented 1 month ago

I created a new branch keyring-debug where I pushed an example and a binary. Could you please try the following:

cd /path/to/himalaya
git fetch
git checkout keyring-debug

# first test: running the example, which should work like previous keyring and secret tests
cargo run --example keyring --features imap --features keyring

# second test: running the binary, to simulate security issues
cargo build --bin keyring --features imap --features keyring
cd target/debug
./keyring

Additionally, could you try to build a --production binary and see if it changes anything about the keyring situation?

ttiurani commented 1 month ago 
# first test: running the example, which should work like previous keyring and secret tests
cargo run --example keyring --features imap --features keyring

result (there was a prompt, I inserted 1234, seemed to find it):

2024-09-21T08:18:24.647008Z DEBUG keyring::service: setting global keyring service name `himalaya-cli`
2024-09-21T08:18:24.647070Z  INFO keyring: checking keyring-lib
2024-09-21T08:18:24.647103Z DEBUG keyring: setting keyring secret for key `key`
2024-09-21T08:18:24.647226Z DEBUG keyring: getting keyring secret for key `key`
2024-09-21T08:18:24.647293Z  INFO keyring: checking secret-lib
2024-09-21T08:18:24.647305Z DEBUG keyring: getting keyring secret for key `key`
2024-09-21T08:18:24.647343Z DEBUG keyring: setting keyring secret for key `key`
2024-09-21T08:18:24.647398Z DEBUG keyring: getting keyring secret for key `key`
2024-09-21T08:18:24.647453Z  INFO keyring: checking email-lib
2024-09-21T08:18:24.647474Z DEBUG keyring: deleting keyring secret for key `key`
2024-09-21T08:18:24.647530Z DEBUG keyring: finding keyring secret for key `key`
2024-09-21T08:18:24.647584Z DEBUG email::account::config::passwd: cannot find imap password from keyring, setting it
2024-09-21T08:18:24.647605Z DEBUG keyring: setting keyring secret for key `key`
2024-09-21T08:18:24.647647Z DEBUG keyring: getting keyring secret for key `key`
2024-09-21T08:18:24.647689Z  INFO keyring: checking himalaya
2024-09-21T08:18:24.647729Z  INFO himalaya::account::command::configure: executing configure account command
2024-09-21T08:18:24.647762Z DEBUG keyring: deleting keyring secret for key `key`
2024-09-21T08:18:24.647814Z DEBUG keyring: finding keyring secret for key `key`
2024-09-21T08:18:24.647873Z DEBUG email::account::config::passwd: cannot find imap password from keyring, setting it
> IMAP password ********
2024-09-21T08:18:38.598189Z DEBUG keyring: setting keyring secret for key `key`
Account account successfully reconfigured!2024-09-21T08:18:38.598332Z DEBUG keyring: getting keyring secret for key `key`
secret: 1234

Same with the other. However, when I changed that "key" in the examples/keyring.rs to "himalayadebug" (for better searching) and then went to Keychain Access on OSX, I could not find anything related to that key there. So I don't think when you read the secret in the end you are actually reading it from the Keychain. Is it coming from some transient memory?

soywod commented 1 month ago

So I don't think when you read the secret in the end you are actually reading it from the Keychain.

It should definitely be the case. In fact, it could easily be tested by cloning the keyring repository (not the Pimalaya keyring-lib, but the low level one), running their CLI example and see how it behave with Keychain Access:

git clone https://github.com/hwchen/keyring-rs.git
cd keyring-rs
cargo run --example keyring-cli

I also noticed that a new major version is available, so I plan to upgrade and see if it helps. For now it is blocked due to a compilation error https://github.com/hwchen/keyring-rs/issues/214.

soywod commented 4 weeks ago

I think I finally understood what happened. The low-level keyring-rs crate uses mock keystore when no keystore matches the current OS. And it does not support logs, which means you cannot know which keystore is internally used by the lib. I believe the mock keystore was used due to a missing cargo feature for MacOS. I proposed a serie of PRs to improve that:

After that Pimalaya will be able to use the v3 of keyring-rs and it should work as expected. I let you know when it is ready for testing.