Permission Denied (v0.11.0)

[azertypwd]

Hello !

I have some issue with v0.11.0, with approle or with userpass or directly root token i have a Permission Denied (403)

But the same configuration work with v0.10.18

Thanks

[glisav]

Hello @Argelbargel I am using the same version as @azertypwd and having the same issue.

[Argelbargel]

Hi,

could you check if vault3 is indeed the leader of your vault-cluster?

Perhaps you could attach your config-files?

I'm using 0.11.0 without leader-detection in production so the problem seems to be with that part of the code...

[glisav]

Hi @Argelbargel I tried to remove the autoDetectLeader field from configurations and got the following logs:

Starting agent... 2024/11/04 13:21:14 INFO Using configuration from /etc/vault.d/snapshots.yaml... 2024/11/04 13:21:14 WARN unable to refresh auth node=https://$VAULT_ADDR:8200 2024/11/04 13:21:14 ERROR Could not take snapshot of vault nextSnapshot=2024-11-04T14:21:14.236Z error="could not (re-)connect to leader: could not connect to leader"

[Argelbargel]

Hi @glisav

now there seems to be a problem with your config. v0.11.0 expects a list of nodes in vault.nodes.url and the elements of that list may not contain environment variables (that is something i'll be looking at if it is important).

[glisav]

Hi @Argelbargel I am using the following configuration:

config: vault: nodes: urls:

• https://$VAULT_ADDR:8200

  autoDetectLeader: true

[Argelbargel]

the entry in urls should be the "real" url without the environment variable. Currently env vars are not expanded in the list or urls

[glisav]

Sorry for the confusion. It is the real actual value in the configuration properties, I have just used the env var only here in the comment @Argelbargel

[Argelbargel]

Ah. Okay. That is a bit strange.

The configuration i'm using looks like this:

config:
  vault:
    nodes:
      urls:
        - "http://vault-active:8200"
      autoDetectLeader: false

I've just built release v0.11.2 https://github.com/Argelbargel/vault-raft-snapshot-agent/releases/tag/v0.11.2 which contains additional logging. @glisav Could you try it with -l debug?

[glisav]

Hi @Argelbargel I replaced the image tag to the latest one v0.11.2, updated the vault > nodes > urls section, setting autoDetectLeader=true and got the following from the logs:

I don't know why the first attempt of making a snapshot, results in an error "403 Permission Denied"

[Argelbargel]

@glisav could you enable debug logging? On the command-line that should be -l debug as additional argument.

[glisav]

Hi @Argelbargel I enabled the debug mode and from the logs I found out that Raft can find the leader node. I solved this issue by adding the URL of the Vault active node in the config file and setting autoDetectLeader=false Thanks!

[Argelbargel]

@glisav: could you post your logs (of the failing leader-detection) anyway, please? I'd like to check to flow/path the code takes and what the error-messages were in yours and @azertypwd case. I'd write some tests for the leader-detection logic but it seems some of my assumptions/my understanding of what the vault-api does were wrong since it does not work in your real-world cases.

If you could at your configuration file (without credentials! ;-)) and which nodes your vault-cluster has, that would be really great!