GVA-Guillaume commented 1 year ago

Hi, I have been following the steps in the "Vault HA Cluster with Integrated Storage" tutorial and in the

But right from the unseal configuration step I get this errors after the cluster.sh setup vault_1, worked fine [server1] enabling the transit secret engine and creating a key to auto-unseal vault cluster Success! Enabled the transit secrets engine at: transit/ Success! Data written to: transit/keys/unseal_key /vault/data # sh /vault/deploy/raft/raft-storage/local/cluster.sh setup vault_2

[vault_2] starting Vault server @ https...

Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal

[vault2] initializing and capturing the recovery key and root token Get "https:// /v1/sys/seal-status": dial tcp :8200: connect: connection refused /vault/data # sh /vault/deploy/raft/raft-storage/local/cluster.sh setup vault_3

[server3] starting Vault server @ https:// 3:8200

Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal /vault/data # sh /vault/deploy/raft/raft-storage/local/cluster.sh setup vault_4

[server4] starting Vault server @ https:/ .4:8200

Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal /vault/data # vault operator raft list-peers

the vault status works fine :

Key Value

Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.13.3 Build Date 2023-06-06T18:12:37Z Storage Type inmem Cluster Name ... Cluster ID ... HA Enabled false

Start vault server locally for the purposes of integration tests.

version: "..." services: server1: image: vault hostname: server1 container_name: server environment:

VAULT_CAPATH=/etc/vault/ca.crt
VAULT_ADDR=https:/...1:8200
VAULT_API_ADDR=https... 1:8200
VAULT_SKIP_VERIFY=true
VAULT_CACERT=....agent-ca.pem
VAULT_CAPATH=...agent-ca.pem
VAULT_CLIENT_CERT=...server-0.pem
VAULT_CLIENT_KEY=...server-0-key.pem ports:
...:8200
...:8201 cap_add:
IPC_LOCK networks:
vault_net privileged: true volumes: ...

volumes: ... networks: ...

yhyakuna commented 1 year ago

Hi @GVA-Guillaume

Did you modify the script to use https instead of http? For the purpose of quick demo, the tutorial runs Vault with TLS disabled.

Looking at your output, it's trying to connect to https://127.0.0.2:8200/ instead of http://127.0.0.2:8200/ which is returning you connection error.

[server2] starting Vault server @ https://127.0.0.2:8200/

Using [server1] root token (hvs....) to retrieve transit key for auto-unseal

[server2] initializing and capturing the recovery key and root token
Get "https://127.0.0.2:8200/v1/sys/seal-status": dial tcp 127.0.0.2:8200: connect: connection refused

GVA-Guillaume commented 1 year ago

Hi yhyakuna I feel better with the script

everything, is doing better and better, I think I would help you to modify the tutorial script, but anyway,

i am blocked to the joining stem

1) i understood that I need to be authenticated with the ROOT_TOKEN to launch servers 👍

the different vues 👍

export VAULT_ADDR="---:8200" /vault/data # vault status Key Value

Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.13.3 Build Date --- Storage Type file Cluster Name --- Cluster ID --- HA Enabled false /vault/data #

export VAULT_ADDR="---:8200"

vault status

Key Value

Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 1 Threshold 1 Version 1.13.3 Build Date .... Storage Type raft Cluster Name .... Cluster ID ... HA Enabled true HA Cluster https://....2:8201 HA Mode active Active Since ... Raft Committed Index 46 Raft Applied Index 46 /vault/data #

vault operator raft list-peers

Node Address State Voter

dzisizasekret-vs2 ...2:8201 leader true /vault/data #

export VAULT_ADDR="---8200"

vault status

Key Value

Recovery Seal Type transit Initialized false Sealed true Total Recovery Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.13.3 Build Date ... Storage Type raft HA Enabled true /vault/data #

vault operator raft join -leader-ca-cert=...-agent-ca.pem -address $FOLLOWER $LEADER Error joining the node to the Raft cluster: Error making API request.

URL: POST https://127.0.0.3:8200/v1/sys/storage/raft/join Code: 500. Errors:

failed to join raft cluster: failed to get raft challenge >> I don't know if I need to be unsealed before the joining process

but I am blocked to join raft 2

My script :

RETIRED

GVA-Guillaume commented 1 year ago

vault env setting : /vault/data # set BB_ASH_VERSION='1.36.1' FUNCNAME='' HISTFILE='/root/.ash_history' HOME='/root' HOSTNAME='...1' IFS=' ' LINENO='' OLDPWD='/' OPTIND='1' PATH='/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' PPID='0' PS1='\w \$ ' PS2='> ' PS4='+ ' PWD='pwd' SHLVL='1' TERM='xterm' VAULT_ADDR=... VAULT_API_ADDR=... VAULT_CACERT='....agent-ca.pem' VAULT_CAPATH='....ca.pem' VAULT_CLIENT_CERT='....cert.pem' VAULT_CLIENT_KEY='...key.pem' VAULT_SKIP_VERIFY='true

GVA-Guillaume commented 1 year ago

ok, after a test it seams the problem is the fact vault_3 and vault_4 there are not unsealed, then we are not able to join them to cluster vault_2, it works when I use manually vault operator init / vault operator unseal

0:08 vault server 2.hcl 0:00 vault server 3.hcl 0:00 vault server 4.hcl

yhyakuna commented 1 year ago

@GVA-Guillaume It seems that the auto-unseal failed which is provided by vault_1 node. So, it worked after you manually unsealed the servers which use Shamir's keys instead of auto-unseal key.

You might want to add api_addr to your config-vault_1.hcl.

storage "inmem" {}

listener "tcp" {
   address = "127.0.0.1:8200"
   tls_disable = false
   tls_cert_file = "/vault/certs/dias-server-dzisizasekret-0.pem"
   tls_key_file  = "/vault/certs/dias-server-dzisizasekret-0-key.pem"
   tls_disable_client_certs = "true"   
}

ui = true
disable_mlock = true
api_addr = "https://127.0.0.1:8200"

GVA-Guillaume commented 1 year ago

Same situation 👍 yhyakuna Yoko Hyakuna,

your assistance is super cool, but I did not check changes

Modu operandi :

1) boot from 1.hcl (the conf you edited above) 2) create network 3) cd /data 4) create config 5) move *.hcl to /config 6) setup vault_1 7) vault status and login with vault_1 root token 8) setup vault_2 9) switch to vault_2 (export ..2:8200) 10) check first raft node (OK) 11) setup vault_3 and vault_4 (pending) 12) switch to vault_3 (export ...3:8200) 13) try to join vault_2 cluster

process failed :

Error joining the node to the Raft cluster: Error making API request.

URL: POST http://127.0.0.3:8200/v1/sys/storage/raft/join Code: 400. Raw Message:

Client sent an HTTP request to an HTTPS server.

(i also tried the autopilot and same thing the vault_2 in 8200 port does not mount, stdout : connection refused) 👍 https://developer.hashicorp.com/vault/tutorials/raft/raft-autopilot

yhyakuna commented 1 year ago

@GVA-Guillaume I'm getting confused... Shouldn't you be setting the environment variable to use https instead?

export VAULT_ADDR="https://127.0.0.3:8200"

GVA-Guillaume commented 1 year ago

Step 1 :

IMG RET

step2: IMG RET

Step 3 : IMG RET

Step4: IMG RET

step5: IMG RET

step6: IMG RET

Step7: IMG RET

step8: IMG RET

Step9: IMG RET

step10: IMG RET

Step11: The server is not initialized and unsealed, IMG RET

Step 12 : Vault operator init ; the server is initialized and automatically unsealed IMG RET

Step 13: the server joined the leader ${} IMG RET

Step14: I am able to see the vault_3, but. not as a follower like a leader IMG RET

step15: the server vault_4 joined the leader ${} IMG RET

GVA-Guillaume commented 1 year ago

I that steps I am not able to see a cluster, each node are operated as leaders, do I have to create a specific file ?

And not able to launch vault web ui

GVA-Guillaume commented 1 year ago

AUTOPILOT : unable to start

hypothesis 1 (start with only vault_1) 👍

1 root 0:00 vault server -config=...1.hcl

sh run_all.sh Active Internet connections (only servers)

[...1] Creating configuration

creating ...-${}1.hcl [...2] Creating configuration
creating ...2.hcl
creating ...raft-${}2 [...3] Creating configuration
creating ...2.hcl
creating ...3

...1 starting Vault server @ ...

...1 initializing and capturing the unseal key and root token

...1 Root token: ---

...1 unsealing and logging in Key Value

Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.13.3 Build Date 2023-06-06T18:12:37Z Storage Type file Cluster Name ... Cluster ID .... HA Enabled false Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key Value

token --- token_accessor --- token_duration ∞ token_renewable false token_policies --- identity_policies [] policies ---

[dzisizasekret-vs1] enabling the transit secret engine and creating a key to auto-unseal vault cluster Success! Enabled the transit secrets engine at: transit/ Success! Data written to: transit/keys/unseal_key

[....2] starting Vault server @ --- Using [....1] root token (---) to retrieve transit key for auto-unseal

[...2] initializing and capturing the recovery key and root token Get "https:/....1:8200/v1/sys/seal-status": dial tcp ...1:8200: connect: connection refused

[....3] starting Vault server @ --- Using [---] root token (---) to retrieve transit key for auto-unseal

hypothesis 2 : start with vault 1 & 2

ps

1 root 0:00 vault server -config=/etc/vault/config-dzisizasekret-vs1.hcl 36 root 0:00 vault server -config=/etc/vault/config-dzisizasekret-vs2.hcl

[...1] starting Vault server @ ....

[...1] initializing and capturing the unseal key and root token

[...1] Unseal key:.... [...1] Root token: ....

[...1] unsealing and logging in Key Value

Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.13.3 Build Date ... Storage Type file Cluster Name .... Cluster ID ... HA Enabled false Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key Value

token .... token_accessor ..... token_duration ∞ token_renewable false token_policies .... identity_policies [] policies ....

[sv1] enabling the transit secret engine and creating a key to auto-unseal vault cluster Success! Enabled the transit secrets engine at: transit/ Success! Data written to: transit/keys/unseal_key

[....2] starting Vault server @ https://... Using [....1] ... token (...) to retrieve transit key for auto-unseal

[...2] initializing and capturing the recovery key and root token ... [INFO] core: security barrier not initialized ... [INFO] core: seal configuration missing, not initialized Error initializing: Error making API request.

URL: ... Code: 400. Errors:

parameters recovery_shares,recovery_threshold not applicable to seal type shamir

[....3] starting Vault server @... Using [...1] root token (....) to retrieve transit key for auto-unseal

[...4] starting Vault server @... Using [...1] root token (...) to retrieve transit key for auto-unseal

yhyakuna commented 1 year ago

I that steps I am not able to see a cluster, each node are operated as leaders, do I have to create a specific file ?

And not able to launch vault web ui

What you have is 3 single-node clusters instead of 1 cluster with 3 nodes because the vault operator raft join command failed, and you initialized them separately.

If vault_3 and vault_4 were able to joined the cluster successfully, you didn't have to initialize because the follower nodes use the same root token and recovery keys as its leader (vault_2).

Since you initialized vault_3 and vault_4, they became the leader (at this point, they are a single-node cluster and ready for other nodes to join).

The UI should work though... You need to use the returned root token for each server (cannot use the root token generated for vault_2).

GVA-Guillaume commented 1 year ago

thank you Yoko, after investigation it seems a problem of certificate authentication in the joining process

i have to know where can I change something with a -tls-skip-verify

maybe in storage "raft" { stanza} .... no idea

this is nice to get your assistance

LOG INFO :

from v2 : [INFO] http: TLS handshake error from 127.0.0.1:53316: remote error: tls: bad certificate from vx : core: failed to get raft challenge: leader_addr=https://127.0.0.1:8200 error="error during raft bootstrap init call: Put \"https://127.0.0.1:8200/v1/sys/storage/raft/bootstrap/challenge\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

GVA-Guillaume commented 1 year ago

Cool, I updated the retry join with some informations and your script works like a charm 👍

update info

storage "raft {

retry_join {

}

retry_join { leader_api_addr = "https://vault_2-fqdn-or-vault_2-ip:8200" leader_tls_servername = "vault_2-name" leader_ca_cert_file = "..." leader_client_cert_file = "..." leader_client_key_file = "..." }

}

TAKE AWAY :

Solving problem :

1 - server have to be bound with basic configuration (not in dev mode, because you need to specify each ip address) 2 - each server must be updated with an export of their ip address export fqdn-or-ipserver1 for server 1, etc ... 3 - each retry_join { ... } of storage "raft" {...} stanza have to be updated with tls leader information as above.

Regards, thanks for Yoko assistance and wonderful script, I have to work on ui issue

GVA-Guillaume commented 1 year ago

I that steps I am not able to see a cluster, each node are operated as leaders, do I have to create a specific file ? And not able to launch vault web ui

What you have is 3 single-node clusters instead of 1 cluster with 3 nodes because the vault operator raft join command failed, and you initialized them separately.

If vault_3 and vault_4 were able to joined the cluster successfully, you didn't have to initialize because the follower nodes use the same root token and recovery keys as its leader (vault_2).

Since you initialized vault_3 and vault_4, they became the leader (at this point, they are a single-node cluster and ready for other nodes to join).

The UI should work though... You need to use the returned root token for each server (cannot use the root token generated for vault_2).

Hi Yoko, thanks for replied, Where/How I have to use it ?

GVA-Guillaume commented 1 year ago

Hi Yoko,

My configuration 'is doing well..." ?? :

cd /vault/data && cp -a ../certs/.sh . && chown vault .sh && sh run_all.sh && netstat -plnt && export VAULT_ADDR="https://127.0.0.1:8200 " && vault operator raft list-peers

[vault_1] Creating configuration

creating /papth/to/config-vault_1.hcl [vault_2] Creating configuration
creating /papth/to/config-vault_2.hcl
creating /papth/to/raft-vault_2 [vault_3] Creating configuration
creating /papth/to/config-vault_3.hcl
creating /papth/to/raft-vault_3 [vault_4] Creating configuration
creating /papth/to/config-vault_4.hcl
creating /papth/to/raft-vault_4 [vault_5] Creating configuration
creating /papth/to/config-vault_5.hcl
creating /papth/to/raft-vault_5 [vault_6] Creating configuration
creating /papth/to/config-vault_6.hcl
creating /papth/to/raft-vault_6 [vault_7] Creating configuration
creating /papth/to/config-vault_7.hcl
creating /papth/to/raft-vault_7

[vault_1] starting Vault server @ https://127.0.0.1:8100

[vault_1] initializing and capturing the unseal key and root token

[vault_1] Unseal key: .... (NOT EXPOSED) [vault_1] Root token: hvs... (NOT EXPOSED)

[vault_1] unsealing and logging in Key Value

Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.14.0 Build Date 2023-06-19T11:40:23Z Storage Type file Cluster Name vault-cluster-f893bb60 Cluster ID 92600673-d741-713f-edea-168c094d0f7a HA Enabled false Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key Value

token hvs.secret... token_accessor ... token_duration ∞ token_renewable false token_policies ["root"] identity_policies [] policies ["root"]

[vault_1] enabling the transit secret engine and creating a key to auto-unseal vault cluster Success! Enabled the transit secrets engine at: transit/ Key Value

allow_plaintext_backup false auto_rotate_period 0s deletion_allowed false derived false exportable false imported_key false keys map[1:1688368670] latest_version 1 min_available_version 0 min_decryption_version 1 min_encryption_version 0 name unseal_key supports_decryption true supports_derivation true supports_encryption true supports_signing false type aes256-gcm96

[vault_2] starting Vault server @ https://127.0.0.1:8200 Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal

[vault_2] initializing and capturing the recovery key and root token

[vault_2] Recovery key: recoverykey... [vault_2] Root token: hvs.2....

[vault_2] waiting to finish post-unseal setup (15 seconds)

[vault_2] logging in and enabling the KV secrets engine Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token.

Key Value

token hvs.2.... token_accessor szhGLXJ1YQ1edIoK7VqnONF5 token_duration ∞ token_renewable false token_policies ["root"] identity_policies [] policies ["root"] Success! Enabled the kv-v2 secrets engine at: kv/

[vault_2] storing secret 'kv/apikey' to demonstrate snapshot and recovery methods = Secret Path = kv/data/apikey

======= Metadata ======= Key Value

created_time 2023-07-03T07:18:27.995701068Z custom_metadata deletion_time n/a destroyed false version 1 = Secret Path = kv/data/apikey

======= Metadata ======= Key Value

created_time 2023-07-03T07:18:27.995701068Z custom_metadata deletion_time n/a destroyed false version 1

===== Data ===== Key Value

webapp ABB39KKPTWOR832JGNLS02

[vault_3] starting Vault server @ https://127.0.0.1:8300 Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal

[vault_4] starting Vault server @ https://127.0.0.1:8400 Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal

[vault_5] starting Vault server @ https://127.0.0.1:8500 Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal

[vault_6] starting Vault server @ https://127.0.0.1:8600 Using [vault_1] root token (hvs....) to retrieve transit key for auto-unseal Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:35445 0.0.0.0: LISTEN - tcp 0 0 127.0.0.1:8500 0.0.0.0: LISTEN 347/vault tcp 0 0 127.0.0.1:8600 0.0.0.0: LISTEN 372/vault tcp 0 0 127.0.0.1:8201 0.0.0.0: LISTEN 74/vault tcp 0 0 127.0.0.1:8200 0.0.0.0: LISTEN 183/vault tcp 0 0 127.0.0.1:8300 0.0.0.0: LISTEN 294/vault tcp 0 0 127.0.0.1:8400 0.0.0.0: LISTEN 321/vault tcp 0 0 127.0.0.1:8100 0.0.0.0: LISTEN 74/vault tcp 0 0 :::8100 :::* LISTEN 1/vault Node Address State Voter

vault_2 127.0.0.1:8201 leader true

But concern the web ui, I have a bad strange page, and in anyway, all the provided token root vault_1 vault_2

does not works and I have no idea why and what I have to use

hashicorp / learn-vault-raft

unable to setup vault_2, vault_3, vault_4 #9

Start vault server locally for the purposes of integration tests.

Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.13.3 Build Date --- Storage Type file Cluster Name --- Cluster ID --- HA Enabled false /vault/data #

export VAULT_ADDR="---:8200"

vault status

vault operator raft list-peers

export VAULT_ADDR="---8200"

vault status

Recovery Seal Type transit Initialized false Sealed true Total Recovery Shares 0 Threshold 0 Unseal Progress 0/0 Unseal Nonce n/a Version 1.13.3 Build Date ... Storage Type raft HA Enabled true /vault/data #