search

wescale / hashistack

Ansible collections for HashiCorp Vault-Consul-Nomad platform automation
MIT License
59 stars 32 forks source link

Consul Connect Vault Sidecar Listening health check fails with "Connection refused" #65

Closed makp0 closed 1 year ago

makp0 commented 1 year ago

archi: mono provider: scaleway HEAD SHA: 10e48969983ba55720dc767b5aa69874938bb165

image

Sorry, cannot extract more logs because of #64

Ronan-WeScale commented 1 year ago

Hi @makp0
This problem aren't linked with nomad envoy This proxy are using default consul proxy, you can check the status of the "sidecarvault" systemd service on master node and post here the output. Thanks

makp0 commented 1 year ago

@Ronan-WeScale 

░░ The job identifier is 605.
Apr 04 15:46:14 devtwo-mono consul[8261]: ==> Consul Connect proxy starting...
Apr 04 15:46:14 devtwo-mono consul[8261]:     Configuration mode: Agent API
Apr 04 15:46:14 devtwo-mono consul[8261]:         Sidecar for ID: vault
Apr 04 15:46:14 devtwo-mono consul[8261]:               Proxy ID: vault-sidecar-proxy
Apr 04 15:46:14 devtwo-mono consul[8261]: ==> Log data will now stream in as it occurs:
Apr 04 15:46:14 devtwo-mono consul[8261]:     2023-04-04T15:46:14.878+0200 [INFO]  proxy: Proxy loaded config and ready to serve
Apr 04 15:46:14 devtwo-mono consul[8261]:     2023-04-04T15:46:14.878+0200 [INFO]  proxy: Parsed TLS identity: uri=spiffe://55462d1f-d271-4250-3ad4-33cc9af224b2.consul/ns/default/dc/devtwo/svc/vault
Apr 04 15:46:14 devtwo-mono consul[8261]:     2023-04-04T15:46:14.878+0200 [INFO]  proxy: Starting listener: listener="public listener" bind_addr=0.0.0.0:22000
Apr 04 15:46:19 devtwo-mono consul[8261]: ==> Consul Connect proxy shutdown
Apr 04 15:46:19 devtwo-mono systemd[1]: Stopping Consul ingress gateway for vault...
░░ Subject: A stop job for unit sidecarvault.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit sidecarvault.service has begun execution.
░░ 
░░ The job identifier is 741.
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit sidecarvault.service has successfully entered the 'dead' state.
Apr 04 15:46:19 devtwo-mono systemd[1]: Stopped Consul ingress gateway for vault.
░░ Subject: A stop job for unit sidecarvault.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit sidecarvault.service has finished.
░░ 
░░ The job identifier is 741 and the job result is done.
Apr 04 15:46:19 devtwo-mono systemd[1]: Started Consul ingress gateway for vault.
░░ Subject: A start job for unit sidecarvault.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit sidecarvault.service has finished successfully.
░░ 
░░ The job identifier is 741.
Apr 04 15:46:19 devtwo-mono consul[8433]: ==> Consul Connect proxy starting...
Apr 04 15:46:19 devtwo-mono consul[8433]: ==> Error preparing configuration: Failed looking up sidecar proxy info for vault: Get "https://devtwo-mono.devtwo.tradent.link:8501/v1/agent/services": dial tcp 127.0.0.1:8501: connect: connection refused
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit sidecarvault.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit sidecarvault.service has entered the 'failed' state with result 'exit-code'.
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Scheduled restart job, restart counter is at 1.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit sidecarvault.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 04 15:46:19 devtwo-mono systemd[1]: Stopped Consul ingress gateway for vault.
░░ Subject: A stop job for unit sidecarvault.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit sidecarvault.service has finished.
░░ 
░░ The job identifier is 744 and the job result is done.
Apr 04 15:46:19 devtwo-mono systemd[1]: Started Consul ingress gateway for vault.
░░ Subject: A start job for unit sidecarvault.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit sidecarvault.service has finished successfully.
░░ 
░░ The job identifier is 744.
Apr 04 15:46:19 devtwo-mono consul[8469]: ==> Consul Connect proxy starting...
Apr 04 15:46:19 devtwo-mono consul[8469]: ==> Error preparing configuration: Failed looking up sidecar proxy info for vault: Unexpected response code: 403 (ACL not found)
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit sidecarvault.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit sidecarvault.service has entered the 'failed' state with result 'exit-code'.
Apr 04 15:46:19 devtwo-mono systemd[1]: sidecarvault.service: Scheduled restart job, restart counter is at 2.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit sidecarvault.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 04 15:46:19 devtwo-mono systemd[1]: Stopped Consul ingress gateway for vault.
░░ Subject: A stop job for unit sidecarvault.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit sidecarvault.service has finished.
░░ 
░░ The job identifier is 813 and the job result is done.
Apr 04 15:46:19 devtwo-mono systemd[1]: Started Consul ingress gateway for vault.
░░ Subject: A start job for unit sidecarvault.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit sidecarvault.service has finished successfully.
░░ 
░░ The job identifier is 813.
Apr 04 15:46:19 devtwo-mono consul[8486]: ==> Consul Connect proxy starting...
Apr 04 15:46:20 devtwo-mono consul[8486]: ==> Error preparing configuration: Failed looking up sidecar proxy info for vault: Unexpected response code: 403 (ACL not found)
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit sidecarvault.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit sidecarvault.service has entered the 'failed' state with result 'exit-code'.
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Scheduled restart job, restart counter is at 3.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit sidecarvault.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 04 15:46:20 devtwo-mono systemd[1]: Stopped Consul ingress gateway for vault.
░░ Subject: A stop job for unit sidecarvault.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit sidecarvault.service has finished.
░░ 
░░ The job identifier is 882 and the job result is done.
Apr 04 15:46:20 devtwo-mono systemd[1]: Started Consul ingress gateway for vault.
░░ Subject: A start job for unit sidecarvault.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit sidecarvault.service has finished successfully.
░░ 
░░ The job identifier is 882.
Apr 04 15:46:20 devtwo-mono consul[8495]: ==> Consul Connect proxy starting...
Apr 04 15:46:20 devtwo-mono consul[8495]: ==> Error preparing configuration: Failed looking up sidecar proxy info for vault: Unexpected response code: 403 (ACL not found)
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit sidecarvault.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit sidecarvault.service has entered the 'failed' state with result 'exit-code'.
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Scheduled restart job, restart counter is at 4.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ Automatic restarting of the unit sidecarvault.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Apr 04 15:46:20 devtwo-mono systemd[1]: Stopped Consul ingress gateway for vault.
░░ Subject: A stop job for unit sidecarvault.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A stop job for unit sidecarvault.service has finished.
░░ 
░░ The job identifier is 951 and the job result is done.
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Start request repeated too quickly.
Apr 04 15:46:20 devtwo-mono systemd[1]: sidecarvault.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit sidecarvault.service has entered the 'failed' state with result 'exit-code'.
Apr 04 15:46:20 devtwo-mono systemd[1]: Failed to start Consul ingress gateway for vault.
░░ Subject: A start job for unit sidecarvault.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit sidecarvault.service has finished with a failure.
░░ 
░░ The job identifier is 951 and the job result is failed.
Ronan-WeScale commented 1 year ago

I can reproduce this.
Restart the service work but I will find the root cause.

aurelienmaury commented 1 year ago

Was caused by an error in the service initial start order choregraphy. Should be fixed with v0.9.0 (released). Reopen if not.