Readiness probe failed: HTTP probe failed with statuscode: 503

[kaushique]

Hi Team, i've setup gitlab ci pipeline exporter using helm chart way , but when i see the pod events it says "Readiness probe failed: HTTP probe failed with statuscode: 503" Because of this the endpoint is not coming and i'm unable to view metrics at /metrics at port 8080 I've made sure no other app service is exposed at 8080 . it's reading the data from gitlab but status of the pod remains 0/1 running how do i overcome it ? i tried increasing the initial delay and failure threshold for readiness probe but no luck

Any help is appreciated and Thanks in advance

[mvisonneau]

👋 can you share the logs of the pods when this situation occurs? Preferably with log.level set to debug or trace?

[hexa2k9]

We experienced the same and after deploying the Exporter like a million times through Helm with different options this worked for us (for the records, we've deployed GitLab EE through Omnibus on EL7):

[...]
config:
  gitlab:
    enable_tls_verify: true
    enable_health_check: true
    maximum_requests_per_second: 60
    token: <token>
    url: https://git.host.name/
[...]

We had config.gitlab.url in our values.yaml without the trailing Slash, but it seems to be required.

So mind the trailing Slash in the GitLab URL, that fixed it for us.

[mvisonneau]

Interesting, I believe that by adding a trailing / on the GitLab URL & not setting health URL the you are hitting a redirect which likely get's you to the login page with a 200.

Have you checked the output of https://git.host.name/-/health ?

[hexa2k9]

This is what various requests gave me

# curl -sS -i https://git.host.name/-/health
HTTP/2 404
server: nginx
[...snipp...]

Not Found

# curl -sS -i https://git.host.name/-/health?token={{config.gitlab.token}}
HTTP/2 404
server: nginx
[...snipp...]

Not Found

# curl -sS -i https://git.host.name/-/health?token={{access token from /admin/health_check}}
HTTP/2 404
server: nginx
[...snipp...]

Not Found

# curl -sS -I https://git.host.name/-/readiness?token={{config.gitlab.roken}}
HTTP/2 404
server: nginx
[...snipp...]

{"error":"Not Found","status":404}

# curl -sS -i https://git.host.name/-/readiness?token={{access token from /admin/health_check}}
HTTP/2 200
server: nginx
[...snipp...]

{"status":"ok","master_check":[{"status":"ok"}]}

[KnKay]

Can confirm, the trailing slash is the magic!

[MichaelSPaik]

I'm experiencing the same issue when pulling metrics from gitlab.com. I've tested with multiple different configs, anyone have any suggestions or seen this too?

[martijnvanderkamp]

Thanks, this tip on the trailing / helped me as well!