Builder restarts over and over when using s3

[krancour]

Config:

I added the following env vars:

• DEIS_OUTSIDE_STORAGE: s3.amazonaws.com
• BUCKET: kents-builder

I also updated the credentials in deis-minio-secret-user.yaml (the minio-user secret) with base64 encoded access key and secret key.

Logs:

$ k logs deis-builder-vyo3y --previous
2016/03/18 01:27:55 Running in debug mode
2016/03/18 01:27:55 Starting health check server on port 8092
2016/03/18 01:27:55 Starting deleted app cleaner
2016/03/18 01:27:55 Starting SSH server on 0.0.0.0:2223
[debug] Generating ssh keys for sshd
[debug] Parsing ssh host keys
[info] Parsed host key /etc/ssh/ssh_host_rsa_key.
[info] Parsed host key /etc/ssh/ssh_host_dsa_key.
[info] Parsed host key /etc/ssh/ssh_host_ecdsa_key.
[debug] Name: EXTERNAL_PORT, Val: 2223
[info] Added hostkey.
[info] Added hostkey.
[info] Added hostkey.
[info] Listening on 0.0.0.0:2223
[info] Accepting new connections.
[info] Checking closer.
[info] Builder is running.
Healthcheck endpoint timed out after 2s
Healthcheck endpoint timed out after 2s
Healthcheck endpoint timed out after 2s
Healthcheck endpoint timed out after 2s
Healthcheck endpoint timed out after 2s

[krancour]

Oddly, on a brand new k8s cluster, I am no longer able to reproduce, so I will close this and re-open it later if need be.

[jschneiderhan]

FWIW I was having the same issue. Ended up tracking it down to sporadic slow S3 requests. My current workaround is to bump the liveness and readiness checks for my deis-builder Deployment:

kubectl --namespace=deis patch deployment deis-builder --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/timeoutSeconds", "value":10}]'

kubectl --namespace=deis patch deployment deis-builder --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/readinessProbe/timeoutSeconds", "value":10}]'