Open Overdrivr opened 7 years ago
The push is rejected because of the Internal Server Error
that happened probably when pushing the built image into registry (I'm not sure if a successful push to registry is logged). This is desired behavior because if the refs get pushed and the image cannot be successfully started in a container it will leave you with a bogus repository state which is not currently deployed. So, failed to push some refs
is really your friend here and your enemies are probably hiding in the logs.
In short, you should inspect deis-builder
and also deis-registry
logs and work your way from there. You can also enable debug mode if you see nothing obvious in the logs.
Thanks for the reply. The registry pod if filled with info messages, is this the default behavior ?
10.12.0.1 - - [15/Jun/2017:05:53:12 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:12.050694806Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=fe682b3f-5756-47d9-8e86-cf19f5d1a9e3 http.request.method=GET http.request.remoteaddr="10.12.0.1:35296" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=8.927239ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0
10.12.0.1 - - [15/Jun/2017:05:53:12 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:22.041218766Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=710364ac-dde4-4bb2-9723-cd2d383ca686 http.request.method=GET http.request.remoteaddr="10.12.0.1:35330" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.648287ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0
10.12.0.1 - - [15/Jun/2017:05:53:22 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:22.04325269Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=80e1cbd3-52e5-405c-ab1c-928957f731de http.request.method=GET http.request.remoteaddr="10.12.0.1:35328" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.529342ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0
The deis-builder log is maybe more interesting
$ kubectl --namespace=deis logs deis-builder-1134541883-4mvsv
2017/06/14 04:37:38 Starting health check server on port 8092
2017/06/14 04:37:38 Starting deleted app cleaner
2017/06/14 04:37:38 Starting SSH server on 0.0.0.0:2223
Listening on 0.0.0.0:2223
Accepting new connections.
---> ---> 2017/06/14 04:37:51 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:01 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:11 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:21 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:31 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:41 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:51 Readinesscheck endpoint timed out after 10s
2017/06/14 04:39:01 Readinesscheck endpoint timed out after 10s
Accepted connection.
Starting ssh authentication
Channel type: session
receiving git repo name: app-dispatcher.git, operation: git-receive-pack, fingerprint: 84:a8:7e:6a:e8:1b:20:34:4b:29:a7:f3:14:5a:e3:42, user: gitlabci
creating repo directory /home/git/app-dispatcher.git
writing pre-receive hook under /home/git/app-dispatcher.git
git-shell -c git-receive-pack 'app-dispatcher.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
Accepted connection.
Starting ssh authentication
Channel type: session
receiving git repo name: app-dispatcher.git, operation: git-receive-pack, fingerprint: 84:a8:7e:6a:e8:1b:20:34:4b:29:a7:f3:14:5a:e3:42, user: gitlabci
creating repo directory /home/git/app-dispatcher.git
writing pre-receive hook under /home/git/app-dispatcher.git
git-shell -c git-receive-pack 'app-dispatcher.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
Accepted connection.
---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> [ERROR] Failed handshake: EOF
Accepted connection.
Accepted connection.
---> ---> [ERROR] Failed handshake: EOF
[ERROR] Failed handshake: EOF
Accepted connection.
Accepted connection.
---> ---> [ERROR] Failed handshake: EOF
Accepted connection.
---> [ERROR] Failed handshake: EOF
Any idea what is causing the handshake to fail ?
This handshake failures are just the healthcheck probe. It's a red herring. What you want to be looking at is the registry logs as that's where the 500 error came from.
Here you go, registry logs with HTTP 500 error
$ kubectl --namespace=deis logs deis-registry-2623437609-x1fcp | grep " 500 "
10.12.0.7 - - [15/Jun/2017:06:15:15 +0000] "PUT /v2/app-dispatcher/blobs/uploads/a0a37674-7680-4e64-a809-05551e124f76?_state=9rUDYtxw_nxrKtVQVQK20EXSMhdleDG4Ci5BbZEZOjt7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiYTBhMzc2NzQtNzY4MC00ZTY0LWE4MDktMDU1NTFlMTI0Zjc2IiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNDozOFoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:20 +0000] "PUT /v2/app-dispatcher/blobs/uploads/05e4cd44-be93-459c-99a4-cf7d77f6519a?_state=omTy6YEGZqDnBs2fqLPOXwWbg6PTtpBzGAh8eVtsIMh7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMDVlNGNkNDQtYmU5My00NTljLTk5YTQtY2Y3ZDc3ZjY1MTlhIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNDozOVoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:54 +0000] "PUT /v2/app-dispatcher/blobs/uploads/f1cb1743-43c3-478b-a4b8-27c4ad3e4a8b?_state=xhjHl05OxmV09G4oskfU0CIC7gsbaczC5Evz1iX36jx7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZjFjYjE3NDMtNDNjMy00NzhiLWE0YjgtMjdjNGFkM2U0YThiIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNToyMVoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/d3fbfd1a-eb57-4cf8-a55b-fb26e0d38ac6?_state=qd0QZx4wanTPwiDZ7aVnXpr1vxR7fMZ_gHTEdmC8b1N7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZDNmYmZkMWEtZWI1Ny00Y2Y4LWE1NWItZmIyNmUwZDM4YWM2IiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNToyNloifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:56 +0000] "PUT /v2/app-dispatcher/blobs/uploads/2a7e85ff-768e-4407-b2e2-810261e0e5d7?_state=2mDRWSFFneGaJL4styC3IQ7Cd1bjnpw5FcOUZpiLUNB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMmE3ZTg1ZmYtNzY4ZS00NDA3LWIyZTItODEwMjYxZTBlNWQ3IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTQ6MzhaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:47 +0000] "PUT /v2/app-dispatcher/blobs/uploads/4213afc5-bd0c-4e89-8545-c7c62d97c973?_state=33N3ehczUMDazWB9sxMH7wN8ftYcLOL4hWyps1L_3P17Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNDIxM2FmYzUtYmQwYy00ZTg5LTg1NDUtYzdjNjJkOTdjOTczIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNjowNVoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:48 +0000] "PUT /v2/app-dispatcher/blobs/uploads/2670a319-80f6-4b31-9c60-8498fb2219c0?_state=wAgGDjU4XQc-K3tBivJ11HF0bi2rgxXNaocIxf5T9pB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMjY3MGEzMTktODBmNi00YjMxLTljNjAtODQ5OGZiMjIxOWMwIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNjowNloifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/46b17962-9a71-4e45-ae03-23b207184548?_state=aunuxweYQROhndJe4ZqAsmnCBLyWbq6r1P-DKY1p-nR7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNDZiMTc5NjItOWE3MS00ZTQ1LWFlMDMtMjNiMjA3MTg0NTQ4IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTY6MDJaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:36 +0000] "PUT /v2/app-dispatcher/blobs/uploads/3a644e41-d3c2-44a8-9596-ba42359aa71c?_state=1WfuVKBc2gc3qjb0siH7Lcbf2asHkHjdBDjYOh8T_ZB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiM2E2NDRlNDEtZDNjMi00NGE4LTk1OTYtYmE0MjM1OWFhNzFjIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzowMloifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:44 +0000] "PUT /v2/app-dispatcher/blobs/uploads/787c7538-1a4e-439c-ad51-a0658ef87e74?_state=b7y6Wvq1uz0pKW1PluUWMABckM3NtIRP6X9zu0IghcZ7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNzg3Yzc1MzgtMWE0ZS00MzljLWFkNTEtYTA2NThlZjg3ZTc0IiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzowNFoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/4b5c16e3-2c1e-4264-a6f4-8cc396affbac?_state=vM16auRY8mezk2pPhMIHKLIzgzVhjV43KRhxQihhYid7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNGI1YzE2ZTMtMmMxZS00MjY0LWE2ZjQtOGNjMzk2YWZmYmFjIiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTc6MDVaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:14 +0000] "PUT /v2/app-dispatcher/blobs/uploads/b3928bcf-36e5-4000-886c-7288fe3389c0?_state=Uhwuv4nSaQCI_MUn8Eeqc8q8OzO1KM74-_YwRZ7JER57Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiYjM5MjhiY2YtMzZlNS00MDAwLTg4NmMtNzI4OGZlMzM4OWMwIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzo1N1oifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:38 +0000] "PUT /v2/app-dispatcher/blobs/uploads/106c9553-86b6-4dfa-8b22-14bbf712b80f?_state=-5FSF76b4zMHCcWh2qKOfujnGVMTYZYp8dC7c73EFuB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMTA2Yzk1NTMtODZiNi00ZGZhLThiMjItMTRiYmY3MTJiODBmIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxODowNVoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:49 +0000] "PUT /v2/app-dispatcher/blobs/uploads/f5c80612-d1a4-44b7-9f1a-75a6096718ed?_state=BS_ZRX1Sv0kdnvr25YbNzOG8-f7jjJef2aTRlFKWf317Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZjVjODA2MTItZDFhNC00NGI3LTlmMWEtNzVhNjA5NjcxOGVkIiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTg6MTBaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:19:38 +0000] "PUT /v2/app-dispatcher/blobs/uploads/0592b1a6-5234-458d-9c8a-fa324a050517?_state=iNYDPE62ScgvCj825K8D-xthNqd_YffLVV2JfTzoQRd7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMDU5MmIxYTYtNTIzNC00NThkLTljOGEtZmEzMjRhMDUwNTE3IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTk6MDlaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
Sounds like it's having a hard time uploading images to the backing storage driver as it's barking during PUT operations. I'd check your object storage logs and see if it's crashing, or perhaps your cluster is misconfigured and the registry is unable to upload blobs to object storage.
$ kubectl --namespace=deis logs deis-minio-3370481340-8nlfx
minio server /home/minio/
2017/06/14 04:37:02 starting Minio server
2017/06/14 04:37:02 starting health check server on 0.0.0.0:8082
Migration from version ‘2’ to ‘3’ completed successfully.
Migration from version ‘3’ to ‘4’ completed successfully.
AccessKey: **************** SecretKey: ******************** Region: us-east-1
Minio Object Storage:
http://127.0.0.1:9000
http://**.**.**.**:9000
Minio Browser:
http://127.0.0.1:9000
http://**.**.**.**:9000
To configure Minio Client:
$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc
$ ./mc config host add myminio http://127.0.0.1:9000 **************** *************************
It's weird that minio is using region us-east-1, because my cluster is located in europe-west1-b. Could this be the problem ?
Minio fakes out and simulates that it's in us-east-1 to mimic the S3 API. It doesn't accurately reflect where it's deployed (in the cluster).
+1
Any updates on this? Got exactly the same problem. I have bare metal kubernetes cluster and Deis workflow on top of that. Sample applications are fine, but it doesn't work with my own.
Hi Guys any update on this Im having the same issue
I wasn't sure how to debug this further, so no updates so far. I wasn't seeing anything in my object storage logs.
Having watched this carefully (I know, great way to debug things) I noticed something interesting-It looks like some of the minio requests are going through, but then the connection gets dropped somewhere (the object is too big?) and minio barfs and stops taking any more data.
[ deis registry logs ]
100.96.6.18 - - [18/Aug/2017:16:40:25 +0000] "POST /v2/aws-dx-selfjoin/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
time="2017-08-18T16:40:27.09674817Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="100.96.6.23:5000" http.request.id=cd874d81-d36b-48ec-8455-4528504f88f8 http.request.method=GET http.request.remoteaddr="100.96.6.1:53204" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" h
ttp.response.contenttype="application/json; charset=utf-8" http.response.duration=1.746968ms http.response.status=200 http.response.written=2 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.1 - - [18/Aug/2017:16:40:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-08-18T16:40:27.09757101Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="100.96.6.23:5000" http.request.id=72f0a983-963a-499e-89d6-d30c1b29fab7 http.request.method=GET http.request.remoteaddr="100.96.6.1:53202" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" h
ttp.response.contenttype="application/json; charset=utf-8" http.response.duration=1.58666ms http.response.status=200 http.response.written=2 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.1 - - [18/Aug/2017:16:40:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-08-18T16:40:35.222805315Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=6aaf4e8a-dc20-4dcb-a9da-9f2df76db945 http.request.method=PATCH http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-
9ab733dd225f?_state=0CLtxmCoH9cYWnDAEmjo9mWBMC3_Q7TX85-MY80Au4B7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDE2OjQwOjI1LjIxODg4NzY5MVoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4
.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.response.duration=1.380391587s http.response.status=202 http.response.written=0 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.18 - - [18/Aug/2017:16:40:33 +0000] "PATCH /v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=0CLtxmCoH9cYWnDAEmjo9mWBMC3_Q7TX85-MY80Au4B7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDE2
OjQwOjI1LjIxODg4NzY5MVoifQ%3D%3D HTTP/1.0" 202 0 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
time="2017-08-18T16:40:35.378123268Z" level=error msg="unknown error completing upload: s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: DZFQC2IC58RHYLEY" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3edb847-d84f-4394-b247-0d03d594ce4f http.request.method=
PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjA
xNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=regis
try vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version=v2.6.0
time="2017-08-18T16:40:35.378179784Z" level=error msg="error canceling upload after error: already closed" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3edb847-d84f-4394-b247-0d03d594ce4f http.request.method=PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/u
ploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f93
0bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version
=v2.6.0
time="2017-08-18T16:40:35.383839588Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: DZFQC2IC58RHYLEY" err.message="unknown error" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3
edb847-d84f-4394-b247-0d03d594ce4f http.request.method=PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMz
ZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.re
sponse.contenttype="application/json; charset=utf-8" http.response.duration=110.023913ms http.response.status=500 http.response.written=117 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version=v2.6.0
100.96.6.18 - - [18/Aug/2017:16:40:35 +0000] "PUT /v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC
0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845 HTTP/1.0" 500 117 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
Hey, guys, you can try with a newer minio image - just edit the minio deployment. Rereading this made me vaguely remember that I had some issues with the minio version that deis deploys by default.
@dumyan What new minio image are you referring to?
@battlemidget the OP is using minio as a storage backend for the registry. The default image for deis-minio
is pretty outdated (or at least was outdated few months ago when I used minio as a backend storage). My suggestion is to change the minio image in the deis-minio
deployment (you can try this image for example minio/minio:RELEASE.2017-07-24T18-27-35Z
)
@dumyan thanks im hitting this same issue, ill give that a try
@dumyan Ok, so not exactly the same issue as this one it seems. I am using off cluster storage (s3) and our distribution of kubernetes uses flannel cni so I just needed to enable that.
FYI, we are heavily promoting Deis workflow as part of our core addons for our Canonical Distribution of Kubernetes, my first blog post on it is here http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/#addons, and the application is https://conjure-up.io. cc @bacongobbler
Thanks for the help!
@battlemidget you may want to read our latest blog post. https://deis.com/blog/2017/deis-workflow-final-release/
@bacongobbler well that is disappointing :(
This issue was moved to teamhephy/workflow#14
I'm trying to deploy from Gitlab CI to Deis using buildpack ( I cannot use Docker-based because of #823 )
Authentication to deis from gitlab-ci runner works fine, code is built successfully. But at the end of the deployment, I get the following error message.
My cluster is GKE 3 node n1-standard1, with 50Go of storage each.
All pods are properly running
How can I debug this ?