Builder error on second push

slack commented 8 years ago

Pushing an app a second time ran into at least two issues:

git bucketname already exists
pod name deis-dockerbuilder already existed

example-dockerfile-http [master]$ g push deis master
Counting objects: 47, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (37/37), done.
Writing objects: 100% (47/47), 7.16 KiB | 0 bytes/s, done.
Total 47 (delta 6), reused 0 (delta 0)
Added host ‘http://10.3.0.133:9000’ successfully.
mc: <ERROR> Unable to make bucket ‘http://10.3.0.133:9000/git’. The requested bucket name is not available.
‘shadow-woodshed.tar.gz’ -> ‘http://10.3.0.133:9000/git/home/shadow-woodshed:git-3edec618/tar’
Total: 2.00 KB, Transferred: 2.00 KB, Speed: 801.59 KB/s
-----> stored tarfile in http://10.3.0.133:9000/git/home/shadow-woodshed:git-3edec618/tar
-----> creating builder pod in namespace deis
Error from server: error when creating "/etc/shadow-woodshed:git-3edec618.yaml": pods "deis-dockerbuilder" already exists
To ssh://git@a097f2098a2ce11e5a5a40272b026e4b-190952671.us-west-2.elb.amazonaws.com:2222/shadow-woodshed.git
 * [new branch]      master -> master

Also issues with deisci/dockerbuilder image

Failed to pull image "quay.io/deisci/dockerbuilder:v2-alpha": image pull failed for quay.io/deisci/dockerbuilder:v2-alpha, this may be because there are no credentials on this request.  details: (Error: Status 403 trying to pull repository deisci/dockerbuilder: "{\"error\": \"Permission Denied\"}")

slack commented 8 years ago

-rw-r--r-- 1 root root 1017 Dec 15 22:12 /etc/shadow-woodshed:git-3edec618.yaml
/ # cat /etc/shadow-woodshed:git-3edec618.yaml
apiVersion: v1
kind: Pod
metadata:
  name: deis-dockerbuilder
  labels:
    heritage: deis
    version: v2-alpha
spec:
  restartPolicy: Never
  containers:
    - name: deis-dockerbuilder
      imagePullPolicy: Always
      image: quay.io/deisci/dockerbuilder:v2-alpha
      env:
        - name: DEBUG
          value: "1"
        - name: TAR_URL
          value: http://10.3.0.133:9000/git/home/shadow-woodshed:git-3edec618/tar
        - name: IMG_NAME
          value: imagename
        - name: ACCESS_KEY_FILE
          value: /var/run/secrets/object/store/access_key
        - name: ACCESS_SECRET_FILE
          value: /var/run/secrets/object/store/access_secret
      volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-socket
        - name: minio-user
          mountPath: /var/run/secrets/object/store
          readOnly: true
  volumes:
    - name: minio-user
      secret:
        secretName: minio-user
    - name: docker-socket
      hostPath:
        path: /var/run/docker.sock
/ #

arschles commented 8 years ago

@slack two questions:

What repo are you git pushing? Builder thinks it's a dockerfile repo and is starting up a docker builder pod, which hasn't been built yet
I can't reproduce the "pod already exists" issue. Old deis-builder images will have name: deis-builder in the metadata, while new ones will successfully overwrite the name to include the git sha in it, making it unique. Can you post the output of kubectl --namespace=deis $DEIS_BUILDER_POD_NAME -- cat /etc/shadow-woodshed:git-3edec618.yaml so we can diagnose further?

arschles commented 8 years ago

Nevermind, ignore (2) I just saw you posted it. This may be running an older deis-builder image because the pod name is old. Can you try restarting it?

helm fetch deis/deis && kubectl --namespace=deis delete rc deis-builder && kubectl --namespace=deis create -f ~/.helm/workspace/charts/deis/manifests/deis-builder-rc.yaml

slack commented 8 years ago

Re 1) it was certainly a Dockerfile based application, my bad!

I reproduced with a buildpack app via deis/example-ruby-sinatra:

example-ruby-sinatra [master]$ g push deis master
Counting objects: 98, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (50/50), done.
Writing objects: 100% (98/98), 21.58 KiB | 0 bytes/s, done.
Total 98 (delta 42), reused 98 (delta 42)
Added host ‘http://10.3.0.133:9000’ successfully.
mc: <ERROR> Unable to make bucket ‘http://10.3.0.133:9000/git’. The requested bucket name is not available.
‘gaslit-jamboree.tar.gz’ -> ‘http://10.3.0.133:9000/git/home/gaslit-jamboree:git-80e20346/tar’
Total: 2.05 KB, Transferred: 2.05 KB, Speed: 196.99 KB/s
-----> stored tarfile in http://10.3.0.133:9000/git/home/gaslit-jamboree:git-80e20346/tar
-----> creating builder pod in namespace deis
pod "gaslit-jamboree-git-80e20346" created
no file
no file
no file
no file
^CKilled by signal 2.

<<< TIME PASSES >>>

example-ruby-sinatra [master]$ g push deis master
Counting objects: 98, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (50/50), done.
Writing objects: 100% (98/98), 21.58 KiB | 0 bytes/s, done.
Total 98 (delta 42), reused 98 (delta 42)
Added host ‘http://10.3.0.133:9000’ successfully.
mc: <ERROR> Unable to make bucket ‘http://10.3.0.133:9000/git’. The requested bucket name is not available.
‘gaslit-jamboree.tar.gz’ -> ‘http://10.3.0.133:9000/git/home/gaslit-jamboree:git-80e20346/tar’
Total: 2.05 KB, Transferred: 2.05 KB, Speed: 667.31 KB/s
-----> stored tarfile in http://10.3.0.133:9000/git/home/gaslit-jamboree:git-80e20346/tar
-----> creating builder pod in namespace deis
Error from server: error when creating "/etc/gaslit-jamboree:git-80e20346.yaml": pods "gaslit-jamboree-git-80e20346" already exists
To ssh://git@a097f2098a2ce11e5a5a40272b026e4b-190952671.us-west-2.elb.amazonaws.com:2222/gaslit-jamboree.git
 * [new branch]      master -> master
example-ruby-sinatra [master]$

arschles commented 8 years ago

Thanks @slack. First, can you try making a change, committing it, and git push deis master again. I want to ensure that the name substitution is working correctly on your cluster.

slack commented 8 years ago

Replaced builder. Deis charts version:

deis [master]$ git rev-parse HEAD
a0ba86398a7e83d14a69f548fca0d24051081b44

Revision, commit, push:

example-ruby-sinatra [foo]$ g push deis master
Counting objects: 98, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (50/50), done.
Writing objects: 100% (98/98), 21.58 KiB | 0 bytes/s, done.
Total 98 (delta 42), reused 98 (delta 42)
mc: Configuration written to [/var/minio-conf/config.json]. Please update your access credentials.
mc: Successfully created ‘/var/minio-conf/share’.
mc: Initialized share uploads ‘/var/minio-conf/share/uploads.json’ file.
mc: Initialized share downloads ‘/var/minio-conf/share/downloads.json’ file.
Added host ‘http://10.3.0.133:9000’ successfully.
mc: <ERROR> Unable to make bucket ‘http://10.3.0.133:9000/git’. The requested bucket name is not available.
‘gaslit-jamboree.tar.gz’ -> ‘http://10.3.0.133:9000/git/home/gaslit-jamboree:git-80e20346/tar’
Total: 2.05 KB, Transferred: 2.05 KB, Speed: 732.00 KB/s
-----> stored tarfile in http://10.3.0.133:9000/git/home/gaslit-jamboree:git-80e20346/tar
-----> creating builder pod in namespace deis
Error from server: error when creating "/etc/gaslit-jamboree:git-80e20346.yaml": pods "gaslit-jamboree-git-80e20346" already exists
To ssh://git@a097f2098a2ce11e5a5a40272b026e4b-190952671.us-west-2.elb.amazonaws.com:2222/gaslit-jamboree.git
 * [new branch]      master -> master
example-ruby-sinatra [foo]$ g rev-parse HEAD
9b599549da6a00259ba189de010a5a7a9dbfedbb
example-ruby-sinatra [foo]$

arschles commented 8 years ago

also, can you post the log output of pod gaslit-jamboree-git-80e20346?

slack commented 8 years ago

example-ruby-sinatra [foo]$ k --namespace=deis logs gaslit-jamboree-git-80e20346 | gist -p
https://gist.github.com/34a1075b82db6e997c60

https://gist.github.com/34a1075b82db6e997c60

arschles commented 8 years ago

Ok, thanks. Here's what I believe is happening, both in the dockerfile and buildpack cases:

You git push your code, but git doesn't consider the push successful:
- In the Dockerfile case, the pod fails to launch completely
- In the buildpack case, the push never returns (because of https://github.com/deis/builder/issues/45)
Since git thinks the rev wasn't successfully pushed, it allows you to push the rev again. Builder starts builder pods (both buildpack and docker) based only on the rev, so it tries to start a pod with the same name as has already run, which results in the error.

Once #45 is fixed, this should go away for buildpack builds, but not for dockerfile builds. Leaving open as a result

slack commented 8 years ago

Makes sense.

Also seems like any time a build for a given sha is aborted from the client, we'll get in this state.

arschles commented 8 years ago

yep. I just created https://github.com/deis/builder/issues/51 to track this.

deis / builder

Builder error on second push #49