search

deis / builder

Git server and application builder for Deis Workflow
https://deis.com
MIT License
40 stars 41 forks source link

Cannot deploy example-go #111

Closed PAStheLoD closed 8 years ago

PAStheLoD commented 8 years ago

Hello,

I'm trying to go through the v2 alpha gist. (Or the deis/charts one, as they seem to be identical.)

This is a single node local k8s "cluster" running on Docker 1.9.1, on Ubuntu Xenial (16.04).

pas@strange:~/projects/kubernetes/example-go$ ../deis create test2
Creating Application... done, created test2
Git remote deis added
remote available at ssh://git@deis.10.0.0.55.xip.io:2222/test2.git
pas@strange:~/projects/kubernetes/example-go$ 
pas@strange:~/projects/kubernetes/example-go$ git push deis master
Counting objects: 75, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (70/70), done.
Writing objects: 100% (75/75), 17.20 KiB | 0 bytes/s, done.
Total 75 (delta 32), reused 0 (delta 0)
-----> Starting build
-----> Go app detected

Builder log 

[info] Adding key 'command="/usr/local/bin/gitreceive run hello 4a:ea:0d:3a:11:9c:26:3b:6c:d6:75:99:08:5b:27:95",no-agent-forwarding,no-pty,no-user-rc,no-X11-forwarding,no-port-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwtZ3B67j6VPIVr7xPo6AYaGbuy7W1pJ+MGgGS0HzokoQ5ihvttiV/SuM6l6FmaLePex27XlKXifqwd8rzKd1U+kRzU6f3CONK9yzvI8xf4wlUHCdoEM/ZIR0pR3h5xQaZt51z7Yo7kDKazIjKSwzXHiagi1I/mBIu/GyDz2oy3gkabMPT0Cr54EanIV6d1npoE7rjgegVpihthHDGOs7yVP1cAZM7vvMKUK+YfQt8Tw5qWAGhAWkelnNQnV6QMc7nPZenlFU7YVoJd110zjP0Wb27t/ikBoTFmRB5Ddu2/jjtk9OBtw8pZiHQFsUAHOtQZGwYtZIT8xBV7gQqlWk3 pas@strange'
[info] Key accepted for user git.
[info] Channel type: session
Key='LC_PAPER', Value='hu_HU.UTF-8'
Key='LC_ADDRESS', Value='hu_HU.UTF-8'
Key='LC_MONETARY', Value='hu_HU.UTF-8'
Key='LC_NUMERIC', Value='hu_HU.UTF-8'
Key='LC_TELEPHONE', Value='hu_HU.UTF-8'
Key='LC_IDENTIFICATION', Value='hu_HU.UTF-8'
Key='LANG', Value='en_US.UTF-8'
Key='LC_MEASUREMENT', Value='hu_HU.UTF-8'
Key='LC_TIME', Value='hu_HU.UTF-8'
Key='LC_NAME', Value='hu_HU.UTF-8'
[info] Checking user /deis/builder/users/hello
[info] Found user hello for fingerprint 4a:ea:0d:3a:11:9c:26:3b:6c:d6:75:99:08:5b:27:95
[info] Creating new directory at /home/git/test2.git
[info] git-shell -c git-receive-pack 'test2.git'
Waiting for git-receive to run.
Waiting for deploy.

Slug builder log

pas@strange:~/projects/kubernetes/kubernetes-docker-dns$ docker logs e9e55925ef0d
-----> Go app detected
-----> Checking Godeps/Godeps.json file.

Workflow log

172.17.0.9 "POST /v2/apps/ HTTP/1.1" 201 183 "Deis Client v2.0.0-dev"
172.17.0.9 "POST /v2/apps/ HTTP/1.1" 400 37 "Deis Client v2.0.0-dev"
INFO test2: config test2-9e9e3e6 updated
INFO test2: release test2-v1 created
INFO [test2]: hello created initial release
172.17.0.9 "POST /v2/apps/ HTTP/1.1" 201 185 "Deis Client v2.0.0-dev"
172.17.0.2 "POST /v2/hooks/push HTTP/1.1" 201 412 "curl/7.39.0"

Thanks for looking at this!

arschles commented 8 years ago

@PAStheLoD thanks for reporting this! Are the logs that you're seeing on the git push (or slugbuilder logs, as they should be very similar) hanging on the -----> Checking Godeps/Godeps.json file. part?

That step indicates that the slugbuilder has been successfully launched and is beginning to build using the buildpack, but the actual build process may take some time, especially if the node that runs the Kubernetes "cluster" is memory and/or CPU constrained.

bacongobbler commented 8 years ago

When you checked the slugbuilder log, did you check to see if the pod had failed? If so can you show what is the output of kubectl describe for us? Thanks!

bacongobbler commented 8 years ago

@arschles this also looks very similar to the issues I was seeing in https://github.com/deis/builder/pull/72#issuecomment-171746221. Considering that it's a local kubernetes cluster on Xenial, I would take a hunch that it's another virtualbox networking issue, which we may need to take a look at the kubernetes issue list to see if anyone else has seen this.

arschles commented 8 years ago

@bacongobbler agreed. Nothing jumps out after a while searching the Kubernetes issue list, but I'll continue searching. I also wonder if there's a compounding memory issue of some kind, but at this point, it's just speculation.

@PAStheLoD we'd love to hear back about your log and kubectl describe output!

PAStheLoD commented 8 years ago

Hello!

Sorry, I had to recreate this from scratch, as there are other interesting problems with hyperkube. (If I docker kill all the containers, then I (re)start kubelet, it'll find in /var/lib/kubelet that it has pods to run, and it runs them furiously, but somehow the network gets messed up, and deis-router was refusing connections, even when I tried it by docker execing wget in the same "pod".)

To answer the questions. Yes, the slugbuilder basically exited unexpected.

@bacongobbler @arschles It's a physical machine, still ~4G RAM free, and plenty of SSD space (~100G) for docker too. So, it's not running in a VirtualBox.

pas@strange:~/projects/kubernetes/example-go$ kd get po
NAME                        READY     STATUS    RESTARTS   AGE
deis-builder-677tp          1/1       Running   1          9m
deis-database-kd30y         1/1       Running   0          9m
deis-etcd-1-68ws1           1/1       Running   2          9m
deis-etcd-1-6wnkz           1/1       Running   2          9m
deis-etcd-1-xq0vc           1/1       Running   2          9m
deis-etcd-discovery-jza4z   1/1       Running   0          9m
deis-minio-fa2sm            1/1       Running   0          9m
deis-registry-pvzda         1/1       Running   0          9m
deis-router-bu73l           1/1       Running   0          9m
deis-workflow-x8ipq         1/1       Running   0          9m
CONTAINER ID        IMAGE                                           COMMAND                  CREATED             STATUS                        PORTS                   
ecd657124511        quay.io/deis/slugbuilder:2.0.0-alpha            "/tmp/builder/build.s"   10 minutes ago      Exited (6) 10 minutes ago                             
c3ed4ce166c1        gcr.io/google_containers/pause:2.0              "/pause"                 10 minutes ago      Exited (0) 9 minutes ago                              
d427853f502e        quay.io/deis/etcd:2.0.0-alpha                   "/usr/local/bin/boot"    13 minutes ago      Up 13 minutes                                         
99747d7abd96        quay.io/deis/etcd:2.0.0-alpha                   "/usr/local/bin/boot"    13 minutes ago      Up 13 minutes                                         
7aecbabf64ae        quay.io/deis/etcd:2.0.0-alpha                   "/usr/local/bin/boot"    13 minutes ago      Up 13 minutes                                         
ec14cb4caaaa        quay.io/deis/builder:2.0.0-alpha                "/entrypoint.sh /bin/"   13 minutes ago      Up 13 minutes    
pas@strange:~/projects/kubernetes/kubernetes/cluster/images/hyperkube$ docker logs ecd657124511
-----> Go app detected
-----> Checking Godeps/Godeps.json file.

docker inspect <...>:

    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 6,
        "Error": "",
        "StartedAt": "2016-01-23T17:34:23.967566493Z",
        "FinishedAt": "2016-01-23T17:34:44.827927721Z"
    },

kd get events:

12m         12m        1         mystic-umbrella-git-5450cbcd   Pod                     spec.containers{deis-slugbuilder}      Pulled             {kubelet 127.0.0.1}         Container image "quay.io/deis/slugbuilder:2.0.0-alpha" already present on machine
12m         12m        1         mystic-umbrella-git-5450cbcd   Pod                     spec.containers{deis-slugbuilder}      Created            {kubelet 127.0.0.1}         Created container with docker id ecd657124511
12m         12m        1         mystic-umbrella-git-5450cbcd   Pod                     spec.containers{deis-slugbuilder}      Started            {kubelet 127.0.0.1}         Started container with docker id ecd657124511

And still I have git push running:

$ git push deis -u master
Counting objects: 75, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (70/70), done.
Writing objects: 100% (75/75), 17.20 KiB | 0 bytes/s, done.
Total 75 (delta 32), reused 0 (delta 0)
-----> Starting build
-----> Go app detected

And it'll just TCP timeout eventually.

arschles commented 8 years ago

@PAStheLoD sorry for the delay here. You indeed do have more than enough memory and disk space to build the slug and deploy it. The part that alarms me is that I believe the Go buildpack tries to download Go (here) shortly after the Go app detected log line. So, I'm wondering if there's some networking issue that's preventing the download?

Finally, is it possible for you to try the v2-beta images, to rule out issues specific to the v2-alpha release? You can install the v2-beta using the deis-dev Helm chart.

PAStheLoD commented 8 years ago

No problem! I wanted to try the "master" branch for some time anyway.

So, I'm wondering if there's some networking issue that's preventing the download?

I'm fairly certain that this isn't the case. 

$ wget https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz 
--2016-01-30 13:49:42--  https://storage.googleapis.com/golang/go1.5.3.linux-amd64.tar.gz
Resolving storage.googleapis.com (storage.googleapis.com)... 216.58.213.48, 2a00:1450:4008:803::2010
Connecting to storage.googleapis.com (storage.googleapis.com)|216.58.213.48|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 80147269 (76M) [application/octet-stream]
Saving to: ‘go1.5.3.linux-amd64.tar.gz’

go1.5.3.linux-amd64.tar.gz                                  100%[========================================================================================================================================>]  76,43M  5,19MB/s    in 16s     

2016-01-30 13:49:58 (4,88 MB/s) - ‘go1.5.3.linux-amd64.tar.gz’ saved [80147269/80147269]

Okay, so with helm install deis/deis-dev. Then uninstall, and using generate deis-dev and then installing, deis nicely booted up.

$ git push deis master

Counting objects: 75, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (70/70), done.
Writing objects: 100% (75/75), 17.20 KiB | 0 bytes/s, done.
Total 75 (delta 32), reused 0 (delta 0)
-----> Starting build... but first, coffee!
-----> Go app detected
-----> Checking Godeps/Godeps.json file.
-----> Build complete.
-----> Launching app.
-----> Launching...
-----> Done, tender-teaspoon:v2 deployed to Deis

-----> Use 'deis open' to view this application in your browser

-----> To learn more, use 'deis help' or visit http://deis.io

To ssh://git@deis.10.0.0.144.xip.io:2222/tender-teaspoon.git
 * [new branch]      master -> master

But, alas, it cannot start. slugrunner just exits :(

$ kd get po -a
NAME                                          READY     STATUS    RESTARTS   AGE
deis-builder-4rae9                            1/1       Running   1          1h
deis-database-ovvbs                           1/1       Running   0          1h
deis-etcd-1-f2yw6                             1/1       Running   0          1h
deis-etcd-1-kj6cd                             1/1       Running   0          1h
deis-etcd-1-xb3yh                             1/1       Running   1          1h
deis-etcd-discovery-gnoqj                     1/1       Running   0          1h
deis-minio-hftzj                              1/1       Running   0          1h
deis-registry-bz1wr                           1/1       Running   0          1h
deis-router-bnb53                             1/1       Running   0          1h
deis-workflow-lt8ou                           1/1       Running   1          1h
slugbuild-tender-teaspoon-5450cbcd-64ec879c   0/1       Error     0          1h

The failed slugbuilder could have been because I did a git push deis without master which just resulted in a seemingly forever wait.

pas@strange:~/projects/kubernetes/example-go$ git push deis
The authenticity of host '[deis.10.0.0.144.xip.io]:2222 ([10.0.0.144]:2222)' can't be established.
ECDSA key fingerprint is SHA256:mz5xEj4zb3ayUzD/0ugh2dTCBY/wRTsIF3+ca/4BXzs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[deis.10.0.0.144.xip.io]:2222,[10.0.0.144]:2222' (ECDSA) to the list of known hosts.
No refs in common and none specified; doing nothing.
Perhaps you should specify a branch such as 'master'.

^C
$ docker events 
...
2016-01-30T12:02:01.392908188+01:00 quay.io/deisci/slugrunner:v2-beta: pull
2016-01-30T12:02:01.487670671+01:00 e6bd18be54f6cd60b9b9c2a1b1265c0e5e46c04ea3bf7712b5884d259459517f: (from quay.io/deisci/slugrunner:v2-beta) create
2016-01-30T12:02:01.490230432+01:00 e6bd18be54f6cd60b9b9c2a1b1265c0e5e46c04ea3bf7712b5884d259459517f: (from quay.io/deisci/slugrunner:v2-beta) start
2016-01-30T12:02:01.681916024+01:00 e6bd18be54f6cd60b9b9c2a1b1265c0e5e46c04ea3bf7712b5884d259459517f: (from quay.io/deisci/slugrunner:v2-beta) die
2016-01-30T12:02:05.786975253+01:00 quay.io/deisci/slugrunner:v2-beta: pull
2016-01-30T12:02:05.914431780+01:00 20d400c311c3456b077b050f1318de0a21c481150e60696c59f9489343bf9511: (from quay.io/deisci/slugrunner:v2-beta) create
2016-01-30T12:02:05.917073445+01:00 20d400c311c3456b077b050f1318de0a21c481150e60696c59f9489343bf9511: (from quay.io/deisci/slugrunner:v2-beta) start
2016-01-30T12:02:06.101892336+01:00 20d400c311c3456b077b050f1318de0a21c481150e60696c59f9489343bf9511: (from quay.io/deisci/slugrunner:v2-beta) die
... and so on
$ kubectl  logs tender-teaspoon-v2-web-xbl4s --namespace=tender-teaspoon -p
+ export HOME=/app
+ HOME=/app
+ mkdir -p /app
++ ls -A /app
+ [[ -n '' ]]
+ [[ -n http://10.0.0.150:9000/git/home/tender-teaspoon:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push/slug.tgz ]]
+ [[ -e /var/run/secrets/object/store/access-key-id ]]
+ [[ -e /var/run/secrets/object/store/access-secret-key ]]
++ cat /var/run/secrets/object/store/access-key-id
+ keyID=8TZRY2JRWMPT6UMXR6I5
++ cat /var/run/secrets/object/store/access-secret-key
+ secretKey=gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
++ echo http://10.0.0.150:9000/git/home/tender-teaspoon:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push/slug.tgz
++ awk -F/ '{print $1}'
+ protocol=http:
++ echo http://10.0.0.150:9000/git/home/tender-teaspoon:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push/slug.tgz
++ awk -F/ '{print $3}'
+ domain=10.0.0.150:9000
+ mc config host add http://10.0.0.150:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk
+ mc --quiet cp http://10.0.0.150:9000/git/home/tender-teaspoon:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push/slug.tgz slug.tgz

Nothing interesting in builder log, yet the slug.tgz seems to be 404.

$ curl http://10.0.0.150:9000/git/home/tender-teaspoon:git-5450cbcdaaf9afe6fadd219c94ac9c449bd62413/push/slug.tgz -I
HTTP/1.1 404 Not Found
Accept-Ranges: bytes
Server: Minio/UNOFFICIAL.GOGET (linux; amd64)
Vary: Origin
X-Amz-Request-Id: GPWFLBZ63IZGS8QK
Date: Sat, 30 Jan 2016 12:33:20 GMT
Content-Type: text/plain; charset=utf-8
$ docker ps -a | grep slugbuilder
c4fdd633d60a        quay.io/deisci/slugbuilder:v2-beta              "/tmp/builder/build.s"   About an hour ago   Exited (6) About an hour ago                                                                                                k8s_deis-slugbuilder.ce439e1f_slugbuild-tender-teaspoon-5450cbcd-64ec879c_deis_bc85840f-c740-11e5-9446-902b34d07023_6e7a3c2d

$ docker logs c4fdd633d60a
-----> Go app detected
-----> Checking Godeps/Godeps.json file.
arschles commented 8 years ago

@PAStheLoD thanks for getting back, and apologies for the huge delay here. We've been working hard on the beta release, and we've been able to reproduce issues very similar to yours. Would you mind trying the git push that you originally tried using the v2-beta images? If you're using helm, you can use the chart at deis/deis-dev

arschles commented 8 years ago

hey @PAStheLoD, just checking in here - have you been able to test the git push on our workflow-dev chart (it was renamed from deis-dev)? Additionally, we've released a beta3 and beta4 chart since my last posting.

Joshua-Anderson commented 8 years ago

@PAStheLoD Thanks for being one of the early testers of deis workflow! It's now been GA for a few months and I've had no issues with example-go, so I'm closing as no longer an issue. Feel free to reopen if you can reproduce this after deploying workflow v2.4.0, which is coming out shorty.