search

deis / builder

Git server and application builder for Deis Workflow
https://deis.com
MIT License
40 stars 41 forks source link

feat(intervals): make pod tick intervals agressive to observe pod change #200

Closed smothiki closed 8 years ago

smothiki commented 8 years ago

interval to check for pod running state is 0.05 second and to observe succeeded change is 0.01 second

smothiki commented 8 years ago

possibly should fix https://github.com/deis/builder/issues/199

smothiki commented 8 years ago

Again needs manual testing. As the behavior observed in https://github.com/deis/builder/issues/199 is intermittent

slack commented 8 years ago

Testing via: smothiki/builder:v2.4

Working well so far after ~ 10 deploys.

smothiki commented 8 years ago

slack commented 8 years ago

Blew up:

remote: 2016/02/24 20:19:07 Error running git receive hook [watching events for builder pod startup (timed out waiting for the condition)]

kubectl get pods -w:

slugbuild-earthy-instinct-e27012df-eddb90ba   0/1       Pending   0         0s
slugbuild-earthy-instinct-e27012df-eddb90ba   0/1       Pending   0         0s        ip-10-0-0-98.us-west-2.compute.internal
slugbuild-earthy-instinct-e27012df-eddb90ba   0/1       Terminating   0         0s        ip-10-0-0-98.us-west-2.compute.internal
slugbuild-earthy-instinct-e27012df-eddb90ba   0/1       Terminating   0         0s        ip-10-0-0-98.us-west-2.compute.internal
slugbuild-earthy-instinct-e27012df-eddb90ba   0/1       Terminating   0         0s        ip-10-0-0-98.us-west-2.compute.internal
slugbuild-earthy-instinct-e27012df-eddb90ba   0/1       Terminating   0         0s        ip-10-0-0-98.us-west-2.compute.internal

Deploy output:

[master e27012d] Bump
Warning: the RSA host key for '[deis.beef.slack.io]:2222' differs from the key for the IP address '[54.68.224.84]:2222'
Offending key for IP in /Users/jhansen/.ssh/known_hosts:111
Matching host key in /Users/jhansen/.ssh/known_hosts:112
Counting objects: 559, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (554/554), done.
Writing objects: 100% (559/559), 57.00 KiB | 0 bytes/s, done.
Total 559 (delta 512), reused 0 (delta 0)
remote: 2016/02/24 20:14:06 Running in debug mode
Running git hook
read [0000000000000000000000000000000000000000,e27012df423398291b236a71bb81c8c7b117978c,refs/heads/master]
Workflow request /v2/hooks/push (body elided)
Workflow request POST /v2/hooks/config
{"receive_user":"jhansen","receive_repo":"earthy-instinct"}
got the following config back for app earthy-instinct: {Owner:jhansen App:earthy-instinct Values:map[POWERED_BY:IPA] Memory:map[] CPU:map[] Tags:map[] UUID:1d881408-a7d7-4c2f-87ac-60d3771a0095 Created:2016-02-23 00:33:31 +0000 UTC Updated:2016-02-23 01:53:13 +0000 UTC}
running [git archive --format=tar.gz --output=earthy-instinct.tar.gz e27012df] in directory /home/git/earthy-instinct.git
running [tar -xzf earthy-instinct.tar.gz -C /home/git/earthy-instinct.git/build/tmp256406575/] in directory /home/git/earthy-instinct.git
Uploading tar to http://10.3.0.80:9000/git/home/earthy-instinct:git-e27012df/tar
Starting build... but first, coffee!
Starting pod slugbuild-earthy-instinct-e27012df-eddb90ba
Pod spec: {
  "metadata": {
    "name": "slugbuild-earthy-instinct-e27012df-eddb90ba",
    "namespace": "deis",
    "creationTimestamp": null,
    "labels": {
      "heritage": "deis",
      "version": "2.0.0-beta"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "minio-user",
        "secret": {
          "secretName": "minio-user"
        }
      }
    ],
    "containers": [
      {
        "name": "deis-slugbuilder",
        "image": "quay.io/deisci/slugbuilder:v2-beta",
        "env": [
          {
            "name": "POWERED_BY",
            "value": "IPA"
          },
          {
            "name": "DEBUG",
            "value": "1"
          },
          {
            "name": "TAR_URL",
            "value": "http://10.3.0.80:9000/git/home/earthy-instinct:git-e27012df/tar"
          },
          {
            "name": "put_url",
            "value": "http://10.3.0.80:9000/git/home/earthy-instinct:git-e27012df/push"
          }
        ],
        "resources": {},
        "volumeMounts": [
          {
            "name": "minio-user",
            "readOnly": true,
            "mountPath": "/var/run/secrets/object/store"
          }
        ],
        "imagePullPolicy": "Always"
      }
    ],
    "restartPolicy": "Never",
    "serviceAccountName": ""
  },
  "status": {}
}
remote: 2016/02/24 20:19:07 Error running git receive hook [watching events for builder pod startup (timed out waiting for the condition)]

To ssh://git@deis.beef.slack.io:2222/earthy-instinct.git
 * [new branch]      master -> master

real    5m1.636s
user    0m0.057s
sys 0m0.055s

K8S event stream for pod:

k:beef ~ $ kd get events --all-namespaces | egrep "slugbuild-earthy-instinct-e27012df-eddb90ba"
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     implicitly required container POD      Pulled             {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Container image "gcr.io/google_containers/pause:0.8.0" already present on machine
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     implicitly required container POD      Started            {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Started with docker id f06e5e0a4ada
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     spec.containers{deis-slugbuilder}      Pulling            {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Pulling image "quay.io/deisci/slugbuilder:v2-beta"
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     implicitly required container POD      Created            {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Created with docker id f06e5e0a4ada
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                                                            Scheduled          {scheduler }                                         Successfully assigned slugbuild-earthy-instinct-e27012df-eddb90ba to ip-10-0-0-98.us-west-2.compute.internal
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     spec.containers{deis-slugbuilder}      Created            {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Created with docker id a5fd5ba59ab1
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     spec.containers{deis-slugbuilder}      Started            {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Started with docker id a5fd5ba59ab1
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     spec.containers{deis-slugbuilder}      Pulled             {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Successfully pulled image "quay.io/deisci/slugbuilder:v2-beta"
deis              9m          9m         1         slugbuild-earthy-instinct-e27012df-eddb90ba     Pod                     implicitly required container POD      Killing            {kubelet ip-10-0-0-98.us-west-2.compute.internal}    Killing with docker id f06e5e0a4ada
smothiki commented 8 years ago

updated image smothiki/builder:v2.4.1

smothiki commented 8 years ago

finished doing 10 git pushes. Same story nothing's miss behaving

slack commented 8 years ago

v2.4.1 isn't working well for me at all.

mboersma commented 8 years ago

I've tested this based on the :v2.4.1 image and I've had builds stall out and never finish, or seen the slug get garbage-collected before it was deployed. @smothiki has some ideas and is going to iterate on this.

bacongobbler commented 8 years ago

I'm assuming that whatever gets eventually merged will be testable by the e2e tests since this affects application builds, correct? Lack of tests with the ongoing comments has made me concerned what the tradeoffs are with this fix.

smothiki commented 8 years ago

@bacongobbler https://github.com/deis/builder/pull/185 this is a permanent fix which is not even close to working as of now. I'm just trying to fix this temporarily.

smothiki commented 8 years ago

closing this as polling is not the appropriate solution for the issue