hammerlab / ketrew

Keep Track of Experimental Workflows
http://www.hammerlab.org/docs/ketrew/master/index.html
Apache License 2.0
76 stars 10 forks source link

Mystery failure #474

Open ihodes opened 7 years ago

ihodes commented 7 years ago

Running on Coclobas, simple node didn't run successfully, don't know why: restarting the node a day later it worked just fine.

### Kube-Job adcb443d-6813-57fb-8ca9-1b22a00a2990
### Freshness: Archived because of error: Shell-command failed:
Command:

kubectl logs adcb443d-6813-57fb-8ca9-1b22a00a2990

Status: Exited with 1

Standard-output: empty.

Standard-error:

Error from server: pods "adcb443d-6813-57fb-8ca9-1b22a00a2990" not found

### Output:

User
biokepi
Host
adcb443d-6813-57fb-8ca9-1b22a00a2990
Machine
Linux adcb443d-6813-57fb-8ca9-1b22a00a2990 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 x86_64 x86_64 GNU/Linux
biokepi
biokepi

screen shot 2016-09-27 at 12 54 10 pm

and

### Kube-Job adcb443d-6813-57fb-8ca9-1b22a00a2990
### Freshness: Archived because of error: Shell-command failed:
Command:

kubectl describe pod adcb443d-6813-57fb-8ca9-1b22a00a2990

Status: Exited with 1

Standard-output: empty.

Standard-error:

Error from server: pods "adcb443d-6813-57fb-8ca9-1b22a00a2990" not found

### Output:

Name:       adcb443d-6813-57fb-8ca9-1b22a00a2990
Namespace:  default
Node:       gke-ihodes-coco-default-pool-f2f6a9fc-s41m/10.142.0.73
Start Time: Sat, 24 Sep 2016 17:38:22 +0000
Labels:     app=adcb443d-6813-57fb-8ca9-1b22a00a2990
Status:     Succeeded
IP:     10.91.66.4
Controllers:    <none>
Containers:
  adcb443d-6813-57fb-8ca9-1b22a00a2990container:
    Container ID:   docker://fba7c812ff8345376e9480a6c5b4bd1934502dfdcbe7ca204bb982bcb127122f
    Image:      hammerlab/biokepi-run
    Image ID:       docker://sha256:392a1d51ad6f562b91556e4cb144af276e79641ccaf8e1230a353839b19eaeff
    Port:       
    Command:
      bash
      /coclo-kube/mount/script
    Requests:
      cpu:          7
      memory:           50G
    State:          Terminated
      Reason:           Completed
      Exit Code:        0
      Started:          Sat, 24 Sep 2016 17:39:38 +0000
      Finished:         Sat, 24 Sep 2016 17:39:38 +0000
    Ready:          False
    Restart Count:      0
    Environment Variables:  <none>
Conditions:
  Type      Status
  Initialized   True 
  Ready     False 
  PodScheduled  True 
Volumes:
  adcb443d-6813-57fb-8ca9-1b22a00a2990655ff27f6486d9c7aa8-volume:
    Type:   Secret (a volume populated by a Secret)
    SecretName: adcb443d-6813-57fb-8ca9-1b22a00a2990655ff27f6486d9c7aa8
  ihodes-strato-pgv-nfs-vm-8017af3:
    Type:   NFS (an NFS mount that lasts the lifetime of a pod)
    Server: ihodes-strato-pgv-nfs-vm
    Path:   /ihodes-strato-pgv-nfs-data/
    ReadOnly:   false
  pgv-datasets-vm-2d98ba0d:
    Type:   NFS (an NFS mount that lasts the lifetime of a pod)
    Server: pgv-datasets-vm
    Path:   /pgv-datasets-storage/
    ReadOnly:   false
  seb-test-nfs-server-vm-1157945b:
    Type:   NFS (an NFS mount that lasts the lifetime of a pod)
    Server: seb-test-nfs-server-vm
    Path:   /seb-test-storage
    ReadOnly:   false
  default-token-393ho:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-393ho
QoS Tier:   Burstable
Events:
  FirstSeen LastSeen    Count   From                            SubobjectPath                           Type        Reason      Message
  --------- --------    -----   ----                            -------------                           --------    ------      -------
  1m        1m      1   {default-scheduler }                                                    Normal      Scheduled   Successfully assigned adcb443d-6813-57fb-8ca9-1b22a00a2990 to gke-ihodes-coco-default-pool-f2f6a9fc-s41m
  1m        1m      1   {kubelet gke-ihodes-coco-default-pool-f2f6a9fc-s41m}    spec.containers{adcb443d-6813-57fb-8ca9-1b22a00a2990container}  Normal      Pulling     pulling image "hammerlab/biokepi-run"
  26s       26s     1   {kubelet gke-ihodes-coco-default-pool-f2f6a9fc-s41m}    spec.containers{adcb443d-6813-57fb-8ca9-1b22a00a2990container}  Normal      Pulled      Successfully pulled image "hammerlab/biokepi-run"
  26s       26s     1   {kubelet gke-ihodes-coco-default-pool-f2f6a9fc-s41m}    spec.containers{adcb443d-6813-57fb-8ca9-1b22a00a2990container}  Normal      Created     Created container with docker id fba7c812ff83
  25s       25s     1   {kubelet gke-ihodes-coco-default-pool-f2f6a9fc-s41m}    spec.containers{adcb443d-6813-57fb-8ca9-1b22a00a2990container}  Normal      Started     Started container with docker id fba7c812ff83
smondet commented 7 years ago

Anything new? reproduction?