Closed mboersma closed 7 years ago
From inside the fluentd container, the symlinked log files don't exist:
root@deis-logger-fluentd-nebvp:/# ls -al /var/log/containers/nearby*
lrwxrwxrwx 1 root root 174 Sep 13 17:58 /var/log/containers/nearby-rabbitry-web-1965060662-x2qe6_nearby-rabbitry_POD-ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a.log -> /mnt/sda1/var/lib/docker/containers/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a-json.log
lrwxrwxrwx 1 root root 174 Sep 13 17:58 /var/log/containers/nearby-rabbitry-web-1965060662-x2qe6_nearby-rabbitry_nearby-rabbitry-web-8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e.log -> /mnt/sda1/var/lib/docker/containers/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e-json.log
root@deis-logger-fluentd-nebvp:/# cat /mnt/sda1/var/lib/docker/containers/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a/ff373bf36f21893880654b9a5bd41040fca54a12c64ad868972cc20fb13e6f7a-json.log
cat: /mnt/sda1/var/lib/docker/containers/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e-json.log: No such file or directory
root@deis-logger-fluentd-nebvp:/#
Although they do exist on the host:
docker@boot2docker:~$ sudo cat /mnt/sda1/var/lib/docker/containers/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e
4930b2019820c96e/8ecd5da18f5b783e27d8b66555481d1065108f7f1bc2e41e4930b2019820c96e-json.log
{"log":"[2016-09-13 17:58:10 +0000] [1] [INFO] Starting gunicorn 19.6.0\n","stream":"stderr","time":"2016-09-13T17:58:10.058972481Z"}
{"log":"[2016-09-13 17:58:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)\n","stream":"stderr","time":"2016-09-13T17:58:10.059739267Z"}
{"log":"[2016-09-13 17:58:10 +0000] [1] [INFO] Using worker: sync\n","stream":"stderr","time":"2016-09-13T17:58:10.05980172Z"}
{"log":"[2016-09-13 17:58:10 +0000] [29] [INFO] Booting worker with pid: 29\n","stream":"stderr","time":"2016-09-13T17:58:10.06381675Z"}
{"log":"Not Found: /favicon.ico\n","stream":"stderr","time":"2016-09-13T18:02:20.410486541Z"}
Alright so in the chart we mount /var/lib/docker
into the same path. But we do not search that path for log files. We should figure out how to better handle this.
Similarly to #50, I worked around this by amending the fluentd daemon mount to /mnt/sda1/var/lib/docker/containers
, the host path:
$ cat ~/.helmc/workspace/charts/workflow-dev/manifests/deis-logger-fluentd-daemon.yaml
#helm:generate helm tpl -d $HELM_GENERATE_DIR/tpl/generate_params.toml -o $HELM_GENERATE_DIR/manifests/deis-logger-fluentd-daemon.yaml $HELM_GENERATE_FILE
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: deis-logger-fluentd
namespace: deis
labels:
heritage: deis
annotations:
component.deis.io/version: canary
spec:
selector:
matchLabels:
app: deis-logger-fluentd
heritage: deis
template:
metadata:
name: deis-logger-fluentd
labels:
heritage: deis
app: deis-logger-fluentd
spec:
serviceAccount: deis-logger-fluentd
containers:
- name: deis-logger-fluentd
image: quay.io/deisci/fluentd:canary
imagePullPolicy: Always
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /mnt/sda1/var/lib/docker/containers
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /mnt/sda1/var/lib/docker/containers
$ deis logs
INFO [orange-chinbone]: config orange-chinbone-94b61e9 updated
INFO [orange-chinbone]: mboersma created initial release
INFO [orange-chinbone]: appsettings orange-chinbone-4406a03 updated
INFO [orange-chinbone]: domain orange-chinbone added
INFO [orange-chinbone]: build orange-chinbone-4a5e377 created
INFO [orange-chinbone]: mboersma deployed c1e2e65
INFO [orange-chinbone]: buildpack type detected. Defaulting to $PORT 5000
INFO [orange-chinbone]: adding 30s on to the original 120s timeout to account for the initial delay specified in the liveness / readiness probe
INFO [orange-chinbone]: This deployments overall timeout is 150s - batch timout is 150s and there are 1 batches to deploy with a total of 1 pods
INFO [orange-chinbone]: waited 10s and 1 pods are in service
INFO [orange-chinbone]: waited 20s and 1 pods are in service
INFO [orange-chinbone]: waited 30s and 1 pods are in service
Behavior is the same with minkube 0.10.0.
Running these commands inside the fluentd container also repairs the symlinking so deis logs
works:
$ mkdir -p /mnt/sda1/var/lib/docker
$ ln -s /var/lib/docker/containers /mnt/sda1/var/lib/docker/containers
I wonder if I could automate that somehow before fluentd starts, in a backward-compatible way.
Kubernetes: 1.3.5 minikube: v0.9.0, xhyve driver host: macOS Sierra v10.12 beta Deis Workflow: v2.5.0 Chart
The e2e test specs covering
deis logs
fail, as it does interactively:If it's possible to accomodate the single-node
minikube
test k8s environment, we should do so.