Is there any limits about the version of docker for promtail? I failed to deploy it on 18.3.1 version of docker, and there's no any error logs.

TimeBye commented 4 years ago

Describe the bug Is there any limits about the version of docker for promtail? I failed to deploy it on 18.3.1 version of docker, and there's no any error logs.

To Reproduce Steps to reproduce the behavior:

Started Loki (SHA or version)

helm repo add loki https://grafana.github.io/loki/charts
helm repo update
helm upgrade --install loki loki/loki --namespace loki

Started Promtail (SHA or version) to tail '...'

helm upgrade --install promtail loki/promtail --set "loki.serviceName=loki" --namespace loki

Expected behavior Normal operation for promtail.

Environment:

Infrastructure:

Kubernetes

[root@stagingx ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

[root@stagingx ~]# kubectl get node -o wide
NAME        STATUS                     ROLES         AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
staging01   Ready                      worker        404d   v1.14.6   192.168.16.214   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.3.1
staging02   Ready                      worker        404d   v1.14.6   192.168.16.215   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.3.1
staging03   Ready                      worker        404d   v1.14.6   192.168.16.216   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.3.1
staging04   Ready                      worker        404d   v1.14.6   192.168.16.217   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.3.1
staging05   Ready                      worker        404d   v1.14.6   192.168.16.218   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.3.1
staging06   Ready                      worker        404d   v1.14.6   192.168.16.219   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.3.1
staging07   Ready                      worker        266d   v1.14.6   192.168.16.220   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.9.6
stagingx    Ready,SchedulingDisabled   etcd,master   404d   v1.14.6   192.168.16.228   <none>        CentOS Linux 7 (Core)   4.20.13-1.el7.elrepo.x86_64   docker://18.9.6

Docker(Version 18.3.1)

[root@staging01 ~]# docker info
Containers: 31
Running: 25
Paused: 0
Stopped: 6
Images: 327
Server Version: 18.03.1-ce
Storage Driver: devicemapper
Pool Name: docker-253:1-134218191-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 45.64GB
Data Space Total: 107.4GB
Data Space Available: 49.12GB
Metadata Space Used: 70.82MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.077GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.20.13-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.64GiB
Name: staging01
ID: FNY6:JXWX:56QD:DSUZ:IX76:IHD5:S64L:CIYX:7RQ6:XMQB:LRWG:GK5D
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
10.233.0.0/18
127.0.0.0/8
Live Restore Enabled: false

WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
       Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

Docker(Version 18.9.6)

[root@stagingx ~]# docker info
Containers: 17
Running: 16
Paused: 0
Stopped: 1
Images: 10
Server Version: 18.09.6
Storage Driver: devicemapper
Pool Name: docker-253:1-100960806-pool
Pool Blocksize: 65.54kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported: true
Data file: /dev/loop0
Metadata file: /dev/loop1
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Data Space Used: 1.13GB
Data Space Total: 107.4GB
Data Space Available: 92.15GB
Metadata Space Used: 18.57MB
Metadata Space Total: 2.147GB
Metadata Space Available: 2.129GB
Thin Pool Minimum Free Space: 10.74GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.149-RHEL7 (2018-07-20)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.20.13-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.829GiB
Name: stagingx
ID: IVWT:RKUN:PW7X:RDV6:7UFR:JDNP:SWX2:N6PU:7CPL:2DX3:C7PZ:ALMV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
10.244.0.0/18
10.244.64.0/18
127.0.0.0/8
Registry Mirrors:
https://dockerhub.azk8s.cn/
https://quay.azk8s.cn/
Live Restore Enabled: false
Product License: Community Engine

WARNING: the devicemapper storage-driver is deprecated, and will be removed in a future release.
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
       Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

Deployment tool:

helm

[root@stagingx ~]# helm version
Client: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.6", GitCommit:"dd2e5695da88625b190e6b22e9542550ab503a47", GitTreeState:"clean"}

Screenshots, Promtail config, or terminal output

terminal output

Pod

[root@stagingx ~]# kubectl get po -n loki -o wide --sort-by {.spec.nodeName}
NAME             READY   STATUS             RESTARTS   AGE    IP              NODE        NOMINATED NODE   READINESS GATES
promtail-qwnwb   0/1     CrashLoopBackOff   36         162m   10.233.67.128   staging01   <none>           <none>
promtail-flwl9   0/1     CrashLoopBackOff   36         162m   10.233.68.190   staging02   <none>           <none>
promtail-9q598   0/1     CrashLoopBackOff   36         162m   10.233.65.109   staging03   <none>           <none>
promtail-xp5zb   0/1     CrashLoopBackOff   36         162m   10.233.66.107   staging04   <none>           <none>
promtail-2mxsf   0/1     CrashLoopBackOff   36         162m   10.233.69.113   staging05   <none>           <none>
promtail-flrnz   0/1     CrashLoopBackOff   36         162m   10.233.70.139   staging06   <none>           <none>
loki-0           1/1     Running            0          161m   10.233.71.12    staging07   <none>           <none>
promtail-5xqdz   1/1     Running            0          161m   10.233.71.11    staging07   <none>           <none>
promtail-qj2dq   1/1     Running            0          162m   10.233.64.133   stagingx    <none>           <none>

Error Pod

[root@stagingx ~]# kubectl logs promtail-qwnwb -n loki
level=debug ts=2020-04-22T05:56:27.702545035Z caller=manager.go:42 msg="detected pipe from stdin"
level=warn ts=2020-04-22T05:56:27.70261709Z caller=stdin_target_manager.go:90 msg="too many scrape configs, skipping 4 configs."
level=info ts=2020-04-22T05:56:27.704016187Z caller=server.go:147 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-04-22T05:56:27.705044385Z caller=main.go:67 msg="Starting Promtail" version="(version=1.4.1, branch=HEAD, revision=2445ccd2)"

Normal Pod

[root@stagingx ~]# kubectl logs promtail-5xqdz -n loki
level=info ts=2020-04-22T05:59:53.446688745Z caller=kubernetes.go:190 component=discovery discovery=k8s msg="Using pod service account via in-cluster config"
level=debug ts=2020-04-22T05:59:53.461332614Z caller=manager.go:224 component=discovery msg="Starting provider" provider=*kubernetes.SDConfig/0 subs="[kubernetes-pods-app kubernetes-pods-direct-controllers kubernetes-pods-indirect-controller kubernetes-pods-static kubernetes-pods-name]"
level=info ts=2020-04-22T05:59:53.461810503Z caller=server.go:147 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-04-22T05:59:53.464532727Z caller=main.go:67 msg="Starting Promtail" version="(version=1.4.1, branch=HEAD, revision=2445ccd2)"
level=debug ts=2020-04-22T05:59:58.446791056Z caller=filetargetmanager.go:209 msg="new target" labels="{__address__=\"10.233.68.226:3306\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"c7ncd-mysql\", __meta_kubernetes_pod_container_port_name=\"\", __meta_kubernetes_pod_container_port_number=\"3306\", __meta_kubernetes_pod_container_port_protocol=\"TCP\"}"
level=debug ts=2020-04-22T05:59:58.447827507Z caller=filetargetmanager.go:227 msg="dropping target, no labels"
level=debug ts=2020-04-22T05:59:58.447859196Z caller=filetargetmanager.go:209 msg="new target" labels="{__address__=\"10.233.71.19:8088\", __meta_kubernetes_pod_container_init=\"false\", __meta_kubernetes_pod_container_name=\"choerodon-cluster-agent-code\", __meta_kubernetes_pod_container_port_name=\"http\", __meta_kubernetes_pod_container_port_number=\"8088\", __meta_kubernetes_pod_container_port_protocol=\"TCP\"}"  ```
......

Promtail config

client:
backoff_config:
  maxbackoff: 5s
  maxretries: 20
  minbackoff: 100ms
batchsize: 102400
batchwait: 1s
external_labels: {}
timeout: 10s
positions:
filename: /run/promtail/positions.yaml
server:
http_listen_port: 3101
target_config:
sync_period: 10s

scrape_configs:
- job_name: kubernetes-pods-name
pipeline_stages:
  - docker: {}

kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
  - __meta_kubernetes_pod_label_name
  target_label: __service__
- source_labels:
  - __meta_kubernetes_pod_node_name
  target_label: __host__
- action: drop
  regex: ''
  source_labels:
  - __service__
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)
- action: replace
  replacement: $1
  separator: /
  source_labels:
  - __meta_kubernetes_namespace
  - __service__
  target_label: job
- action: replace
  source_labels:
  - __meta_kubernetes_namespace
  target_label: namespace
- action: replace
  source_labels:
  - __meta_kubernetes_pod_name
  target_label: instance
- action: replace
  source_labels:
  - __meta_kubernetes_pod_container_name
  target_label: container_name
- replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
  - __meta_kubernetes_pod_uid
  - __meta_kubernetes_pod_container_name
  target_label: __path__
- job_name: kubernetes-pods-app
pipeline_stages:
  - docker: {}

kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
  regex: .+
  source_labels:
  - __meta_kubernetes_pod_label_name
- source_labels:
  - __meta_kubernetes_pod_label_app
  target_label: __service__
- source_labels:
  - __meta_kubernetes_pod_node_name
  target_label: __host__
- action: drop
  regex: ''
  source_labels:
  - __service__
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)
- action: replace
  replacement: $1
  separator: /
  source_labels:
  - __meta_kubernetes_namespace
  - __service__
  target_label: job
- action: replace
  source_labels:
  - __meta_kubernetes_namespace
  target_label: namespace
- action: replace
  source_labels:
  - __meta_kubernetes_pod_name
  target_label: instance
- action: replace
  source_labels:
  - __meta_kubernetes_pod_container_name
  target_label: container_name
- replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
  - __meta_kubernetes_pod_uid
  - __meta_kubernetes_pod_container_name
  target_label: __path__
- job_name: kubernetes-pods-direct-controllers
pipeline_stages:
  - docker: {}

kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
  regex: .+
  separator: ''
  source_labels:
  - __meta_kubernetes_pod_label_name
  - __meta_kubernetes_pod_label_app
- action: drop
  regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
  source_labels:
  - __meta_kubernetes_pod_controller_name
- source_labels:
  - __meta_kubernetes_pod_controller_name
  target_label: __service__
- source_labels:
  - __meta_kubernetes_pod_node_name
  target_label: __host__
- action: drop
  regex: ''
  source_labels:
  - __service__
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)
- action: replace
  replacement: $1
  separator: /
  source_labels:
  - __meta_kubernetes_namespace
  - __service__
  target_label: job
- action: replace
  source_labels:
  - __meta_kubernetes_namespace
  target_label: namespace
- action: replace
  source_labels:
  - __meta_kubernetes_pod_name
  target_label: instance
- action: replace
  source_labels:
  - __meta_kubernetes_pod_container_name
  target_label: container_name
- replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
  - __meta_kubernetes_pod_uid
  - __meta_kubernetes_pod_container_name
  target_label: __path__
- job_name: kubernetes-pods-indirect-controller
pipeline_stages:
  - docker: {}

kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
  regex: .+
  separator: ''
  source_labels:
  - __meta_kubernetes_pod_label_name
  - __meta_kubernetes_pod_label_app
- action: keep
  regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
  source_labels:
  - __meta_kubernetes_pod_controller_name
- action: replace
  regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
  source_labels:
  - __meta_kubernetes_pod_controller_name
  target_label: __service__
- source_labels:
  - __meta_kubernetes_pod_node_name
  target_label: __host__
- action: drop
  regex: ''
  source_labels:
  - __service__
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)
- action: replace
  replacement: $1
  separator: /
  source_labels:
  - __meta_kubernetes_namespace
  - __service__
  target_label: job
- action: replace
  source_labels:
  - __meta_kubernetes_namespace
  target_label: namespace
- action: replace
  source_labels:
  - __meta_kubernetes_pod_name
  target_label: instance
- action: replace
  source_labels:
  - __meta_kubernetes_pod_container_name
  target_label: container_name
- replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
  - __meta_kubernetes_pod_uid
  - __meta_kubernetes_pod_container_name
  target_label: __path__
- job_name: kubernetes-pods-static
pipeline_stages:
  - docker: {}

kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
  regex: ''
  source_labels:
  - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
- action: replace
  source_labels:
  - __meta_kubernetes_pod_label_component
  target_label: __service__
- source_labels:
  - __meta_kubernetes_pod_node_name
  target_label: __host__
- action: drop
  regex: ''
  source_labels:
  - __service__
- action: labelmap
  regex: __meta_kubernetes_pod_label_(.+)
- action: replace
  replacement: $1
  separator: /
  source_labels:
  - __meta_kubernetes_namespace
  - __service__
  target_label: job
- action: replace
  source_labels:
  - __meta_kubernetes_namespace
  target_label: namespace
- action: replace
  source_labels:
  - __meta_kubernetes_pod_name
  target_label: instance
- action: replace
  source_labels:
  - __meta_kubernetes_pod_container_name
  target_label: container_name
- replacement: /var/log/pods/*$1/*.log
  separator: /
  source_labels:
  - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
  - __meta_kubernetes_pod_container_name
  target_label: __path__

crezy8 commented 4 years ago

Same issue to me, I just get start to promtail,kubernetes version 1.13.2, docker version 18.6.1. In my case, all pod turn to CrashLoopBackOff status, kubectl describe pod shows the pod was Terminated by Completed reason, and the Exit Code was 0. The same log message, no error, I just can't figure it out.

I try to chanege pod "command" to sleep 3600 started the pod, execute /usr/bin/promtail -config.file=/etc/promtail/promtail.yaml -client.url=http://loki:3100/loki/api/v1/push manually, it works, process seems started and communicate with loki server

For sure, update docker version to 18.9.x would help.

TimeBye commented 4 years ago

I just upgraded Docker to 18.9.6 on all nodes, and promtail works fine.

[root@stagingx ~]# kubectl get po -n loki -o wide
NAME             READY   STATUS    RESTARTS   AGE     IP              NODE        NOMINATED NODE   READINESS GATES
loki-0           1/1     Running   0          2m18s   10.233.66.156   staging04   <none>           <none>
promtail-7lcmh   1/1     Running   0          2m18s   10.233.71.65    staging07   <none>           <none>
promtail-f74ct   1/1     Running   0          2m27s   10.233.70.198   staging06   <none>           <none>
promtail-jmmpg   1/1     Running   0          2m27s   10.233.69.148   staging05   <none>           <none>
promtail-ltvk5   1/1     Running   0          2m25s   10.233.67.185   staging01   <none>           <none>
promtail-qg4x4   1/1     Running   0          2m22s   10.233.65.161   staging03   <none>           <none>
promtail-rnpfx   1/1     Running   0          2m26s   10.233.64.139   stagingx    <none>           <none>
promtail-wc4mt   1/1     Running   0          2m28s   10.233.68.234   staging02   <none>           <none>
promtail-xgt4z   1/1     Running   0          2m18s   10.233.66.157   staging04   <none>           <none>

vs102000 commented 4 years ago

Same issue here. changing the Deamonset command to sleep 99999 | sleep 999999999 | /usr/bin/promtail .... fixes the topic.

cyriltovena commented 4 years ago

Hey folks !

This is a known issue with 1.4 and 1.4.1 for some docker deamon please use master until we release 1.5

Thanks

grafana / loki

Is there any limits about the version of docker for promtail? I failed to deploy it on 18.3.1 version of docker, and there's no any error logs. #1971