schakko
commented
1 year ago We are experiencing a similar issue (file watcher event has been logged by promtail for large log files, pointer in positions.yaml is 0 for large log files, Grafana does not show anything).
Environment:
- Rancher Kubernetes Engine (docker) / RKE1 without SELinux enabled
- Loki w/ Helm chart 5.3.0
- Promtail w/ Helm chart 6.11.0
/etc/docker/daemon.json: not configured - defaultjson-filedriver with unlimited logging is used
Loki's values.yaml:
loki:
auth_enabled: false
analytics:
reporting_enabled: false
commonConfig:
replication_factor: 1
storage:
type: 'filesystem'
server:
log_level: warn
grpc_server_max_concurrent_streams: 0
grpc_server_max_recv_msg_size: 8388608
rulerConfig:
storage:
type: 'local'
local:
directory: /rules
rule_path: /tmp/
alertmanager_url: http://prometheus-alertmanager.telemetry.svc.cluster.local:9093
ring:
kvstore:
store: inmemory
enable_api: true
structuredConfig:
compactor: #https://grafana.com/docs/loki/latest/upgrading/#240
retention_enabled: true
limits_config:
retention_period: 14d
ingestion_rate_mb: 10
ingestion_burst_size_mb: 25
max_streams_per_user: 0
max_global_streams_per_user: 0
per_stream_rate_limit: 10MB
per_stream_rate_limit_burst: 50MB
chunk_store_config:
max_look_back_period: 336h
monitoring:
dashboards:
enabled: false
namespace: grafana
rules:
enabled: false
alerting: false
alerts:
enabled: false
serviceMonitor:
enabled: false
selfMonitoring:
enabled: true
grafanaAgent:
installOperator: true
lokiCanary:
enabled: false
test:
enabled: falsePromtail's values.yaml:
serviceMonitor:
enabled: false
extraVolumes:
- name: journal
hostPath:
path: /var/log/journal
extraVolumeMounts:
- name: journal
mountPath: /var/log/journal
readOnly: true
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/etcd
operator: Exists
effect: NoExecute
config:
clients:
- url: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push
tenant_id: 1
external_labels:
cluster: cl1
snippets:
extraScrapeConfigs: |
- job_name: journal
journal:
max_age: 12h
labels:
job: systemd-journal
path: /var/log/journal
relabel_configs:
- source_labels:
- __journal__systemd_unit
target_label: systemd_unit
- source_labels:
- __journal__hostname
target_label: node
- source_labels:
- __journal_syslog_identifier
target_label: syslog_identifier
pipelineStages:
- docker: { }
# ...Additional clues:
- Querying Promtail's Prometheus API endpoint returns the large log file.
- Querying Loki's
/loki/api/v1/seriesendpoint does no longer show the series belonging to the large log file. - Smaller logs from other containers in the same pod are still scraped by Promtail.
Suspects:
- We are collecting Java/Tomcat logs with exceptions. Maybe we are affected by #2281 and this has something to do with the issue.
- There might be a high load in the cluster so that Promtail is not able to ship the files due to Loki's default policy of denying out-of-order writes.
/cc @thomashoell
liangjunmo
Hello,
Describe the bug Promtail received file watcher event but positions do not increase after a few hours of working. From grafana we are experiencing times without logs
Promtail logs :
Promtail position file :
To Reproduce
Install loki helm chart 5.0.0 on RKE cluster (CRI docker). values.yaml
Install promtail helm chart 6.10.0 values.yaml
resources: requests: cpu: 100m memory: 128Mi
defaultVolumes:
defaultVolumeMounts:
extraVolumes:
extraVolumeMounts:
Expected behavior The logs should be sent in real time and the position file should be increased accordingly.
Environment:
Screenshots, Promtail config, or terminal output
Docker is installed in other partition (
/data/docker).I do not understand if it is a problem of limits, rotate logs, throughput or other. The stack works some hours after pod restarting but not after.
Thanks for your help.