Parsing issues with huge HAProxy config file in Kubernetes

[senattcsgit]

We use grok heavily on VM's to parse HAProxy logs and we never had any issues but recently started migrating this work loads to Kubernetes and experiencing that grok exporter is not reading the HAProxy logs, when the HAProxy config file has more than 10,000 lines. but same grok config, exporter works well when we reduce that 10,000 lines config to 100 lines or so...

wondering what is the link between HAProxy config file and grok not to work? Does grok need some extra memory or any specific size requirements for POD/container? Note: grok and haproxy starts as two separate processes on same pod. please help us!

Docker File:

---

FROM haproxy:1.8.19 LABEL Name="grok_exporter" LABEL Version="v1.0.0.RC3" ENV GROK_ARCH="grok_exporter-1.0.0.RC3.linux-amd64" ENV GROK_VERSION="v1.0.0.RC3" RUN apt-get update -qqy \ && apt-get upgrade -qqy \ && apt-get install rsyslog --no-install-recommends -qqy \ wget unzip ca-certificates \ && update-ca-certificates \ && wget https://github.com/fstab/grok_exporter/releases/download/$GROK_VERSION/$GROK_ARCH.zip \ && unzip $GROK_ARCH.zip \ && mv $GROK_ARCH /grok \ && rm $GROK_ARCH.zip \ && apt-get --autoremove purge -qqy \ wget unzip ca-certificates \ && rm -fr /var/lib/apt/lists/*

LOGROTATE

RUN mkdir -p /etc/rsyslog.d/ \ && mkdir -p /opt/logs \ && mkdir -p /var/prod/haproxy/pid \ && mkdir -p /var/prod/haproxy/stats \ && mkdir -p /var/prod/haproxy/cfg/hk \ && touch /opt/logs/hap.log

RUN mv grok /etc EXPOSE 9000 EXPOSE 8808

GROK_PORT

EXPOSE 9142

LOGROTATE

COPY rsyslog.d_haproxy.conf /etc/rsyslog.d/haproxy.conf COPY logrotate.d_haproxy /etc/logrotate.d/haproxy

GROK

COPY grok_config.yaml /etc/grok/config.yml COPY grok_hap_pattern /etc/grok/patterns/haproxy COPY startup_file . COPY haproxy.cfg /var/prod/haproxy/cfg/haproxy.cfg COPY healthcheck.http /var/prod/haproxy/cfg/hk/healthcheck.http

CMD STARTUP File

CMD sh startup_file *End of Docker*****

startup_file: echo "Restarting rsyslog service" service rsyslog restart echo "Starting HAProxy" haproxy -f /var/prod/haproxy/cfg/haproxy.cfg & echo "Starting GRok Exporter" cd /etc/grok pwd ls -lrt ./grok_exporter -config /etc/grok/config.yml echo "Grok Started Sucessfully" *end of startup file*****

[fstab]

Please tell me you the input section in your grok_config.yaml is configured.

[senattcsgit]

input: type: file path: /home/haproxy/logs/hap.log readall: false # Read from the beginning of the file? False means we start at the end of the file and read only new lines.

[fstab]

This is weird. If you configure readall: false, grok_exporter starts reading from the end of the log file. It should not matter if the file has 10,000 lines or 100 lines, because grok_exporter skips right to the end and reads only new lines appended to the file, ignoring all existing lines.

The first thing to investigate should be if this is really related to grok_exporter or if there is another problem with reading the file. Could you in your startup file replace

./grok_exporter -config /etc/grok/config.yml

with

tail -F /home/haproxy/logs/hap.log

and check if the Docker container writes all newly appended log lines to the console output? Checking if tail -F works is a good way to make sure that there is nothing that prevents reading newly appended lines from the file.

If tail -F can read newly appended lines but grok_exporter cannot, could you check if

• grok_exporter writes any errors to the console
• check grok_exporter's built-in metrics for any hints that something is wrong (processing time, buffer load, etc)

I hope this will help narrow this down.