search

netobserv / flowlogs-pipeline

Transform flow logs into metrics
Apache License 2.0
77 stars 24 forks source link

Debuggin and tracing write Loki setup and connection #482

Open aslom opened 1 year ago

aslom commented 1 year ago

Is there a flag or some other way to see DEBUG/ or TRACE log with connection to Loki and what is sent there?

Right now I have simple pipeline with 

  - name: write_loki
    write:
      type: loki
      loki:
        url: http://localhost:3100
        staticLabels:
          job: flowlogs-pipeline

and it runs without errors (and no output) even if there is nothing on localhost:3100 ( checked with curl -v localhost:3100/ready fails)

@ronensc @eranra

ronensc commented 1 year ago

When setting the log level to trace, FLP emits a log message for every flowlog passing through the WriteLoki stage. https://github.com/netobserv/flowlogs-pipeline/blob/25d78d701c64f1cbc7ff701950c713b69a955bc5/pkg/pipeline/write/write_loki.go#L209 FLP uses loki-client-go to send the flowlog to Loki. loki-client-go is async. It waits for a full batch or for some timeout to expire before actually sending the flowlogs to loki. I couldn't find any trace log messages emitted when the actual sending is happening. A PR against loki-client-go should be submitted to add it.

You probably don't see anything on localhost:3100 because FLP exits before loki-client-go has sent the flowlogs to Loki. As a quick and dirty hack, You may increase the waiting time to 10 seconds. https://github.com/netobserv/flowlogs-pipeline/blob/665de17e78e6fcc469903e7daef9a0ce65d92834/cmd/flowlogs-pipeline/main.go#L228

A more proper solution would be to make FLP wait for all in-processing stuff to be finished before exiting.