Pipeline stage.multiline not working for all multiline-logs

[Uscholdo]

Describe the bug We are using the flow-mode configuration in our grafana-agent. To compromise multiline-logs into one log-message that is displayed in grafana we use the "stage.multiline" block in our flow-mode configuration. Unfortunatly not all matching "first lines" are used to add the following logs to a compromised multiline-log.

As you can see in the screenshots, there is one multiline-log compromised and the other(at the top of the picture) isn't and is shown in multiple lines. The regex that is used in the stage.multiline matches both first-lines (already tested), so why is one multiline-log compromised and the other isnt?

To Reproduce Steps to reproduce the behavior:

1. Started Loki (version 2.9.3)
2. Started grafana-agent
3. Query logs in grafana

Expected behavior All multi-line logs where the first line matches the configured regex should be displayed as multiline logs in grafana.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output Grafana-agent configuration (flow mode):

      stage.match {
        selector = "{namespace=~\".*mfc.*\"}"           

        stage.multiline {
          firstline     = "^(\\[[^\\]]*\\] ){4}"
          max_lines     = 10000
          max_wait_time = "5s"
        }

        stage.regex {
          expression = "^\\[(?P<time>[^\\]]*)\\] \\[(?P<level>[^ \\]]*) *\\] \\[(?P<threadId>[^ \\]]*) *\\] \\[(?P<logger>[^\\]]*)\\] ( ?\\[(?P<threadExecutionContext>[^\\]]*)\\])? ?(?P<message>[\\S\\s]*)"
        }

        stage.replace {
          source     = "filename"
          expression = "^\\/var\\/log\\/pods\\/.*\\/.*\\/(.*).log"
          replace    = "*"
        }

        stage.labels {
          values = {
            filename = "",
          }
        }

        stage.timestamp {
          source = "time"
          format = "RFC3339Nano"
        }

        stage.structured_metadata {
          values = {
            level                  = "",
            logger                 = "",
            message                = "",
            threadExecutionContext = "",
            threadId               = "",
          }
        }
      }

Grafana log output:

Raw logs of the first-line (+2 following lines) of the multiline-log that is NOT compromised by stage.multiline:

2024-03-22 08:13:06.960 at System.Environment.FailFast(System.String, System.Exception) 2024-03-22 08:13:06.959 Process terminated. Last breath finished 2024-03-22 08:13:06.959 [2024-03-22T07:13:06.9595308+00:00] [WARN] [1 ] [XXXXX.Systems.Exceptions.CorruptedStateTools] Last breath finished - FailFast

Raw logs of the first-line (+3 following lines) multiline-log that is compromised by stage.multiline:

2024-03-22 08:13:06.785 [2024-03-22T07:13:06.7854790+00:00] [INFO] [1 ] [TGW.Systems.Logging.LoggingExtensions] Configure Logging for container environment (using NativeConsoleRuntime) Starting ApplicationLoader CustomMfcApplicationLoader Error received: System.InvalidOperationException : No service for type 'TGW.Systems.Localization.ITranslator' has been registered. at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)

[Uscholdo]

For testing I added this temporarly to test whether the regex is correct for both "first lines": stage.replace { expression = "^(\\[[^\\]]*\\] ){4}" replace = "*****" }

The outcome shows that both first lines match the regex, so this seems not to be the problem.