auto_multi_line_extra_patterns is not supported

[sullerandras]

The logs_config.auto_multi_line_extra_patterns option is described in this doc but it looks like it's not supported by the helm chart. Would it be possible to add support for this?

[clamoriniere]

Hi @sullerandras

Is it difficult to add an option in the chart for each possible option in the agent. When an option is not available the recommended solution is to add it thank to the environment variable.

it the values.yaml file

datadog:
  env:
  - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
    value: "true"

please let me know if it solve the log agent configuration issue.

[vrej-abramian]

Hi @clamoriniere

Is it difficult to add an option in the chart for each possible option in the agent. When an option is not available the recommended solution is to add it thank to the environment variable.

it the values.yaml file

datadog:
  env:
  - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
    value: "true"

please let me know if it solve the log agent configuration issue.

I have used this parameter and value just as you mentioned, this helped to enable the auto detection of multiline logs with the built-in patterns.

For my case I need logs_config.auto_multi_line_extra_patterns to work as expected, to define specific patterns to be detected (outof the box) for some multiline logs. I have tested logs_config.auto_multi_line_extra_patterns for my multiline logs but it did not work (seems has no impact at all).

P.S. Using datadog agent deployed by helm (chart version: datadog-2.30.16).

[olenirwing]

@sullerandras did you get it to work by setting DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS env variable? Not working for me.

[clamoriniere]

Hi @sullerandras Indeed you can set the logs_config.auto_multi_line_extra_patterns thanks to env variable. If you want to add several patterns it is possible to do using the " " separator between the patterns.

[vrej-abramian]

Hi @clamoriniere , Would you please help? When I have set DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true by setting the following in the helm values.yaml file:

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
  logs_config:
    auto_multi_line_detection: true

Unfortunately I have java multiline logs like starts with 23-03-2022 blah-blah.. but these logs next lines are not submitted as a single log :(

I've tried to add the EXTRA_PATTERNS as ENV variables as you mentioned (see bellow code) but from inside the container the I got MULTI_LINE_DETECTION value changed to false: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=false

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS"
      value: ^23-03.*  ### also tried with quoted "^23-03.*"
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"  ### also tried without quotes
... omitted lines ...
  logs_config:
    auto_multi_line_detection: true

[clamoriniere]

Hi @vrej-abramian

I was able add the extra patterns as env var thanks to the single quote: ``

agents:
  containers:
    agent:
      env: # to only set the env var to the container responsible of logs.
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
        value: "true"
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS
        value: '^23-03.*'

the agent status output:

$ k exec -it dd-datadog-2x65r -c agent -- agent config
...
logs_config:
  ...
  auto_multi_line_default_match_threshold: 0.48
  auto_multi_line_default_match_timeout: 30
  auto_multi_line_default_sample_size: 500
  auto_multi_line_detection: true
  auto_multi_line_extra_patterns:
  - ^23-03.*

[vrej-abramian]

Hi @vrej-abramian

I was able add the extra patterns as env var thanks to the single quote: ``

agents:
  containers:
    agent:
      env: # to only set the env var to the container responsible of logs.
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
        value: "true"
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS
        value: '^23-03.*'

the agent status output:

$ k exec -it dd-datadog-2x65r -c agent -- agent config
...
logs_config:
  ...
  auto_multi_line_default_match_threshold: 0.48
  auto_multi_line_default_match_timeout: 30
  auto_multi_line_default_sample_size: 500
  auto_multi_line_detection: true
  auto_multi_line_extra_patterns:
  - ^23-03.*

Thanks @clamoriniere for fast reply, The DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS parameter is taken properly but even with or without EXRA_PATTERNS the problem is that from within the container I see auto_multi_line_detection: false so the multiline logs detection not working, any idea how can that possible?

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
  logs_config:
    auto_multi_line_detection: true

[vrej-abramian]

Thank you @clamoriniere and every one else here, I've got the fix, while using DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION env variable, logs_config.auto_multi_line_detection should be commented or removed, by this now I can see from within the container that the auto_multi_line_detection: true is set to true and multiline logs are automatically detected (yet at least) :)

# Under datadog:
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
##  logs_config:            # Removed
##     auto_multi_line_detection: true   # Removed

[clamoriniere]

hi again @vrej-abramian Sorry about that... I think if found why you have the issue but not me. let me explain: You are setting the env var with datadog.env, but on my side i used agents.containers.agent.env. I also discover that the option datadog.logs.autoMultiLineDetection exist and it is set to false by default, and this option is set thank to the envvar. But in the Container spec, we first add datadog.env then datadog.logs.autoMultiLineDetection and finally agents.containers.agent.env. In the ContainerSpec.env we can set several time the same env var, but the last one win.

that is why adding DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION in datadog.env was override by the value from datadog.logs.autoMultiLineDetection.

For now what you can do is:

datadog:
  logs:
    enabled: true
    #...
    autoMultiLineDetection: true

agents:
  containers:
    agent:
      env: 
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS
        value: '^23-03.*'

And it should work. I will fill file a card to improve the logic in the chart. to not add DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION if the value is false.

please let me know if it finally works for you.

[vrej-abramian]

    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"

Hey @clamoriniere, No worries, thank you for the clarification. Well lets forget about the DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS for a moment because auto_detection, detects my logs, I'll use EXTRA_PATTERNS maybe latter if I need to. But the important update that I can share with you is the :

This did not work ( from within container the multi_line_detection is not enabled auto_multi_line_detection: false )

By setting the datadog.logs.autoMultiLineDetection to true. I have also added DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION to agents.containers.agent.env like the following but that doesn't help.

  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"

So I've finally get back to my last working configuration which is as follows:

This worked for me to have auto_multi_line_detection: true within the container and multi_line logs detection works without extra patterns configured.

### Under datadog.env config block.
  env:
    - name: "DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION"
      value: "true"
... omitted lines ...
### Under datadog.logs_config block
##  logs_config:            # Removed
##     auto_multi_line_detection: true   # Removed
... omitted lines ...
### under agents.containers.agent.env block
  env: []

UPDATE 1

@clamoriniere

Same configuration that worked in one of our clusters, does not work in another cluster, I have no idea about this (((

Update 1.1

I have tried previously discussed configuration combination, but none of them worked on this second cluster.

Update 1.2

@clamoriniere I wanted to update that I was using wrong parameter datadog.logs_config.auto_multi_line_detection instead of datadog.logs.autoMultiLineDetection which you have suggested. Probably I was tired and did not notice the difference between them. By this my issue is resolved. So thanks again ;)

[sullerandras]

Thanks for all the replies. I set the env variable but i'm not sure what should i set it to. Our application is running in a kubernetes cluster, and we log using JSON. Mostly works fine, however, long json logs are cut at 8192 bytes, and the rest is going on a new line. This must be a line length limit somewhere, i'm not sure where. So basically all log lines start with {, except when it's split into two lines. I tried enabling multi line detection and set DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS: "{". It worked for the first split log line when i tested this, and i was very happy that i was able to solve this problem. However it never worked for any subsequent tests, it only merged the split log lines from the first occurrence. What am i doing wrong? Any help is appreciated.

[keatsfonam]

With latest 2.32.3: I have found that setting datadog.logs.autoMultiLineDetection: true is being overriden in the config unless also setting the env var DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true when I am using the workaround for DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS

Setting only one of DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true or datadog.logs.autoMultiLineDetection: true results in auto_multi_line_detection: false at runtime for some reason.

[vrej-abramian]

With latest 2.32.3: I have found that setting datadog.logs.autoMultiLineDetection: true is being overriden in the config unless also setting the env var DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true when I am using the workaround for DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS

Setting only one of DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION=true or datadog.logs.autoMultiLineDetection: true results in auto_multi_line_detection: false at runtime for some reason.

@keatsfonam good finding, I did not used the DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS yet, but I'll consider your pointing if some day I need to use extra patterns. Thanks for sharing.

[iahmad-khan]

Hi, I am having problem with how to provide the value field for DD_LOGS_CONFIG_AUTO_MULTI_LINE_EXTRA_PATTERNS in case there are multiple patteerns. thanks.

[iahmad-khan]

Also, what should be the patterns for log lines like these?

/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:106 +0x15d
github.com/go-chi/chi/middleware.prettyStack.decorateLine({}, {0xc000247985?, 0xc000509800?}, 0x9f?, 0x1?)

/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:130 +0x545
github.com/go-chi/chi/middleware.prettyStack.decorateFuncCallLine({}, {0xc000247985, 0x1c}

[vrej-abramian]

Also, what should be the patterns for log lines like these?

/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:106 +0x15d
github.com/go-chi/chi/middleware.prettyStack.decorateLine({}, {0xc000247985?, 0xc000509800?}, 0x9f?, 0x1?)

/go/src/recording/vendor/github.com/go-chi/chi/middleware/recoverer.go:130 +0x545
github.com/go-chi/chi/middleware.prettyStack.decorateFuncCallLine({}, {0xc000247985, 0x1c}

@iahmad-khan, assuming that all your logs start with /go, and using the relevant environmental variable in the datadog-values.yaml file deployed by helm in K8S environment, have you tried such pattern?

• - [/]go

Kind reminder: Please consider using a test environment and test configurations for specific issues/cases.

[parasappafluke]

Hi, I have enabled datadog.log.autoMultiLineDetection: true and the agent config command shows below output

logs_config:
  auto_multi_line_default_match_threshold: 0.48
  auto_multi_line_default_match_timeout: 30
  auto_multi_line_default_sample_size: 500
  auto_multi_line_detection: true

But still the exception logs are not coming in single line for java springboot application. Date format for logs is yyyy-MM-dd hh:mm:ss.SSS andyyyy-MM-dd hh:mm:ss

agent: 7.37.1-jmx helm chart: 2.36.0

Thanks Parasappa

[dyleph]

Hi @clamoriniere and @vrej-abramian , Currently I have the strange issue after setting DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION env variable to allow the datadog agent to aggregate multi lines logs.

Helm chart v3.3.3 Agent 7.40.1

Here is my scenario with step by step:

1. Adding autoMultiLineDetection: true in values.yaml

   datadog:
    logs:
      enabled: true
      autoMultiLineDetection: true

Then upgrade datadog-agent by helm command. Result: Logs are not aggregated as single line as expected -> KO

This is my agent config:

 logs_config:
   ...
   auto_multi_line_default_match_threshold: 0.48
   auto_multi_line_default_match_timeout: 30
   auto_multi_line_default_sample_size: 500
   auto_multi_line_detection: false
  ...

2. Then I try adding DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION: true as an environment parameter under datadog as you guys did, and disabling autoMultiLineDetection: true in the previous (1)

   datadog:
    logs:
      enabled: true
      #autoMultiLineDetection: true
    env:
      - name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION
        value: "true"

Then upgrade datadog-agent again. Result: Logs are aggregated as single line as expected -> OK I re-configured (1) and (2) in many times to make sure (2) is a right config.

This is my agent config:

 logs_config:
   ...
   auto_multi_line_default_match_threshold: 0.48
   auto_multi_line_default_match_timeout: 30
   auto_multi_line_default_sample_size: 500
   auto_multi_line_detection: true
  ...

3. Now, when merging this change to main. I start upgrading on other cluster agents on different environments and also re-upgrading on this working cluster agent. But logs are not affected after the upgrade. It's really strange, not sure what's happened. :(

[clamoriniere]

hi @dyleph

I was not able to reproduce the issue that you describe.

Could you check when it happens that the value is properly set in helm get values <release-name>

and also in the daemonset resource that with only datadog.logs.autoMultiLineDetection:true set. the envvar DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION is added to the agent container with the value "true"?

thanks 🙇

[vrej-abramian]

Hi @dyleph, @clamoriniere, Well to be honest I wish I could try to reproduce and dig deep in your case, but I won't be able soon, so I'll going to share from one of my experiences, hopping it will be useful.

If you want to get rid of the multiline-logs issue forever, the shortest, best, and official resolution from Datadog (which I tested and satisfied) is to logging-out to JSON format from your Application level if possible.

[dyleph]

Hi @clamoriniere and @vrej-abramian, Thank you for your reply.

ATM, the logs are aggregated as well on datadog agent after I set both configs: datadog: logs: enabled: true autoMultiLineDetection: true env:

• name: DD_LOGS_CONFIG_AUTO_MULTI_LINE_DETECTION value: "true"

About configure JSON logging from the application, I did try by writing consoleLog and fileLog with JSON format.

[babbottscott]

Try removing the ^ from your pattern. The code that processes these strings prepends one for you (which feels a bit bad form...).