Logging splits up multiline log messages

[kstephan-wescale]

Log entries spanning multiple lines like stack traces are split up and end up as multiple oneline entries in the Logs Explorer. Fluentbit provides the “Multiline” filter to deal with that, so I propose to implement it and add a filters element to the configuration model.

This is not a duplicate of #239: This issue requests support of GCP Cloud Tracing whereas I’m asking for proper treatment of stack traces for GCP Logging.

[msoderberg]

225 that was included in the 2.5.0 release looked promising, but after looking into it more I realized that it's only for the apps at the moment. It's mentioned here: https://github.com/GoogleCloudPlatform/ops-agent/pull/225#pullrequestreview-770655339

[wlatic]

+1 on this. I believe Stackdriver is going EOL as of early next year and to convert we need multiline support.

[sophieyfang]

Yes, this is currently in our roadmap. Hopefully, it can be delivered early 2022. Stay tuned!

[wlatic]

Going to be using the agent for another set of logs in the next week or so. Is there any rough time frame for this currently?

[sophieyfang]

Yes, we plan to have it released in early 2022 ：）

[alogishetty]

Hey, any updates on these? We are also looking for multiline support for our logs.

[singhsaurabh20]

Hi Can you please provide any updates on this?

[jbrunsek]

Was the solution to this the parse_multiline processor from here?

Is there a solution where it's not explicitly looking for exceptions from specific languages? I have cases where logs are printed locally from multiple sources that span multiple lines and the parse_multiline processor does not pick these up.

[braydonk]

parse_multiline currently only supports Java, Python, or Go exceptions/stack traces. The Ops Agent currently doesn't support any other/custom formats.

[jbrunsek]

Is there any plans to allow for multiline support outside of exceptions? For example, apt history.log prints multiline and currently gets split up when sending to Google Logging.

Without this support, I feel like Ops Agent isn't feasible for my needs and I'd need to revert back to google-fluentd, which we're being pushed away from with all the notifications in the GCP UI.

[sophieyfang]

Right, ops-agent has not yet supported customizable multiline configuration. Internally there's a tracking bug. We will notify in this thread when we get to it.

google-fluentd uses https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions#optional- Which doesn't support customized multiline either.

[lamsorsen]

I would love this feature. For fluentd I use this: https://docs.fluentd.org/parser/multiline which works really well. I'm dealing with someone else's gcp ops agent setup right now and I'm really missing the ability to aggregate logs the way I do in fluentd.

[rpereira-1]

Is there a workaround? We have some software that reports multiline logs and we need to convert it to single line. Can we run both legacy logging agent and ops-agent on the same vm? This way the legacy agent only needs to be responsible for forwarding the multiline logs.

[igorpeshansky]

Note that we have no current plans to EOL the standalone ("legacy") agents yet. We are tracking the feature gaps in the Ops Agent, and do plan to address them, so if the Ops Agent does not yet fit your needs, please let us know and feel free to continue using the standalone agents for now. We encourage you to follow the Ops Agent releases, however, so that you can switch when the feature becomes available.

It's not currently possible to run the standalone agents and the Ops Agent on the same VM. We are investigating solutions, but nothing concrete yet. Stay tuned.

I should also mention that some of the Ops Agent third-party application logging receivers do handle application-specific multiline formats. If there is an application that routinely writes multiline logs (e.g., apt), please file a feature request for a receiver that handles it — that way you don't need custom multiline configuration.

[rpereira-1]

Thanks for the response.

How can I uninstall the OpsAgent from a VM? Or how can I prevent the OpsAgent from running on the vm so that the legacy agent can run?

Yes, I saw that there are application specific multiline formats but they do not meet our needs. We have a custom application with various multiline formats(core logs, audit logs, etc.). I would really like a feature that is similar to the multiline format for fluentd: https://docs.fluentd.org/parser/multiline. This allows us the freedom to specify our own multiline formats.

[alvaro-gh]

I'm at the same spot, we have some logs coming from groovy and I can't get them parsed with the multiline processor.

[wlatic]

To me it seams madness they swapped out a fully working product to one without basic support.

We are at the point of moving away from Google Logging based on this.

[igorpeshansky]

The Ops Agent is not (and not designed to be) a drop-in replacement for the standalone agents. While we encourage you to use it when it fits your use case, we continue to support the standalone agents. You can follow the public documentation for uninstalling the Ops Agent.

@mxdlx Supporting groovy exceptions is a reasonable use case — if you have an example of a multiline logged groovy exception that isn't parsed properly by the Java rules, please feel free to open a separate issue/support case so that we could investigate adding support for that. As previously mentioned, we currently have no timeline for supporting non-exception multiline logs.

Also note that official support for the Ops Agent (and the standalone agents) is primarily provided via Google Cloud Support. While GitHub issues are regularly triaged, any responses here are on a best effort basis. If you need regular updates on bugs or feature requests, it's best to file a Cloud support case.

[alvaro-gh]

@igorpeshansky I appreciate your response, we're not looking for exceptions at the moment, some users use groovy and have some custom logs that include hive queries directly written to the log. I understand my use case requires going for something else. Thanks once again.