Keep log output path on v1+

[calvinbui]

Description

** READ BEFORE CONTINUING: If your issue is not specific to AWS, please cut a ticket in github.com/kubernetes-sigs/karpenter.

What problem are you trying to solve?

Reference: https://github.com/aws/karpenter-provider-aws/issues/5288#issuecomment-1855229201

• logConfig is currently deprecated and will be removed in v1.
• We run Karpenter on Fargate.
• To forward logs on Fargate, you can create an logging configMap to use AWS' Fluent Bit log forwarder
• Fluent Bit does not support Loki yet (https://github.com/aws/containers-roadmap/issues/1242)
• We run Promtail as a sidecar instead to forward logs to Loki
• Promtail reads logs from the filesystem.
• If logConfig is removed, we can no longer configure log output to the filesystem

How important is this feature to you?

Logging is important for us to see how Karpenter is working and the decisions it has made when debugging.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Promtail reads logs from the filesystem

Can you use the var/log/pods path on the host to grab logs from the filesystem? Stdout and stderr logs should go over to this filepath AFAIK. Is there some restriction that doesn't allow you to scrape these logs on the host?

[jonathan-innis]

Looking at this deeper, I'm realizing that using Fargate would mean that you can't grab logs from the hostpath here, which is why you are indicating that you can't just use the standard stdout/stderr write-streams. I need to look a bit deeper into how Fargate expects that you scrape the logs.

This seems crazy to me that every component would have to support a separate volume output path for this to work. @calvinbui How do you handle this with other components that don't support output paths?

[calvinbui]

How do you handle this with other components that don't support output paths?

We use Fargate to avoid having to spin up a node at the beginning in our cluster just to run Karpenter. All our nodes are created and managed by Karpenter afterwards.

[jonathan-innis]

We use Fargate to avoid having to spin up a node at the beginning in our cluster just to run Karpenter. All our nodes are created and managed by Karpenter afterwards

Right, but I was asking more generally about how other components surface logs when it comes to fargate. I know that most kubernetes controller components don't surface output paths through their environment variable/CLI configuration so (for these components), if there's no way to access the hostPath on the node, I'd be interested to see how other components do log forwarding on Fargate. The callout here is that I'd be surprised if this problem is unique to Karpenter.

[calvinbui]

We use Fargate to avoid having to spin up a node at the beginning in our cluster just to run Karpenter. All our nodes are created and managed by Karpenter afterwards

Right, but I was asking more generally about how other components surface logs when it comes to fargate. I know that most kubernetes controller components don't surface output paths through their environment variable/CLI configuration so (for these components), if there's no way to access the hostPath on the node, I'd be interested to see how other components do log forwarding on Fargate. The callout here is that I'd be surprised if this problem is unique to Karpenter.

For non-fargate hosts/pods, they have promtail, grafana agent or filebeat running as a daemonset to forward logs to their respective servers.

We don't run anything else on fargate to comment any further.

[jonathan-innis]

Got it. I'll pull-in some Fargate people into this conversation. I'm hoping there's a workaround to this problem outside of having to configure custom outputPaths in the logger.

[timbgn]

We are also looking at this currently. Karpenter is spun out on Fargate and then our EKS cluster nodes are handled from there. We already have a logging stack using fluentbit and elastic running on our kubernetes clusters so would prefer to make use of those rather than breakout in to cloudwatch. I was hoping to make use of the logconfig options to output to a file we can then scrape via a sidecar and push in to elastic running in cluster.

[aston-r]

I was hoping to make use of the logconfig options to output to a file we can then scrape via a sidecar and push in to elastic running in cluster.

+ 1 native fluent-bit in fargate does not support loki, so I have to write to file and deploy flb or promtail as sidecar container.