Open kiranmeduri opened 4 years ago
Discussing this in our weekly triage, we've come up with an initial set of action items here
@PettitWesley can you provide update on this issue. Thanks
@kiranmeduri Current testing shows that the CloudWatch Fluent Bit plugin now works with App Mesh.
Root cause and remediate bugs when launching an ECS task with Firelens and App Mesh
The bugs seem to all be remediated.
Provide a reference example showing how to configure FireLens with App Mesh
@CarmenAPuccio has this piece.
@kiranmeduri - The blog went live yesterday and we have the walkthroughs for EKS and ECS on Fargate/FireLens.
I can add links to those repos and the blog in aws/aws-app-mesh-examples. Would you just want a folder under /examples called fluent-bit?
There is one open question here. Is firelens traffic flowing through Envoy? If so it should not because it is actually monitoring Envoy. I would like to see if fluentbit traffic can bypass Envoy. Today it is done by setting User:1337 on container. But AFAIK, that is not allowed with Firelens container. Please confirm @PettitWesley.
@kiranmeduri Yeah, with how things work right now, the UID for the FireLens container has to be 0.
Yep just confirmed. If you try and set the user field on the FireLens log router you get this:
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: If 'user' field is specified on firelens container, then 'UID' has to be '0'.
I think the GID can be anything though- is there a way you can set that to bypass envoy?
We had issues with FireLens + AppMesh + using a output other than CloudwatchLogs (in our case ElasticSearch).
We tried a couple of things but the only thing that worked (thanks @PettitWesley ):
IgnoredGID
to 1337
in the proxyConfiguration
of the task-definition1337:1337
0:1337
It would be great though to know why this was necessary. It almost seems like AppMesh was interfering with the traffic between the Fargate Host and the logrouter?
@lifeofguenter We also hit this issue. The mentioned solution works only when using the Fargate platform version 1.3.0
— once we switch to 1.4.0
logging breaks without obvious reason. log_router/fluent bit does not log anything after the bootup process.
@thisismana our solution works for us with 1.4.0
I confirm that the solution works. @thisismana make sure that you have only IgnoredGID in the proxy configuration. It was not working for me at first because I had both IgnoredUID and IgnoredGID. Thank you @lifeofguenter for this solution.
@kamilhristov nicely spotted. We set both IgnoredGID
and IgnoredUID
and it did not work (failing silently). But setting only IgnoredGID: 1337
with uid:gid
for envoy
as 1337:1337
and fluentbit
as 0:1337
did the trick.
I'm so grateful ❤️
@thisismana @kamilhristov @lifeofguenter @thisismana What endpoints is FireLens sending data to? AWS endpoints? VPC endpoints? Public endpoints (ex datadog)?
I'm trying to figure out if setting IgnoredUID
is always required with FireLens or if it depends on what endpoint FireLens needs to talk to.
@PettitWesley in our case we were forwarding logs to an internal alb.
We are forwarding to AWS endpoint - Kinesis Firehose.
Tell us about your request What do you want us to build? As a user, I want to enable access-logs in Envoy via App Mesh and have those logs be published to sinks supported by Fluentd and Fluent Bit.
AWS recently announced firelens that can be used to achieve this, but there is no documentation or recipes on how to set this up.
Which integration(s) is this request for? Any
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Stream Envoy access-logs to fluentd supported destinations.