search

guardian / path-manager

Service that manages paths for a domain. Ensures they are unique etc.
2 stars 0 forks source link

change from kinesis logstash to fluentbit for logging #51

Closed twrichards closed 2 years ago

twrichards commented 2 years ago

Co-authored-by: @andrew-nowak Co-authored-by: @Aracho1 Co-authored-by: @akash1810

https://github.com/guardian/editorial-tools-platform/pull/605 must be deployed first.

This PR changes our logging to the safer 'FluentBit' solution (which doesn't affect runtime behaviour of the app, but instead scoops up console output and formats before sending on to central ELK).

NOTE the editorial-tools-bionic-java8 amigo recipe was updated to add cdk-base role (rather than fluentbit role) to piggy-back on the https://github.com/guardian/devx-logs solution (which handles the fluentbit configuration and process)

✅ tested on CODE (in conjunction with https://github.com/guardian/editorial-tools-platform/pull/605)

akash1810 commented 2 years ago

https://github.com/guardian/devx-logs can be used instead here 🎉 @jacobwinch recently added support for the automatic shipment of application logs.

jonathonherbert commented 2 years ago

Is this still required given @akash1810's comment? Looking at the repo, it looks like support for application logs is still in beta, so perhaps it is!

twrichards commented 2 years ago

Is this still required given @akash1810's comment? Looking at the repo, it looks like support for application logs is still in beta, so perhaps it is!

Indeed @jonathonherbert , keen to replace this once 'application logs' feature of devx-logs is GA.

In the meantime, I'm keen to get this closed out soon as its been open for a while.

akash1810 commented 2 years ago

devx-logs shipment of application logs for Play is solid now. We've not GA'd it just yet as we're testing the story for Node apps.

https://github.com/guardian/cdk-playground/pull/198 provides a Play example. The majority of the commits are preparation work (adding log lines, configuring logback to use the JSON encoder, etc). The only other change needed would be to add a tag to your ASG.

I'd be really keen to use it here, if possible?

twrichards commented 2 years ago

This PR now uses https://github.com/guardian/devx-logs solution (thanks @akash1810) - PR description updated accordingly.