Add fluentbit logging support, sending logs to ELK

[rhystmills]

What does this change?

This send our logs to ELK via FluentBit. This is quite simple now that DevX provide FluentBit in their cdk-base AMI role. This PR moves to an ami that include CDK-base and adds the minimal riff-raff changes required to support FluentBit logging. The commented guidance in the cdk repo was helpful.

Logs appear in ELK under typerighter-rule-manager and typerighter-checker&_a=(columns:!(app,message,log),filters:!(),index:'67604290-baa6-11e9-bea2-633437abb232',interval:auto,query:(language:kuery,query:'app.keyword%3Dtyperighter-rule-manager%20OR%20app.keyword%3Dtyperighter-checker'),sort:!(!('@timestamp',desc)))).

I've deployed to CODE and seen logs for the Rule Manager and Checker:

Logs for Rule Manager	Logs for Checker

How to test

Deploy to CODE and check ELK&_a=(columns:!(app,message,log),filters:!(),index:'67604290-baa6-11e9-bea2-633437abb232',interval:auto,query:(language:kuery,query:'app.keyword%3Dtyperighter-rule-manager%20OR%20app.keyword%3Dtyperighter-checker'),sort:!(!('@timestamp',desc)))) for Typerighter logs.

[jonathonherbert]

Nice, such a win for gu-cdk! 🎉

If we're doing this for both apps, let's remove the now-redundant ELK logging from the checker app (and any related dependencies) and verify that checker is still logging – a nice cathartic cleanout 😀

[rhystmills]

Nice, such a win for gu-cdk! 🎉

If we're doing this for both apps, let's remove the now-redundant ELK logging from the checker app (and any related dependencies) and verify that checker is still logging – a nice cathartic cleanout 😀

Ah, didn't know that was a thing. Good plan.

[rhystmills]

That's now gone, deployed to CODE and logs for checker were still showing up in ELK. Will look into dependencies on Monday.

[jonathonherbert]

Nice! We're almost there, I've taken a look at the logs, and I think they're being parsed in a slightly odd way – we're including some meta in the message field, and we're missing some logging markers too – compare e.g. this log entry from main to one from this branch.

Suspect it's a formatting thing. Taking a look at another app that uses applicationLogging, I wonder if adding a LogstashEncoder to our stdout appender might help?

Worth noting that this is not clear from the CDK docs at the moment! Once we've confirmed a fix, it'd be good to update them to make it clear for all users that a given format is expected at the point at which you're setting the flag in CDK.

[rhystmills]

Good spot

Nice! We're almost there, I've taken a look at the logs, and I think they're being parsed in a slightly odd way – we're including some meta in the message field, and we're missing some logging markers too – compare e.g. this log entry from main to one from this branch.

Suspect it's a formatting thing. Taking a look at another app that uses applicationLogging, I wonder if adding a LogstashEncoder to our stdout appender might help?

Worth noting that this is not clear from the CDK docs at the moment! Once we've confirmed a fix, it'd be good to update them to make it clear for all users that a given format is expected at the point at which you're setting the flag in CDK.

Good spot Jon, I've updated those logback.xml files now - will deploy to CODE and check the logs to see if they're looking right.

[rhystmills]

Looking better now with logs from CODE&_a=(columns:!(app,message,log),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'67604290-baa6-11e9-bea2-633437abb232',key:stage,negate:!f,params:(query:CODE),type:phrase),query:(match_phrase:(stage:CODE)))),index:'67604290-baa6-11e9-bea2-633437abb232',interval:auto,query:(language:kuery,query:'app.keyword%3Dtyperighter-rule-manager%20OR%20app.keyword%3Dtyperighter-checker'),sort:!(!('@timestamp',desc)))) including the additional fields.