I've created a standardized library for python logging at https://github.com/CanDIG/candigv2-logging that unifies the formatting and configuration for most of our python-based modules. By switching the python-based modules to the same format, we can use fluentd's configuration to process the messages to a more standardized JSON format.
For now, I've written separate filters to process flask and Django logs, tyk logs, and opa logs, but we can write more as needed. Alternatively, we may want to write a custom plugin (https://docs.fluentd.org/plugin-development/api-plugin-parser) to handle the parsing of our custom log format more thoroughly.
To test: check out this branch and make sure that the submodules are up to date with the commits in this branch. Then if you do a clean-all/build-all, you should start seeing logs accumulating in a buffer log file in tmp/logging, and they should roll over every day to a new file.
I took a quick look, and it seems we’ll need to modify Katsu so it can run without the stack. Currently, the logging module is required even for local use. I can write the code when we make the PR for Katsu, but I wanted to bring this up for clarity.
This looks like the old version of fluentd.conf...can you make sure that you've deleted any relevant volumes and images to see if you can trigger it to pick up the new version.
I've created a standardized library for python logging at https://github.com/CanDIG/candigv2-logging that unifies the formatting and configuration for most of our python-based modules. By switching the python-based modules to the same format, we can use fluentd's configuration to process the messages to a more standardized JSON format.
For now, I've written separate filters to process flask and Django logs, tyk logs, and opa logs, but we can write more as needed. Alternatively, we may want to write a custom plugin (https://docs.fluentd.org/plugin-development/api-plugin-parser) to handle the parsing of our custom log format more thoroughly.
To test: check out this branch and make sure that the submodules are up to date with the commits in this branch. Then if you do a clean-all/build-all, you should start seeing logs accumulating in a buffer log file in tmp/logging, and they should roll over every day to a new file.