I have noticed that the logs we currently capture can often be noisy, obscuring its usefulness. I might be beneficial to incorporate some additional structure into our logging system.
There are two scenarios we can consider:
In a situation where all steps execute successfully, a concise summary of the actions that were performed, instead of an extensive log output, could be more practical.
When some steps end up failing, we would just print debug log messages related to the final failed step, instead of all log entries.
I have not been able to find any existing library or reference implementation that could be used to realize this. Python tools like Celery, Luigi, and Airflow don't seem to solve this specific problem.
Developing such a feature would be potentially time-consuming and thus, likely falls outside the scope of this project.
However, there are more simple alternatives we could try. For example:
We could save log outputs to separate files as opposed to directly to stdout. So each step of the process would have a corresponding log file, such as:
step1_k8s-cluster-create.debug.log
step2_k8s-create-deployment.debug.log
step3_switch-create.debug.log
If a command fails, we would fetch the specific step's logs and output it to stdout. In addition, we could keep the metadata (e.g, cluster IP, switch IP, etc.) within a structured file (a json file for example), and then output the concise necessary details once the steps have been completed.
I have noticed that the logs we currently capture can often be noisy, obscuring its usefulness. I might be beneficial to incorporate some additional structure into our logging system.
There are two scenarios we can consider:
I have not been able to find any existing library or reference implementation that could be used to realize this. Python tools like Celery, Luigi, and Airflow don't seem to solve this specific problem.
Developing such a feature would be potentially time-consuming and thus, likely falls outside the scope of this project.
However, there are more simple alternatives we could try. For example:
We could save log outputs to separate files as opposed to directly to stdout. So each step of the process would have a corresponding log file, such as:
step1_k8s-cluster-create.debug.logstep2_k8s-create-deployment.debug.logstep3_switch-create.debug.logIf a command fails, we would fetch the specific step's logs and output it to stdout. In addition, we could keep the metadata (e.g, cluster IP, switch IP, etc.) within a structured file (a json file for example), and then output the concise necessary details once the steps have been completed.