While running Serratus in development mode (i.e. working on stuff) it's important we have good logging of what is going on (verbosity) from each container and each processing step.
When we're in 'production' mode and processing thousands of samples these logs are far less important as we don't have time/resources to track down every error, we just accept it happens at a low rate (<1%) and we will swing back around to those errors in in a 'clean-up' run where logging will matter.
Because we are logging all runs currently, this accounts for 20% of our total cost via CloudWatch Data (i.e. we had 2.7 TB of logging in May). To save these resources we should set Serratus defaults to be 'silent' and have an optional '-v' flag passed to worker.sh which will make things verbose when we are debugging.
This should be done by approximately June 6th prior to the next major runs and for an updated/accurate cost estimate.
ababaian
commented
4 years ago
While running Serratus in development mode (i.e. working on stuff) it's important we have good logging of what is going on (verbosity) from each container and each processing step.
When we're in 'production' mode and processing thousands of samples these logs are far less important as we don't have time/resources to track down every error, we just accept it happens at a low rate (<1%) and we will swing back around to those errors in in a 'clean-up' run where logging will matter.
Because we are logging all runs currently, this accounts for 20% of our total cost via CloudWatch Data (i.e. we had 2.7 TB of logging in May). To save these resources we should set Serratus defaults to be 'silent' and have an optional '-v' flag passed to
worker.shwhich will make things verbose when we are debugging.This should be done by approximately June 6th prior to the next major runs and for an updated/accurate cost estimate.