enrico-usai
commented
3 years ago Hi Roman,
thanks for your suggestions and for the self-explanatory screenshot.
- In the shared configuration file you have multiple
[cluster]sections so it's not clear to me if these costs are coming from a single cluster and which is the used configuration. - It would be useful to know the kind of job submission you're doing to understand if there is a corner case that is producing tons of logs (e.g. from slurm, /var/log/messages or syslog).
- Could you create and share an archive of CloudWatch logs by following this guide or share the larger files in your head node from this list?
- How did you produce this screenshot?
Regarding the suggested improvements:
- keep logs across cluster lifecycle:
- We decided to not preserve logGroups across cluster lifecycles because you can create a new cluster with the same name but with a totally different configuration (different IP ranges, different version of ParallelCluster, different OS or scheduler, different uploaded log files) so the number of log streams would keep growing, increasing costs but losing its meaning and usefulness.
- In any case it's already possible to retain the logs of a running cluster by executing
pcluster delete --keep-logs ...command and you can configure the retention time by using theretention_daysconfiguration parameter incw_logssection.
- log file filters:
- I think it's a good idea, so I'm marking the issue as feature enhancement, thanks for the suggestion.
Thanks
Enrico
brainstorm
demartinofra
alexiswl
Required Info:
2.9.1tothillBug description and how to reproduce:
This issue is simple: Using the above config details we instantiated a cluster that consumes more on logging than on actual compute:
Furthermore the
LogGroupwhere those ~$25 worth of logs are gone since the cluster's CloudFormation was deleted as part of the shutdown of the cluster.So a couple of improvements that could help a lot:
1) Have a
cw_logssection in the pcluster config that allows to keep the LogGroups across cluster lifecycles (we usually tear down the cluster when done with the job(s)). 2) Have some sort ofcloudwatch-agent-related section directives that allow us to filter which specific logfiles to watch and (perhaps) filter or rate-limit some contents before they get sent to AWS CW?I guess that the latter could be done via some
bootstrapfine-tuning, but still, it'd be useful to have it exposed in the config, I reckon.We'll have to disable CW logs for the time being, which is not very optimal in terms of security/auditability of parallel cluster, but log ingestion costing more than actual compute is not good either :-S
/cc @reisingerf @alexiswl @victorskl