search

m-lab / annotation-service

Annotation integration service for M-Lab data
Apache License 2.0
3 stars 5 forks source link

Annotator instances shutting down because of inadequate disk #156

Open gfr10598 opened 5 years ago

gfr10598 commented 5 years ago

Monitoring shows (for some deployments) that instances are being killed because of too little disk space. Minimum seems to be around 1GB. Could this be caused by excessive logging?

resource.type="gae_app" resource.labels.module_id="annotator" logName="projects/mlab-sandbox/logs/appengine.googleapis.com%2Fvm.syslog" "unhealthy"

Dec 19 22:17:01 aef-annotator-20181219t200546-3dwf vm_check_disk_space.sh: Free disk space (883 MB) is lower than threshold value 998 MB. Reporting instance permanently unhealthy. Note: different instances will have a different threshold value.

gfr10598 commented 5 years ago

After cleaning up some spammy logs, the problem seems to be much improved. We should probably just be aware of this problem, and try to avoid spamming the logs too severely.

Alternatively, we could certainly allocate more disk, but this might just hide a problem that it would be better to avoid.

gfr10598 commented 5 years ago

image