search

fabric8-services / fabric8-jenkins-idler

OpenShift.io service to idle resp.unidle Jenkins instances
Apache License 2.0
4 stars 15 forks source link

Idler is creating too many user-idlers #291

Closed chmouel closed 5 years ago

chmouel commented 5 years ago

While deploying latest idler in production we have experience some pods running out of memory and get killed by openshift.

While digging onto the ELK logs we have seen that idler is creating&_a=(columns:!(message),index:'project.dsaas-production.129c1244-1b37-11e7-8db9-124160897002.',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'kubernetes.pod_name:%22jenkins-idler-28-msvqb%22%20AND%20kubernetes.namespace_name:%22dsaas-production%22%20AND%20msg:%22creating%20user-idler%22')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:response))))) ~24k user-idler while the stable idler deployment with the previous image is only creating&_a=(columns:!(message),index:'project.dsaas-production.129c1244-1b37-11e7-8db9-124160897002.',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'kubernetes.pod_name:%22jenkins-idler-29-z74l2%22%20AND%20kubernetes.namespace_name:%22dsaas-production%22%20AND%20msg:%22creating%20user-idler%22')),sort:!('@timestamp',desc),uiState:(spy:(mode:(fill:!f,name:!n))))) ~2k

sthaha commented 5 years ago

Going to close this as the number of user-idlers created is dependent on the number of user and the build and dc events per user. This issue has been fixed in prod by increasing the memory and cpu .