jenkins-x / lighthouse

Apache License 2.0
184 stars 114 forks source link

Approval plugin fails with very large repositories #1085

Closed patrickleet closed 3 years ago

patrickleet commented 3 years ago

I have a repo that contains some machine learning models - the size of the repo is around 250mb, and can take a few minutes to clone. It appears there is some sort of timeout preventing the clone from finishing.

Here are the logs, only edited to remove the auth token:

{"Branch":"master","Clone":"https://github.com/thepathapp/nsf-pathway-models.git","ID":"276190324","Link":"https://github.com/thepathapp/nsf-pathway-models","Name":"nsf-pathway-models","Namespace":"thepathapp","Webhook":"issue_comment","level":"info","msg":"invoking Issue Comment handler","time":"2020-09-29T17:13:23Z"}
{"Branch":"master","Clone":"https://github.com/thepathapp/nsf-pathway-models.git","ID":"276190324","Link":"https://github.com/thepathapp/nsf-pathway-models","Name":"nsf-pathway-models","Namespace":"thepathapp","Webhook":"issue_comment","author":"patrickleet","level":"info","msg":"Issue comment created.","org":"thepathapp","pr":34,"repo":"nsf-pathway-models","time":"2020-09-29T17:13:23Z","url":"https://github.com/thepathapp/nsf-pathway-models/pull/34#issuecomment-700839044"}
{"level":"info","msg":"found plugins approve, assign, blunderbuss, help, hold, lgtm, lifecycle, override, size, trigger, wip, heart, cat, dog, pony\n","time":"2020-09-29T17:13:23Z"}
{"client":"git","level":"info","msg":"Cloning thepathapp/nsf-pathway-models for the first time.","time":"2020-09-29T17:13:24Z"}
{"client":"git","level":"warning","msg":"Running /usr/bin/git [clone --mirror https://pathnsf-bot:xxx@github.com/thepathapp/nsf-pathway-models /tmp/git173713999/thepathapp/nsf-pathway-models.git] returned error exit status 128 with output Cloning into bare repository '/tmp/git173713999/thepathapp/nsf-pathway-models.git'...\nerror: index-pack died of signal 9\nfatal: index-pack failed\n.","time":"2020-09-29T17:15:40Z"}
{"client":"git","level":"warning","msg":"Running /usr/bin/git [clone --mirror https://pathnsf-bot:xxx@github.com/thepathapp/nsf-pathway-models /tmp/git173713999/thepathapp/nsf-pathway-models.git] returned error exit status 128 with output Cloning into bare repository '/tmp/git173713999/thepathapp/nsf-pathway-models.git'...\nerror: index-pack died of signal 9\nfatal: index-pack failed\n.","time":"2020-09-29T17:17:55Z"}
{"client":"git","level":"warning","msg":"Running /usr/bin/git [clone --mirror https://pathnsf-bot:xxx@github.com/thepathapp/nsf-pathway-models /tmp/git173713999/thepathapp/nsf-pathway-models.git] returned error exit status 128 with output Cloning into bare repository '/tmp/git173713999/thepathapp/nsf-pathway-models.git'...\nerror: index-pack died of signal 9\nfatal: index-pack failed\n.","time":"2020-09-29T17:20:15Z"}
{"Branch":"master","Clone":"https://github.com/thepathapp/nsf-pathway-models.git","ID":"276190324","Link":"https://github.com/thepathapp/nsf-pathway-models","Name":"nsf-pathway-models","Namespace":"thepathapp","Webhook":"issue_comment","author":"patrickleet","error":"failed to clone thepathapp/nsf-pathway-models: git cache clone error: exit status 128. output: Cloning into bare repository '/tmp/git173713999/thepathapp/nsf-pathway-models.git'...\nerror: index-pack died of signal 9\nfatal: index-pack failed\n","level":"error","msg":"Error handling GenericCommentEvent.","org":"thepathapp","plugin":"approve","pr":34,"repo":"nsf-pathway-models","time":"2020-09-29T17:20:19Z","url":"https://github.com/thepathapp/nsf-pathway-models/pull/34#issuecomment-700839044"}
vbehar commented 3 years ago

we had the same issue, it's the git process which was using too much memory and was killed by the OOMKiller. You can easily "fix" it by changing the memory limit for the webhook pods

hferentschik commented 3 years ago

fixed by #1087