search

apache / buildstream

BuildStream, the software integration tool
https://buildstream.build/
Apache License 2.0
85 stars 28 forks source link

BuildStream doesn't retry on Git HTTPS with error 429 #1292

Open BuildStream-Migration-Bot opened 3 years ago

BuildStream-Migration-Bot commented 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @nanonyme] on Apr 25, 2020, 10:29

Summary

Noticed in tracking that if remote server returns 429 (Too Many Requests), BuildStream bails out immediately. It will not retry even with grace period. Noticeable in eg https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/526479997

Steps to reproduce

Hard to repro, you have to have a backend that returns 429. Mock server that always returns 429 might work.

What is the current bug behavior?

BuildStream fails with no retry.

What is the expected correct behavior?

BuildStream retries with eg incremental backoff jitter if no further sources available. Retry time should probably be configurable.

Relevant logs and/or screenshots

        [--:--:--] STATUS  bootstrap/build/rsync.bst-0: Running host command
            /usr/sbin/git fetch origin --prune --tags --force
        fatal: unable to access 'https://git.samba.org/rsync.git/': The requested URL returned error: 429

Possible fixes

Other relevant information

BuildStream-Migration-Bot commented 3 years ago

In GitLab by [Gitlab user @jjardon] on Apr 26, 2020, 21:28

From IRC:

10:25 <juergbi> jjardon: 429 should probably be detected as temporary failure such that buildstream will retry on its own
10:26 <juergbi> (exception parameter `temporary`)
10:26 <juergbi> I don't remember whether we already wait properly between retries for `temporary` errors
BuildStream-Migration-Bot commented 3 years ago

In GitLab by [Gitlab user @nanonyme] on Apr 26, 2020, 21:35

[Gitlab user @jjardon] that sounds a bit tricky as we're invoking git executable in the plugin. But yes, once figured out how to detect this, sounds like right approach.

BuildStream-Migration-Bot commented 3 years ago

In GitLab by [Gitlab user @nanonyme] on May 1, 2020, 19:52

mentioned in merge request freedesktop-sdk/freedesktop-sdk!2503

BuildStream-Migration-Bot commented 3 years ago

In GitLab by [Gitlab user @jjardon] on May 1, 2020, 20:08

marked this issue as related to freedesktop-sdk/freedesktop-sdk#999

gtristan commented 2 years ago

Note: This issue should be moved into a separate repo after migrating the git plugin out of core.

nanonyme commented 1 year ago

This may also not be fixable in context of the git plugin. But we may be able to really fix this with the new git_repo plugin that's heading into bst-plugins-experimental. Should anyway be closed from this repo. Due to migration, issue creators cannot close their issues.

nanonyme commented 7 months ago

@gtristan I can't close this even though this was created by me because this is migrated.