Be nice for GMP infrastructure ak. potential DoS by CI

[ad-m-ss]

Hello,

I am not sure if you have received that information in any way, but to make sure I suggest reading about potential DoS caused by your CI setup: https://gmplib.org/list-archives/gmp-devel/2023-June/006162.html

Do you mind changing your CI setup to avoid pulling repositories from GMP in so a wasteful way?

I see a few possible solutions:

• setup mirror repository on GitHub & use the CI system to refresh it incrementally
• setup required steps to download the repository once, upload it as an artifact, and then download artifacts in each parallel step from artifacts

Let me know if you need any further help with that.

Kind regards

Disclaimer: I am not related to GMP nor Microsoft or GitHub, just see drama on the mailing list as a HackerNews reader.

[BtbN]

It already is cached by the docker image. The problem why it's SO MANY requests is because hundreds of people have forked this repository, and now at the same time (due to the workflow cron) start an image build. 99% of the days, it'll just use the cached image and not cause any network requests whatsoever.

If it builds new images, it'll be at max 10 requests to fetch the latest snapshot in parallel from my end (that's how many parallel jobs Github allows for free), which shouldn't be an issue. I've already switched GMP back to use an outdated release tarball because of how annoyingly flakey their mercurial server is.

I'd happily use an official Github mirror if there was one, but that doesn't seem to be the case. And I'm not gonna claim to be in any way official for GMP and set one up myself (it's not even git, so doing that would be non-trivial).

I have no control over the over 700 forks that exist at this point, which all potentially will stay on the outdated version that's using hg for a long time, given poor/no maintenance. So those will very likely keep on making requests until they eventually succeed in building fresh images they can cache.

GMP is at this point the only dependency that does not offer a sane way to clone its repository. Almost everything else has official mirrors at a major hoster (ideally Github itself, so the traffic doesn't leave their infra). GMP is one of the last trouble makers.

[Nothing4You]

As it seems like you're already using versioned dependencies by revision/version number, you could also make use of the cache action as well, which can be used with custom cache keys to allow e.g. automatically invalidating (by no longer using) old caches after a version/revision change.

[nukeop]

GMP is at this point the only dependency that does not offer a sane way to clone its repository. Almost everything else has official mirrors at a major hoster (ideally Github itself, so the traffic doesn't leave their infra). GMP is one of the last trouble makers.

One look at their website and their hysteric reply to Github's calm explanation is all you need to know. There must be something really wrong with their infrastructure and/or approach.

[BtbN]

The cache action is useless since it very much is not compatible with how the entire setup is working. The results of the downloads are already cached. The download load of one single repository is not all that great.

And yes, their infrastructure has been super flaky from the get go, even when I tested initially just at home when setting things up. The server at times just randomly returns seemingly garbled data. And the reaction to that is for hg to just... try again. This probably massively amplified the whole issue, since now you got a bunch of repos running hg, against an already flakey repo, and instead of all of them being done within seconds, they all go into exponential backoff retry loops.

[nukeop]

I have no idea why they're not using edge caching, which would immediately solve this problem in a very cheap way. Why is cloning a repo putting any significant load on their servers?

[BtbN]

Keep in mind, if the stars align (this is really the exception) and the cache gets invalidated (For example, a new Ubuntu base image is published. In a normal day, nothing is downloaded and built at all) in all forked repos at the same time, and then Github launches all the easily hundreds of parallel scheduled build jobs might hit up the gmp hg server at around the same time.

So it is a quite significant amount of traffic, but it's far below anything I'd expect to take down any half decent server. I have no experience with running an HG server, maybe it's stupidly CPU intensive or something. The total lack of a shallow or minimal clone is also an amplifying factor.

[ad-m-ss]

@BtbN Thanks for the clarification. It turns out that the world is always more complicated than it seems at first glance.

I will leave the issue open for visibility for some time so that people can see the point of view here more easily.

[Mathiasdm]

(As a Mercurial contributor, I want to try to add a little bit of context regarding the Mercurial usage)

And yes, their infrastructure has been super flaky from the get go, even when I tested initially just at home when setting things up. The server at times just randomly returns seemingly garbled data. And the reaction to that is for hg to just... try again. This probably massively amplified the whole issue, since now you got a bunch of repos running hg, against an already flakey repo, and instead of all of them being done within seconds, they all go into exponential backoff retry loops.

Normally, Mercurial itself does not do retries if it encounters garbled data. It might be that the CI scripts or some kind of wrapper is doing the 'try again'?

So it is a quite significant amount of traffic, but it's far below anything I'd expect to take down any half decent server. I have no experience with running an HG server, maybe it's stupidly CPU intensive or something. The total lack of a shallow or minimal clone is also an amplifying factor.

Doing an 'hg clone' typically results in the server side doing some compression and then transferring the data to the client (which does some decompression). It's probably the compression that is generating the load. Mercurial has a feature called 'clonebundles', which resolves this by using a generated bundle, but it doesn't seem to be enabled on the gmplib repository. I've sent them a mail to clarify how they can enable this functionality, see https://gmplib.org/list-archives/gmp-devel/2023-June/006180.html

[darkpixel]

This bug is about to be Slashdotted: https://tech.slashdot.org/story/23/06/28/2045228/microsofts-github-under-fire-for-ddosing-crucial-open-source-project-website