Using this action with CygWin mirrors - best practices?

[t-w]

Is there a recommended way of using this action with CygWin mirrors?

Is it OK. to use them for every build in GH, or better to make a container/docker image, store it in github containers (or whatever it's called) and use for builds?

Also: which ones to use? Normally, to install locally, it can be wise to select geographically, but in case of GH, which ones should be used? Or maybe all of them (if they are used in some random or round-robin way by the CygWin installer?

I usually want to prevent resource overuse (esp. unnecessary one...), but it is also good to have things up to date. But it depends on several factors (number/frequency of builds, how heavy is installation vs. using ready to use image etc.).

Any thoughts/experiences on this with CygWin?

[jon-turney]

Is there a recommended way of using this action with CygWin mirrors?

Is it OK. to use them for every build in GH, or better to make a container/docker image, store it in github containers (or whatever it's called) and use for builds?

By default mirrors.kernel.org is used by the action. I'm not sure if that's a single site or backed by some sort of CDN. There's no particular thought behind this, except that it has good availability and peering.

I don't think you can use a Docker container on GitHub's Windows CI infrastructure, although this might have changed since the last time I checked.

In any case, when I did some experiments with that, I found that a Windows container image is sufficiently large that it takes longer to retrieve that than installing Cygwin from scratch. (Obviously that might also have changed, and depends on various factors)

Also: which ones to use? Normally, to install locally, it can be wise to select geographically, but in case of GH, which ones should be used? Or maybe all of them (if they are used in some random or round-robin way by the CygWin installer?

Unfortunately, Cygwin's setup program isn't very intelligent about this, currently.

Idea solutions here would perhaps be: (i) for setup to learn how to select "fastest" mirror, (ii) put our mirrors behind some sort of geolocating redirector, or use someone else's CDN (apparently Fastly provides a free service for open-source projects)

But these are things I lack the time to explore, so someone else would have to do all that!

I usually want to prevent resource overuse (esp. unnecessary one...), but it is also good to have things up to date. But it depends on several factors (number/frequency of builds, how heavy is installation vs. using ready to use image etc.).

Indeed.

[me-and]

I have a (decidedly stalled) plan to add caching of package directories to this installer, which would at least reduce the number of hits to the distribution servers to a per-repository level rather than a per-build level. I'm still planning on picking that project up again, but other obligations have been taking priority, and I'd be very happy for someone else to pick up the work, either starting from scratch or building on my fork.

I imagine that kernel.org and GitHub both pay attention to what's using significant amounts of bandwidth, and would be able to arrange for a mirror local to the GitHub servers, masked behind a CDN, if they thought it worth the effort. (Indeed, they might already have done so!)

[Cyan4973]

tangentially related: cygwin install action has been failing in our CI tests for a few days now. Considering the error log, I would expect this issue to impact other users, but is that the case ?

net: Preconfig
site: http://mirrors.kernel.org/sourceware/cygwin/
HTTP status 404 fetching http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.zst.sig
mbox note: Unable to get http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.zst.sig from <http://mirrors.kernel.org/sourceware/cygwin/>
unattended_mode is set at mbox: returning default value
HTTP status 404 fetching http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.xz.sig
mbox note: Unable to get http://mirrors.kernel.org/sourceware/cygwin/x[86](https://github.com/facebook/zstd/actions/runs/8213375045/job/22464707785?pr=3937#step:4:88)_64/setup.xz.sig from <http://mirrors.kernel.org/sourceware/cygwin/>
unattended_mode is set at mbox: returning default value
HTTP status 404 fetching http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.bz2.sig
mbox note: Unable to get http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.bz2.sig from <http://mirrors.kernel.org/sourceware/cygwin/>
unattended_mode is set at mbox: returning default value
HTTP status 404 fetching http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.ini.sig
mbox note: Unable to get http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.ini.sig from <http://mirrors.kernel.org/sourceware/cygwin/>
unattended_mode is set at mbox: returning default value
mbox note: Unable to get setup from <http://mirrors.kernel.org/sourceware/cygwin/>
unattended_mode is set at mbox: returning default value

[jon-turney]

tangentially related: cygwin install action has been failing in our CI tests for a few days now. Considering the error log, I would expect this issue to impact other users, but is that the case ?

The way to confirm if it's affecting other Cygwin users is probably to check https://cygwin.com/pipermail/cygwin/

site: http://mirrors.kernel.org/sourceware/cygwin/ HTTP status 404 fetching http://mirrors.kernel.org/sourceware/cygwin/x86_64/setup.zst.sig

This looks like an unrelated problem which has been remedied.

See https://cygwin.com/pipermail/cygwin/2024-March/255640.html

(Assuming it's been "a few days" since 20:44 UTC 2024-03-08 in your timezone)

[t-w]

I had exactly the problems you mentioned above (using a different action, but I switched to this one, not being sure where the problem is...). And, indeed, it seems solved, now.

As for my questions - thank you for the response. So for now, I stick to using somewhat geographically close mirror(s) then.

Just to precise one detail - this action allows to use several "sites [...] separated by a whitespace". So how are they used by the installer? According to the list order (if first one failing, then try the next etc.) or something else (random)? (Rephrasing: does order matter or not?)

[jon-turney]

So for now, I stick to using somewhat geographically close mirror(s) then.

It seems like the GitHub CI VM could be anywhere...

Just to precise one detail - this action allows to use several "sites [...] separated by a whitespace". So how are they used by the installer? According to the list order (if first one failing, then try the next etc.) or something else (random)? (Rephrasing: does order matter or not?)

I'd have to look at the source code of the installer to answer that question.

I think because of the way the installer (currently) handles them, It's ususally not very useful to specify multiple cygwin mirror sites (i.e. it doesn't increase speed or reliability).

I think the only sensible use for multiple sites currently is where they are for overlay repositories (i.e. with a disjoint set of packages)

[t-w]

Multiple sites could be used by the installer to help distribute load (without use of a CDN) - not necessarily in one installation, downloading each package from a different mirror (to speed-up), but each time it is executed, use a different mirror for the whole installation. If it was random, each time a different one could be selected from the list, distributing the load on different sites. That could be useful in CI systems, which have static configuration. It does not matter for local/manual installation.

But, unfortunately, it could be something else than people expect (most likely: trying different mirrors in order). Probably a command-line switch for selecting the behaviour would be necessary for this.

Anyway - OK. I thought someone here could just know how it works (as it is directly related to this action). I do not want to bother you with going to the code (maybe I'll manage to do it myself if I ran into trouble with mirrors again ;).

Thanks!