search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.7k stars 13.84k forks source link

Current GNOME ISO exceeds Hydra output limit on Hydra #159612

Open sternenseemann opened 2 years ago

sternenseemann commented 2 years ago

Describe the bug

The GNOME ISO job currently fails bcause it's output exceeds the configured output limit on Hydra: https://hydra.nixos.org/build/167219646, eventually preventing nixos-unstable from advancing.

I'm not sure what caused this, presumably the last staging-next rotation?

cc @NixOS/gnome

jtojnar commented 2 years ago

See also https://github.com/NixOS/nixpkgs/issues/141521#issuecomment-1008150858

andersk commented 2 years ago

Here are some suspicious -dev output dependencies in the GNOME ISO closure, found with nix-tree:

andersk commented 2 years ago

Some selected large packages that jump out at me:

andersk commented 2 years ago
jtojnar commented 2 years ago
  • pipewire-media-session-0.4.1 → pipewire-0.3.45-dev
  • python3.9-brotlicffi-1.0.9.2 → python3.9-cffi-1.15.0-dev

I tried removing these references but the size changes were insignificant, IIRC.

speech-dispatcher-0.11.1 pulls in multiple speech systems, including mbrola-3.3 at 677 MB and flite-2.2 at 64 MB; do we need them all?

Presumably, different implementations have different quality for different languages. Mbrola was specifically requested.

There are two copies of webkitgtk-2.34.5 with different hashes, at 124 MB each; one of them is used only by yelp-41.2.

This is a consequence of libsoup-3.0 changing ABI so webkitgtk-4.0 linked against libsoup-2.0 cannot be used in packages linked against libsoup-3.0. However, most of those (including Yelp) will still build with webkitgtk-4.0 and libsoup-2.0 so we could switch to that for now.

jtojnar commented 2 years ago

By the way, as more and more apps switch to GTK 4, we will need to ship webkitgtk_5_0 ABI variant so there will need to be two webkit versions anyway (unless everything manages to switch this cycle).

andersk commented 2 years ago

strip would shave 27 MB from libwebkit2gtk-4.1.so.0.0.5 and libjavascriptcoregtk-4.1.so.0.0.5. We could probably save truckloads of space with some stdenv-wide equivalent of Debian’s dh_strip.

jtojnar commented 2 years ago

Would not that also destroy debugging symbols? We want those for webkit because having people rebuild webkit to report crashes is not really feasible.

andersk commented 2 years ago

webkitgtk has separateDebugInfo = stdenv.isLinux. I don’t know exactly what needs to be preserved for separate debug info to work, but I do know that Debian also uses separate debug info, and its webkitgtk libraries are much smaller than ours, and cannot be further reduced by strip.

andersk commented 2 years ago

https://stackoverflow.com/q/46197810/115030 addresses exactly that question, and suggests that we should be able to safely use --strip-unneeded.

sternenseemann commented 2 years ago

I feel like the long term solution to this problem may very well be having a way to allow bigger outputs on Hydra for certain jobs. Having a big graphical live system may very well be a fact of life.

dasJ commented 2 years ago

@sternenseemann I think it's worthwile to keep it below 4.7G so it still be burned onto a regular DVD. Burning it onto a CD seems pointless at this point because 700M is pretty much out of reach by now :D

06kellyjac commented 2 years ago

I think a small image should be a goal but yes a graphical live system can only be so small.

POP!_OS is 2.47GB elementary OS is 2.5GB Ubuntu is 2.9GB Linux Mint is 2.1GB Manjaro XFCE which I expected to be fairly small is a wopping 3.4GB and their minimal XFCE edition is 2.6GB Endevor OS is 1.9GB

vcunat commented 2 years ago

Burning it onto a CD seems pointless at this point because 700M is pretty much out of reach by now :D

The -minimal ISO is around 0.8G, so maybe there it isn't out of reach, but I don't think CDs are really common anymore. As for flash drives, the interesting limits often are around powers of two.

NobbZ commented 2 years ago

@vcunat It is pretty out of reach for a Gnome based installer.

All: Can't we just make the ISO a failable/non-required build, such that it unblocks unstable quickly?

vcunat commented 2 years ago

No, we most likely don't want that. The ISO is linked from web, etc.

NobbZ commented 2 years ago

We can continue linking the old working version

vcunat commented 2 years ago

Maybe, if it wasn't all automated.

drupol commented 2 years ago

Sarcasm: let's get rid of gnome !

NobbZ commented 2 years ago

@drupol Honestly? Why not… The installer at least… The Gnome installer has a bad UX, as it just boots into an empty desktop… The Plasma installer has at least 3 desktop icons that make sense…

Whoever chooses to use the Gnome installer needs to know in advance how to find and what to find…

alex-robbins commented 2 years ago

Can we start by temporarily increasing the output limit for Hydra? We can talk later about how to reduce the size of the gnome installer and what the eventual size limit should be.

Right now, though, unstable has been blocked for 7 days, and the first order of business should be unblocking it, correct?

drupol commented 2 years ago

@drupol Honestly? Why not… The installer at least… The Gnome installer has a bad UX, as it just boots into an empty desktop… The Plasma installer has at least 3 desktop icons that make sense…

Whoever chooses to use the Gnome installer needs to know in advance how to find and what to find…

While I was being sarcastic, there was a part of truth behind my statement. Of course I don't want to offend Gnome developers and users... but I definitely agree with your comment.

andersk commented 2 years ago

In case there’s still anyone on this issue who’s interested in, like, working towards solving it, I’d appreciate reviews and merging of

FranzStrudel commented 2 years ago

Honest question, Is even burning a DVD still a thing ?

I feel like people don't even have a CD/DVD player anymore.

For the same reason a 4Gb limit seems more appropriate than a 4.7Gb limit. In term of flash drive it is either 4 or 8, and I'm not willing to dedicate an 8Gb flashdrive for an OS installer.

marcelarie commented 2 years ago

A minimal install that downloads everything needed for the GNOME version will be better.

I understand why live cd isos are still a thing, but having that as the main option now a days makes no sense, just my opinion. It will be even better to have a small menu where you can choose between Gnome and Plasma etc...

I know there is a minimal install, but I mean one that just haves the menu to download from the internet everything needed to install the GNOME version.

jtojnar commented 2 years ago

Can we please move this discussion elsewhere (e.g. Discourse)? Let’s keep this thread focused on fixing the immediate problem.

nixos-discourse commented 2 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-should-the-hydra-output-size-limit-be/17761/1

vcunat commented 2 years ago

For reference, last I checked the size was 1.98 GiB, but we're blocked due to Hydra not being able to finish an evaluation.

bobvanderlinden commented 2 years ago

Could someone have a look at https://github.com/NixOS/nixos-org-configurations/pull/206 and decide whether it is good idea or not?

mannp commented 2 years ago

Could someone have a look at NixOS/nixos-org-configurations#206 and decide whether it is good idea or not?

Or perhaps move it so more important security updates and patches can be released.

Having patches released very quickly, and then waiting a week for the unstable channel update because of this failed job, seems counterintuitive :-(

vcunat commented 2 years ago

The -small channels were created (years ago) to get security fixes quickly. Those have been unaffected. They tend to update at least daily, quite reliably.

mannp commented 2 years ago

They do, to which I have tried, but they only include a small subset of the packages, including no gnome.

Am I able to merge channels for security updates and general apps together? As i'd understood, it was one or the other.

It seems perhaps more pragmatic to move longer term challenges into their own lower priority channel.

vcunat commented 2 years ago

The -small channels block on a small set of builds, but everything shares the same cache.nixos.org. So each binary is available in cache immediately after it gets built, and you'll get them regardless of which channel you follow.

vcunat commented 2 years ago

After Graham resolving some resulting complications, we got a >2G ISO build: https://hydra.nixos.org/build/167219646

bobvanderlinden commented 2 years ago

Awesome! Thanks for checking. I was still looking https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents, but that is lower in the queue I guess.

drupol commented 2 years ago

After Graham resolving some resulting complications, we got a >2G ISO build: https://hydra.nixos.org/build/167219646

Greater than 2G ? Or smaller?

vcunat commented 2 years ago

Well, we've been fighting this from more sides. Limit got raised, so the week-old ISO build succeeded when retried.

File size: 2160033792 bytes (2059.97 MiB)

vcunat commented 2 years ago

The tested job is also at the top of the queue, but it needs many builds to complete, which is probably why it takes time.

zowoq commented 2 years ago

As this isn't blocking the channel now I've removed the label so this issue doesn't appear at the top of https://status.nixos.org/

vcunat commented 2 years ago

The nixos-unstable channel updated 🎉 (to a commit from 28 hours ago) Thanks everyone who's helped with this. Some suggestions from above would still be nice to resolve.

bobvanderlinden commented 2 years ago

Some of the sub-problems have also been moved to separate issues by @andersk. I think the main goal of this issue is now solved. Can this issue be closed?

Should there be an issue for moving ISO generation away from Hydra channels into something separate?

vcunat commented 2 years ago

Separate? I can't recall what you mean. It's always been running on a separate machine, I believe.

bobvanderlinden commented 2 years ago

Ah, I was referring to separating the generation of the ISO like discussed here https://github.com/NixOS/nixpkgs/pull/161070 or suggested here.

LunNova commented 2 years ago

There are two copies of openblas-0.3.19 with different hashes, at 29 MB each, one pulled in by opencv-4.5.4, and the other by blas-3 and lapack-3.

Should we even have opencv around? It seems like it only gets pulled in by gst-plugins-bad.

Looks like that supports building without opencv as a dep.

Could either drop gst-plugins-bad or disable opencv in it by default?

K900 commented 2 years ago

Should we close this one?

vcunat commented 2 years ago

It's not a blocker anymore, though in my eyes there's at least one point above worth looking into:

  • speech-dispatcher-0.11.1 pulls in multiple speech systems, including mbrola-3.3 at 677 MB and flite-2.2 at 64 MB; do we need them all?
K900 commented 2 years ago

Maybe this is better handled as a separate issue?

vcunat commented 2 years ago

Hmm, picking up the parts that are still relevant and posting them into a new issue? Maybe.

Madouura commented 1 year ago

This has been affecting the ROCm builds a bit, particularly concerning rocm-llvm (with clang-tools-extra), miopen (with KDBs), and rocfft. (It's just a gigantic .so, with everything built) I'm not sure how realistic it is to ask this, or the reasons why we would not want to do this, but raising the limit to around 5GiB would be ideal concerning the ROCm packages.