search

conda-forge / staged-recipes

A place to submit conda recipes before they become fully fledged conda-forge feedstocks
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
689 stars 4.72k forks source link

Package request: X11 Utility Apps, X.Org Server, and Remaining X11 Libraries #26241

Open ehfd opened 1 month ago

ehfd commented 1 month ago

Package name

More to be added.

Package version

Newest

Package website

https://www.x.org/releases/individual/app/

https://www.x.org/releases/individual/util/

Package availability

https://www.x.org/releases/individual

https://gitlab.freedesktop.org/xorg/app

https://gitlab.freedesktop.org/xorg/util

https://salsa.debian.org/xorg-team/app

Additional comments

These are many numbers of command-line or GUI applications from X.org which are typically packaged in Debian or RHEL as a group of packages, such as x11-apps, x11-utils, x11-xserver-utils, and xfonts-utils.

Instead of creating >73 different feedstocks for iceauth, xrandr, xhost, xset, etc... (of which none are currently in feedstocks) it might be a good idea to build them all in one shot together and package them in categories like how Debian or RHEL does, reducing the number of feedstocks to <10.

Moreover, it would be less demanding to first limit the builds of utils and apps to Linux or Mac, unlike libs.

https://salsa.debian.org/xorg-team/app https://src.fedoraproject.org/repo/pkgs/xorg-x11-server-utils/

Looking for help from experts for this complex recipe.

Package is not available

No previous issues or open PRs

ehfd commented 1 month ago

CC @xhochy @hmaarrfk @pkgw and others.

ehfd commented 1 month ago

Additional CC @epruesse @zklaus @scopatz @erykoff @ccordoba12 @mingwandroid @n-elie

I apologize for the tag bomb.

ehfd commented 1 month ago

Moreover, it's possibly the opportunity to port the Windows X11 stack from MSYS2/MinGW to VC++ like how https://github.com/marchaesen/vcxsrv did it, or at least try to update the MSYS2 compiler.

I will contribute a subset of the X11 apps packages, typically packaged standalone by Debian or RHEL (such as xorg-libcvt, xauth), which I immediately need.

hmaarrfk commented 1 month ago

I understand the maintenance challenge with X11 are somewhat tiring, but I'm rather hesitant to adding to our maintenance burden unless absolutely necessary.

We don't have a good way to 'transition' packages at conda-forge, so a new package name isn't necessarily an "easy swap".

Can you point us to conda-forge feedstock you need help with?

ehfd commented 1 month ago

@hmaarrfk None of the packages here I listed are currently in a non-CDT feedstock. Xvfb and xorg-utils (possibly others) are CDT packages. x11-util-macros does not overlap here.

These packages are utility executables (some are important utility tools which control the interaction of an X application with X11 servers where applications depending on X11 are crippled without) with dependencies to currently packaged X11 libraries. We are not changing existing feedstocks.

And my suggestion is simply to merge >30 small feedstocks if done separately into ~5 larger feedstocks similar to Debian or RHEL.

hmaarrfk commented 1 month ago

Hmm. Ok. My inclination is to say that unfortunately this is quite troublesome to execute.

https://github.com/conda-forge/conda-forge.github.io/issues/1894

Discusses the reverse problem, but ultimately we don't have a way of "renaming" packages and thus this is really a difficult excercise.

We ignored problem for jpeg and jpeg-turbo and it left many I knew with no option to update. Only to tear down the environment to rebuild it.

ehfd commented 1 month ago

Then, this would be a long-term thing, I'll leave this issue up for now.

We're not merging existing packages, but this might be a decision that should be made while creating the initial versions of such packages.

ehfd commented 1 month ago

Also in the wishlist: xorg-libxv, xorg-libxxf86dga1

hmaarrfk commented 1 month ago

I think the first step in the "long term project" would be to create a table of the new package names, and which existing conda-forge package they "amalgamate".

Typically it helps to edit the first comment to keep the list alive.

ehfd commented 1 month ago

Guidelines for X11 package recipes from @pkgw :

Note: https://anaconda.org/search?q=mingw-w64-ucrt-x86_64-gcc

https://github.com/conda-forge/xorg-libxmu-feedstock/pull/10#issuecomment-2096054913

https://github.com/conda-forge/xorg-makedepend-feedstock/pull/9#issuecomment-1930098269

h-vetinari commented 1 month ago

Out of curiosity, what's the use-case for having the binaries (as opposed to just the libraries)? In the bigger picture, while the Wayland future is still going to take a good while, it's coming inexorably, so I'm not sure how much of our scarce resources the X11 ecosystem still merits. 🤔

ehfd commented 1 month ago

@h-vetinari

In principle, utilizing the same HPC environment (Conda) researchers use for ML and AI, to deliver rootless, isolated, and portable graphical workloads (Robotics including ROS, Carla, OpenCV, PyMOL, VMD, ParaView, OpenDroneMap, etc.).

HPC facilities now mostly use either Conda or Apptainer (known as Singularity). Other rootless environments such as LinuxBrew (HomeBrew), Pkgsrc, Nix, Spack, and EasyBuild (where the latter two are basically Conda wrappers adding some more packages on their sides) exist. One example: https://hpc.nmsu.edu/discovery/software/modules/

Out of these, LinuxBrew, Pkgsrc, and Nix have X11 server binaries and the above CLI tools, but only Conda (with Apptainer coming close) is universal to all global HPC facilities and supercomputer centers, and most importantly, considered a first-class citizen of NVIDIA and/or ROCm, either through pip or conda.

The above binaries are required for functional usage of the xorg-xserver binary (needed for XDummy) and xorg-xvfb, as well as VirtualGL (if we are stretching a bit more for headless OpenGL support).

While Wayland is coming fast, the vast majority of HPC facilities are still optimized for X11, and despite Conda already having Wayland libraries (much like X libraries), people are, as of now, not going to start a Wayland server solely inside Conda.

In that sense, Wayland and X11 are similarly limited in Conda.

Moreover, Wayland does not mean that X11 will completely disappear. There will still be legacy applications still on X11 (very prevalent in academic situations), and thus XWayland will be required. Since XWayland is another X server, it comes back to still needing the X.Org CLI tools.

Speaking of which, unlike the libraries, the X.Org CLI tools, as you've said, aren't mandatory in essence, as these feedstocks don't have explicit dependence on other feedstocks. So conversely, they don't need to be built through all sorts of patching for VC or MSYS2 in Windows.

VcXsrv and Xming already do this well (after some meticulous code patches for the whole X11 stack on their sides) for everyone in Windows, and unlike Mac or Linux, Windows doesn't have the ABI and glibc hell which otherwise requires Conda for portability.

Thus, the maintenance strain would be much less if it was only targeted for Mac and Linux because the only things probably required include autoconf boilerplate and a script for automatic build recursion (in my experience writing new feedstocks, >90% of the time spent making X11 feedstocks pass the test comes from patching Windows). I'm also simply omitting Windows for application feedstocks that depend on X11 when they don't build and have some weird Windows incompatibilities such as fork() (see the PRs for xclip, xsel, xdotool, etc.).

h-vetinari commented 1 month ago

Thanks for the detailed reply! In principle it's a good idea to provide these things of course, but it might be a substantial effort, and that would need a champion. It sounds like that could potentially be you? 🙃

You might also want to join one of the core calls (biweekly Wednesday 18h UTC, next one is today, agenda not too crammed yet) to discuss the idea and how to approach it.

ehfd commented 1 month ago

It sounds like that could potentially be you?

It's not immediate for me in the current sense, but in a year or so if people can wait. It will be in demand eventually. If someone's interested before that, they can do it. Else, I'll do it eventually.

For now, I'm just fully focused on the depending library structures that should land for the Alma8 sysroot launch. The future structure of mesalib and X11 libraries should be decided before Alma8, then it will benefit this cause some time later.

From https://github.com/conda-forge/cdt-builds/issues/66:

This seems to be the feedstock for libxcomposite: https://github.com/conda-forge/xorg-libxcomposite-feedstock

And libxshmfence is a critical package I forgot that isn't in Conda. This is a CDT package, so should be done ASAP. @hmaarrfk

h-vetinari commented 1 month ago

I brought this up in the core call to get a feel for what people think about this. Overall there's no issue with doing this, though:

Not discussed directly in the call but from my POV: any of the missing lib* packages should be added and PRs are very welcome for that.

ehfd commented 1 month ago

@h-vetinari Great, thank you!

I am also waiting for more potential contributors to contribute to this job. Perhaps from my own project.

We can wait and see. Meanwhile, I'd thank the core team if they can focus on the CDT refactoring roadmap.