Undefined symbol issues

[fanzhuyifan]

Currently undefined symbols still happen. This is typically caused by the following:

1. package A depends on package B
2. The source of package B is updated, so B is rebuilt
3. But often A is not rebuild based on the rebuilt B, causing issues.

I am opening this issue for possible discussion on how to solve this.

Ideally, when the above happens，we want to automatically recompile, and bump the version in some way such that the users can get the recompiled versions. In addition, the packages should be built in the order of their dependencies, so that if A depends on B, B is built after A, and against the compiled version of A.

[fanzhuyifan]

Here is a tentative proposal on how to bump the versions. Currently the versions are <tag>.r<revision>.g<hash>. We could do daily builds and append yyyymmdd to the version: <tag>.r<revision>.g<hash>.<yyyymmdd>

[fanzhuyifan]

I am happy to write a small script to figure out the correct build order of packages. Please let me know what you want the inputs and outputs to be.

[alexjp]

I am replying after reading this, so no much thought has gone into it, but I was thinking of something like outputting a kf6.json, where it would have the name of the package and some options.

One of those options would be order... and, to tackle rebuilds, something like "rebuild dependencies".

So... different ideas:

• save in the json/toml/yaml the dependencies (say kwin needs kscreen) and then the builder checks the date of kscreen. If the date of kscreen is superior to kwin, rebuild.
• save in the json/toml/yaml the commit hash of the dependencies, and if the commit is different, rebuild.

one could go the "npm packages" file, and for each dependency in the PKGBUILD, save either the date or the commit.

EDIT: see an example of what I was thinking of going for, when I moved from own PKGBUILDs to following arch kde-unstable repo: pkgs.yaml

[fanzhuyifan]

Here is a tentative proposal of what should happen:

Everyday, run the following:

1. Update all sources using makepkg -o [1]
2. Read all PKGBUILDs to form DAG of dependencies (I am not 100% sure if there aren't cyclic dependencies)
3. Perform a topological sort of all the packages we are rebuilding, and traverse them in order
4. For each package, perform a makepkg -efsi [1]
5. Use <tag>.r<revision>.g<hash>.<yyyymmdd> for versioning
6. Ideally, new packages should only be pushed if all builds succeed.

The two-pass method ensures that all packages for a daily build are built against the same source. In particular, the -e in step 4 skips source updating. The -i command is needed to ensure that if A depends on B, A is built against the latest package of B.

This can also be extended to rebuild only necessary packages, I think. But I would need to think a little bit more on how to handle that.

What do you think?

[1] https://man.archlinux.org/man/makepkg.8.en#OPTIONS

[alexjp]

hum.... I think point 1,2,3 and 4, would be reproducing the dependency work of pacman and portage. that would be a project into itself. It would be a "last resort" solution, I think. As a "quick hack", one could try to install kde in a fresh installation and see what order pacman installs it (or how other distros do it)

point 5, like and could be implemented right away.

point 6, could be make in the PKGBUILDs, a fixed commit hash. The PKGBUILDs could be built locally and if all good, push the commits into the chaotic repo to build. but then, we the devs, wouldn't actually be chaotic users.

point 6 could also be done with a "chaotic-kde-aur-testing", and going like manjaro style. but for that, the myth, the man, the legend @dr460nf1r3 has to chime in !

[fanzhuyifan]

hum.... I think point 1,2,3 and 4, would be reproducing the dependency work of pacman and portage. that would be a project into itself. It would be a "last resort" solution, I think. As a "quick hack", one could try to install kde in a fresh installation and see what order pacman installs it (or how other distros do it)

Actually I can try code up something for 2 and 3 over the weekend. If that gets done, wouldn't 1 and 4 be quite easy to do?

[fanzhuyifan]

As a proof of concept for 2 and 3, I created a tool here: https://github.com/fanzhuyifan/kde-git/blob/master/src/get_build_order.sh

It basically uses makepkg --printsrcinfo to parse the dependencies from the PKGBUILD, and uses tsort for topological sorting.

The other scripts combined enable building KDE packages from source on Arch Linux. (seems to work)

[dr460nf1r3]

point 6 could also be done with a "chaotic-kde-aur-testing", and going like manjaro style. but for that, the myth, the man, the legend @dr460nf1r3 has to chime in !

Yes, I think that would be feasible. I'd propose doing it on the same builder, building to chaotic-kde-aur-testing and pushing packages to chaotic-kde-aur manually via a script in case builds were deemed stable (replacing the repo directories packages). What do you say?

[alexjp]

sorry @fanzhuyifan and @dr460nf1r3 , things got in the way.

@dr460nf1r3 sounds good!

@fanzhuyifan will have a look at your tool and report!

About step 1 and step 4, yeah, seems easy. Actually, the chaotic builder already does 1 automatically.

[fanzhuyifan]

About step 1 and step 4, yeah, seems easy. Actually, the chaotic builder already does 1 automatically.

I am not familiar with the tooling on chaotic builder. Are packages always built in a clean chroot? If so you could enable the chaotic-kde-aur-testing repo in the chroot, such that you always build against the latest versions of the dependencies.

[fanzhuyifan]

There is cyclic dependencies between xdg-desktop-portal-kde, plasma-integration, and plasma-workspace in KDE-unstable though.

xdg-desktop-portal-kde depends on plasma-workspace, which depends on plasma-integration, which depends on xdg-desktop-portal-kde. Not sure what to do about those.

[alexjp]

I am not familiar with the tooling on chaotic builder. Are packages always built in a clean chroot?

Yes, they are built always inside an "minimal" container.

xdg-desktop-portal-kde depends on plasma-workspace, which depends on plasma-integration, which depends on xdg-desktop-portal-kde. Not sure what to do about those.

Yeah, just noticed. Generally its only an issue when "starting new". What I do is disable some feature that makes those cyclic dependencies.

Pretty cool script. Could you make it all one file (the get_dependencies be a bash function inside get_build_order) ? and make a pull request and I would try to incorporate it. Probably generate a list, and then the routines.nu do that list, not the available folders.

[fanzhuyifan]

Yeah, just noticed. Generally its only an issue when "starting new". What I do is disable some feature that makes those cyclic dependencies.

Not sure if it is an Arch packaging issue.

Pretty cool script. Could you make it all one file (the get_dependencies be a bash function inside get_build_order) ? and make a pull request and I would try to incorporate it. Probably generate a list, and then the routines.nu do that list, not the available folders.

Sure, could you be more explicit about what the interface that you want? What should be the input and what should be the output.

[alexjp]

Sure, could you be more explicit about what the interface that you want? What should be the input and what should be the output.

The interface looks pretty good! a list of folders and outputs a list of folders ordered (being the folders, packages).

I will use it to create a list, which the script will process and build. Seems very flexible and have plans to create a "build_list_of_yyyy_mm_dd_HH_MM.json" which has each package as "build/already up to date/failed" for easy checking and fixing.

At least, that is the plan :) Agree?

[alexjp]

I integrated the list with order of dependencies, which the builder now uses.

Instead of ./routine.nu group, with ./routine.nu list, it will use the list.

Maybe close this one, and do the more detailed list in a new issue. OK with you @fanzhuyifan ?