libaur: plans for a common AUR helper library

[AladW]

Continuation from #565. The idea is that instead of duplicating code and functionality across AUR helpers, there would be a central AUR client library handling tasks such as dependency resolution, interacting with aurweb, SRCINFO, file inspection, and so on. Since this wouldn't be really something for end-users (to abuse), and the AUR misses many features on the server-side relevant to helpers (sometimes by design), it might eventually be included to [community].

A natural choice of language would be C, since it can be called from most other languages. An existing effort in this direction, based on the cower code base: https://github.com/kyrias/libaur Of course, this is not a fixed choice and we could argue for some higher-level language like Go (as yay) or C++ (as auracle).

cc. @kyrias @eli-schwartz @polygamma @kitsunyan @rafasc @maximbaz

[maximbaz]

The idea is interesting, but I'm curious to see whether there will be interest from developers of other AUR helpers — after all, to some extend this logic is already implemented in all of them (for example, I'm very happy with aurutils at the moment 😛 ), and your proposal will bring nothing new besides reduced codebase and possibly heated debates over the design of the new library. But let's see 🙂

For the language, I strongly vote for using Go, it is a simple language with good set of tools (so the development is easy and pleasant), that can be compiled in C shared library (.so) (so that it can be called from other languages).

[eli-schwartz]

it is a simple language with good set of tools (so the development is easy and pleasant)

As though no other language has a "good set of tools"? I'd expect every language whatsoever ever would strive towards that, and a fair number have succeeded. Moreover, this is a quite vague claim. I'd also claim that the language semantics aside, the tools are the worst part of it (golang).

• import paths for source code in many programming languages show up like #include <foo/bar.h> or from foo import bar. Golang has import (github.com/authors_github_account/foo/bar) which discourages changing your username let alone forking, because you need to do tree-wide rewrites just to get the compiler to find your code...

Anyway, opinions... markedly differ. I'm okay using programs written in golang if they do something I find useful, but the language itself terrifies me and the implementation and infrastructure proceed directly to horrifying me. :p

e.g. from the perspective of distro packagers, golang is by far the most nightmarish to do properly. It seems to have been designed in essentially every way possible to make the task unpleasant.

• Need I point out how long it took golang just to get reproducible source code, a task which still requires thirdparty tools that are only barely coming to be agreed upon?
• The golang devs seem to regard symlinks as evil and carefully removed support for this standard filesystem feature from the compiler infrastructure. This helpfully prevents many attempts to work around the lack of reproducible source code by utilizing, e.g. makepkg's source=() array instead.

...

All in all, it's one thing to write tools helper libraries for your own tool using your preferred programming language, it's another thing entirely to choose one for what you hope to be "the" universal answer to AUR helpers in general doing common things. At least, the choice of programming language should be one which encourages users of different programming languages to be interested in contributing...

that can be compiled in C shared library (.so) (so that it can be called from other languages

I must admit this is a surprise to me, since every golang project I've ever seen is statically compiled except for, sometimes, libc. And I can think of so many golang projects that would be so much better off using shared libraries, too... e.g. one extreme example is keybase which is 3 different 40MB binaries which contain all the same backend code, it's larger than some browsers and updates approximately 100 times more frequently...

[eli-schwartz]

I wonder why everyone almost seems to forget that https://github.com/archlinuxfr/package-query has existed for years. It's a shame that such an interesting tool is completely forgotten about except as "some unidentified package which seems to be a yaourt dependency so it is probably something terrible".

[polygamma]

I've thought a bit more about this, and there might be another problem... Even if libaur existed as of now, I am not sure that it would actually be used. Take e.g. the dependency solving, I guess there are subtle but relevant differences in how dependency solving is being done in the different AUR helpers which means that it may even be impossible to maintain the current behavior of the helpers if one wanted to use libaur. Also: Since those things already exist in the helpers, why should one take the burden to rewrite the code to use another library to get exactly the same output all in all?

[eli-schwartz]

The most significantly useful thing I could see such a library doing is providing some direct interface to query raw data from AUR packages. Then AUR helpers would not need to provide custom logic to (re)implement the RPC interface, or adapt to new versions. Again, that's something which package-query (a.k.a. expac for the AUR) already provides as a CLI tool.

[AladW]

Again, that's something which package-query (a.k.a. expac for the AUR) already provides as a CLI tool.

Seems like package-query functionality is only exposed through a compiled binary: https://github.com/archlinuxfr/package-query/blob/master/src/package-query.c

[rmarquis]

[rafasc]

e.g. the dependency solving, I guess there are subtle but relevant differences in how dependency solving is being done in the different AUR helpers

But wouldn't that be actually a point in favor of something like libaur deal with the dependency solving? It would be nice if all helpers could agree on how to build a package.

But what I like more is to have the eyes of the community on a shared piece of code rather than having the attention divided by code on multiple projects that is doing essentially the same task. On the other hand, sometimes it's hard to make the community agree on something.

But even for that, helper authors must be willing to change their existing, (hopefully) functioning, code to make use of said lib.

I guess the only way to know is to just do it and see if it sticks.

[AladW]

@rmarquis It worked for systemd!

[Morganamilo]

Originally I had the idea to create another AUR helper. Mainly just to explore the idea of local repos, using a similar design to aurutils. This was mostly just for fun and experimentation, not really a public thing.

Now of course with all the work on yay I didn't want to duplicate all the code, so I had the idea of making a common dependency between both. I talked to alad about it and a more general aurlib that any helper could depend on.

Anyway the discussion here is interesting, although I get the feeling it's an interesting idea and that's about it. Especially with every helper being in a different language, every one would need to write c bindings which just makes things more annoying.

[Jguer]

For yay it wouldn't be as bad, it would be replacing the libalpm bindings by another set of bindings most likely, but there would also be a loss in concurrency by using a C based solution, at least initially and definitely not as nice to develop concurrency in C as in go.

[Morganamilo]

Yeah my original idea. Was to write not-a-aurtils-clone in go so I can share the codebase with yay. So of course the library would be written in go. I was just curious about a larger scale project between everyone.

[AladW]

Just speaking for myself, I wouldn't mind throwing out a whole bunch of aurutils code and replace it with some Go alternative shared by both projects. Only problem: I don't know anything on Go. :disappointed:

[rmarquis]

Only problem: I don't know anything on Go.

That's the issue once you stray away from the language of the underlying tools (C and bash, for pacman and makepkg respectively): the barrier of entry for many authors is raised rather than lowered. To please everyone you'd also have to aim for the lowest common denominator, which would be something like cower --print or package-query.

I could imagine a common python lib shared between the main python projects, but that's about it.

[maximbaz]

That's the issue once you stray away from the language of the underlying tools (C and bash, for pacman and makepkg respectively): the barrier of entry for many authors is raised rather than lowered.

I slightly disagree with this, but it is based on my personal experience and probably does not apply to everyone. Before discovering aurutils I knew some basics about bash scripting, and even though bash is native in makepkg world, understanding aurutils' code and making reasonable improvements was quite difficult for me (who am I kidding, I still don't fully understand how all of the scripts work!). And let's not forget, it is soooo easy to break a bash script. But at least I learned a lot about shell scripting in the process and it is a useful skill to have 🙂

In contrast, Go is a very simple language, the entry barrier for contributions from people of different backgrounds is low, and it's much more complicated to accidentally break Go code 🙂 In my first message I was first of all trying to emphasize the simplicity of the language, not the availability of tools (although both contribute to the fact that developing Go code is easy and pleasant).

By the way, instead of making a .so library I would also recommend considering the package-query approach: make a compiled binary! Let it spit out some JSON, then all you need to agree upon is the structure of "responses" to different requests. It's easy to parse JSON, aurutils can use jq, many other languages have native libraries to do so... and yay will cheat and import the library package from code if both are written in Go 😛 .

[eli-schwartz]

Shared .so libraries can also be executed as binaries if you write them correctly...

But if you just want a common denominator shell tool, package-query works fine.

[AladW]

@polygamma There's no need to implement a complete dependency solver that everyone's supposed to use; you could just have somethin to output JSON, like @maximbaz suggests, and add some (optional) recursive functionality to get all raw information required for dependency resolution.

Resolution itself might then be implemented in an external tool. So sort of like auracle buildorder but with no additional processing.

You could include further functionality to abstract away long-standing issues with the AUR, such as splitting of GET requests, versioned queries, and so on. That way you could close all the server-side bugs with "it's implemented client-side in libaur". Example: https://lists.archlinux.org/pipermail/aur-dev/2018-June/004553.html

[AladW]

If you did that, even a "low level" tool like package-query would in the end just be a wrapper to libaur. Similar to how expac is a wrapper to libalpm.

[ringo32]

just was curius if this was on topic, was thinking to have a bug report, but probably mayby al the same, as example i was on it, i wanted something from Aur, lxqt-archiver-git , but in the depency list of the pkgbuild is there a libfm-qt-git version but i dont want te git version basicly when i edit those, yay want have that package, personaly i expected behaviour , yay should reinspect the adapted pkgbuild ? and go forward on the new pkgbuild, or am i wrong, ? should i make a bug report for this, im not sure ?

[AladW]

How is this in any way related to the original topic???

On Fri, 25 Jan 2019 14:13:58 -0800 ringo32 notifications@github.com wrote:

just was curius if this was on topic, was thinking to have a bug report, but probably mayby al the same, as example i was on it, i wanted something from Aur, lxqt-archiver-git , but in the depency list of the pkgbuild is there a libfm-qt-git version but i dont want te git version basicly when i edit those, yay want have that package, personaly i expected behaviour , yay should reinspect the adapted pkgbuild ? and go forward on the new pkgbuild, or am i wrong, ? should i make a bug report for this, im not sure ?

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/Jguer/yay/issues/579#issuecomment-457749766

-- Alad Wenter alad@archlinux.org

[Morganamilo]

@ringo32 The FAQ covers your question. Anyway this thread has run its course. Closing.