Redesign Aura

[fosskers]

Create a design document in .md format for easy viewing on github. This also allows for code snippets to be added. The main section needed is Functional Requirements, as many of Aura's features were tacked on later because the use cases they involve were not known about at first. As many use cases as possible should be considered before recoding begins.

Related issues:

• Hotedit: #208
• Install method: #213 #214 #203 #125 #114 #107 #202
• Package signing: #211 #38
• Help: #111
• Colour: #102 #158 #202
• customizepkg: #93 #143 #198
• Dependencies: #215

[fosskers]

There needs to discussion about --hotedit and customizepkg. @dhouck I merged your pull request, and things looked good, but when I got thinking about it, and read this I think forgoing any PKGBUILD editing might be the way to go.

The Problem

Aura's current behaviour is this, assuming --hotedit:

0. Call API to confirm existence of package.
1. Download PKGBUILD.
2. Write to a temporary file.
3. Open it in editor.
4. Read it back as data.
5. Download package sources (PKGBUILD on AUR, .install, etc.)
6. Overwrite that PKGBUILD will the edited one.
7. Check dependencies
8. Call makepkg

With AUR 3.0, dependency information is going to be available via the AUR API, meaning the PKGBUILD will never have to be downloaded in first place. Fewer HTTP calls mean a faster Aura. The process becomes:

0. Call API to confirm existence of package and get its dep info.
1. Check dependencies.
2. Download package sources (PKGBUILD on AUR, .install, etc.)
3. Call makepkg.

As dep checks happen before the PKGBUILD is downloaded, editing the dep fields in the PKGBUILD is pointless. If the user were given the opportunity to alter the deps given by the API, they would also have to alter the PKGBUILD in the same way, or else makepkg would crash.

The Case for --(hot)edit

Some packages require you choose a certain build method by (un)commenting certain sections in build(). xmobar-git is one such example. Which is better:

• Allow the user to use --edit to change the PKGBUILD so they can choose their build style, or;
• Disable all manual editing and tell them to build it manually.

[bb010g]

I would still like a --edit.

[fosskers]

@bb010g Which would allow the user to edit all files in the downloaded tarball? They would have to understand that editing the deps fields wouldn't do anything.

[bb010g]

Just have it edit the metadata from the AUR. (That could go well with something like customizepkg built in, but acting on those fields.)

[fosskers]

I'll have to see what the shape of the metadata is before I can make a call on that.

[dhouck]

For the ABS, wouldn't you still need to read the PKGBUILD to find the makedepends and checkdepends (even if you use the local binary package database for the regular and optdepends)? Given that, would it really be simpler to use the AUR's dependency list instead of reusing the dependency checking code needed for the ABS?

Also, for the current (pre-AUR3) workflow, what's the point of doing steps 1 and 5 separately? It seems like it might be more efficient to do the following, but if this approach is missing something crucial:

0. Call API to confirm existence of package.
1. Download package sources (PKGBUILD on AUR, .install, etc.)
2. Prompt for edits; open relevant files in editor
3. Check dependencies
4. Call `makepkg`
EDIT: Start list at 0 to match ones above

[fosskers]

Question: How much dependency information is stored in the database? Are {make,check}depends excluded?

As for the current flow, you're probably right that doing those things separately is silly. There are redundant http requests, and the size difference between downloading a .tar.gz and just the PKGBUILD is nearly negligible. The only problem I can see is getting back into a folder once you've untarred it. That is, with your suggested flow, each tar would be downloaded, opened, and have their PKGBUILD read. Then aura has to back out of that folder, go into another one, and after all dep checks are finalized, go back into the folders in the proper order and call makepkg in there. So the problem is, right now all tars are downloaded into temp folders that aura's haskell underpinnings delete automatically after building. This gives the following sort of folder hierarchy:

foo1234            # temp name created by haskell
  |--- foo.tar.gz  # the downloaded tarball
  |--- foo/        # the tarball's uncompressed contents
        |--- PKGBUILD
        |--- foo.install

And every folder for every package would look like this. Navigating from folder to folder would take some regexing on package names to find out where to navigate to. Right now, it just does it one at a time. The pre-downloaded PKGBUILD is held internally, dep checks are done, packages are sorted into a nice build order, and then one-by-one their tars are downloaded and built.

Regardless, we don't have to worry about any of this in Aura 2 because of the new API coming, and the libalpm bindings to rely on. I agree with you that the current method is inefficient, but until Aura 2 it probably won't be changed.

[dhouck]

I assume the question at the top of your comment was about libalpm, because AUR 3.0 handles all four types of dependencies and even allows separate dependencies in split packages (which I don't think Aura correctly handles now).

I believe ALPM only records regular and optdepends (which makes sense, given that it's designed for binary packages and therefore doesn't need to know about the others). The ALPM doxygen documentation only gives functions to retrieve these two dependencies (or it puts the makedepends and checkdepends functions somewhere other than where it puts the depends and optdepends ones. Thus, unless it's hiding somewhere I (and Google) haven't seen it, it's undocumented, or you want to drop ABS support entirely, you'll need to get these fields from PKGBUILDs directly when using the ABS.

Because of the ABS, the updated workflow you posted might not always be usable and some variant of the existing one might be necessary, at least sometimes, so I have another question about it. Would there be a problem with having a single temp directory with all the packages under it, like below:

Aura1234            # temp name created by haskell
  |--- foo.tar.gz  # the downloaded tarball
  |--- foo/        # the tarball's uncompressed contents
        |--- PKGBUILD
        |--- foo.install
  |--- depend.tar.gz # foo depends on depend
  |--- depend/
        |--- PKGBUILD
        |--- depend.rc

[rmarquis]

Just a few remarks, since I had also time to think about these issues for another project:

Support for customizepkg is dropped, as AUR 3.0 provides dependency information via its API. Support for --hotedit dropped. As dependency checking is done through the API and not the PKGBUILDs, editing PKGBUILDs could cause a discrepancy between aura and makepkg. If you need fine-grained control over how the package is being built, building it manually with makepkg is recommended. If you're trying to fix a broken package, please contact the maintainer.

I fully agree with the above. There shouldn't be any reason to modify dependencies in the PKGBUILDs in the first place. And if there is one, an additional PKGBUILD should probably be uploaded in the AUR instead of doing "magic" locally (that's the very reason I believe the idea behind customizepkg is silly in the first place).

An --edit flag should be kept for special cases like xmobar-git, although any modification to the deps fields would be ignored.

Just have it edit the metadata from the AUR.

The .AURINFO metadata is only useful to make the complete info available in the RPC interface in a secure way (and adding split package support in the AUR too), so "editing the metadata" would be the equivalent of editing the Json response.

Then, it's possible to edit the json info on the fly, if you keep one info call per package or dependency. But Aura should use the multiinfo RPC call instead, because it is way faster as it requires only one call per level of dependencies, no matter what is the total number of dependencies in the transaction.

Of course it would also theoretically be possible to edit the returned multiinfo answer, but that's would probably a nightmare for the user in practice. Better forget the customizationpkg thing entirely, it's not worth it (and silly in the first place imho).

or you want to drop ABS support entirely, you'll need to get these fields from PKGBUILDs directly when using the ABS.

I know some core pacman developers are interested in integrating ABS closely with pacman itself (read it some time ago on the ML). I haven't seen any patch for it yet, but I know this will eventually happen. Maybe in two weeks, maybe in two years, I don't know. So it's up to the aura dev to decide if it is worth it integrating it now, or if the extra work and time would be better invested in some other area of aura.

[fosskers]

Premise: If you know enough about package building to know you want different dependencies, you're smart enough to build them manually while hand-editing the PKGBUILD. Conclusion: No dependency editing will be supported. If dependencies are broken, the maintainers need to be contacted and the AUR versions updated. Things like --hotedit are actually letting packages lag behind if users are able to edit everything themselves on the fly, because it doesn't force the maintainers to keep up to date packages. And if the users really want custom deps that badly, they can building manually as stated above.

An --edit flag will be used purely for editing build behaviour. Any dep changes will be ignored by Aura (but notably, not by makepkg).

---

And yes, one multiinfo call for everything will be quite nice.

[dhouck]

Since you've decided to keep hotediting, is there a reason not to also keep customizepkg support? If it's made clear that customizepkg also cannot be used for Aura's metainformation useful for changing building behavior when you want to make the same changes to a new version's files. Often, you want to keep the same changes whenever you reinstall the package. With customizepkg (or at least customizepkg-scripting or customizepkg-patching; I don't know about the original), this can be handled automatically; without, you need to re-run Aura with --hotedit (and -k, but I always use that) in order to see exactly what changes you made and to make them again.

I'll agree that changing dependencies isn't that useful, but there is a case where "contact the maintainer" or "release a clone package" aren't better options: adding optional dependencies to the depends array (because you want the dependency, and installing optional dependencies still doesn't work well whether or not you use --asdeps even if things are getting better). In this case, it's possible to install the optdepend --asdeps manually, then to make the changes to the package and just not worry about Aura not knowing. This would be a bit hacky, but would still work even without Aura tracking package metadata changes.

[rmarquis]

Often, you want to keep the same changes whenever you reinstall the package.

What prevents you from uploading your own version in the AUR? I don't see in which way that wouldn't be a better option than patching again and again a package locally.

[dhouck]

In general: when somebody wants to make a change to an package and they believe their change would only benefit a small number of people (with others neutral or disliking the change), it is generally not worth uploading the package to the AUR. It's a minor hassle for the maintainer (and doesn't get rid of the problem of "patching again and again a package locally"), and if only a small number of people would prefer their version to a version already in the AUR or the official repositories, it doesn't give much benefit. Often, this change may only be useful to that one person (one example might be somebody who wants to configure dwm but still have it managed by a package manager, but I'm not sure if this example works because I don't personally use dwm or any of the other several tiling window managers that require recompilation for a configuration change ).

The specific package that prompted my earlier comment: There's a package in the community repo called terminus-font. There are also four versions of it in the AUR with different configurations, only one of which has much support and none of which are up-to-date. I want a fifth configuration which is almost but not quite identical to one of the ones without much support already in the AUR, so uploading the changed package to the AUR (and then maintaining it) probably wouldn't be worth the effort and might make things harder for anybody looking through the AUR (because they now have more nearly identical choices and they're unlikely to want my particular changes).

[fosskers]

Could you not just keep a local PKGBUILD that you bump the version of every once and a while? customizepkg has in the past introduced a number of problems (which to use? how to catch its errors? is it secure at all?) Making aura handle customizepkg properly is probably much more work than the payoff would be.

You mention avoiding confusing AUR users with multiple versions on the AUR. I agree with this. We've arrived at one of the fundamental problems of the AUR: it's filled with garbage. Vast expanses of 0-vote, out-of-date, orphaned packages. Is there any system in place to have these removed?

In the mean time, aura -A{i,s} show maintenance status and votes. That's about the only tool we have right now for telling what packages are legitimate.

[rmarquis]

You mention avoiding confusing AUR users with multiple versions on the AUR. I agree with this.

If the modified version is useful for someone, chances are that it is useful for another one at least. If the pkgbuild is kept up-to-date and has a logical name (software-mydiff), I don't really see why it shouldn't be uploaded in the AUR. The fact that it is popular or not is just a side effect.

We've arrived at one of the fundamental problems of the AUR: it's filled with garbage. Vast expanses of 0-vote, out-of-date, orphaned packages. Is there any system in place to have these removed?

Unmaintained or obsolete packages are indeed a big issue. There have been some discussion in the forums and on the aur-dev mailing list, but nothing is set on stone yet.

When I read the following, I really hope the entire AUR content get wiped and starts again on a clean basis:

There are thousands of packages using old PKGBUILD syntax in aur. 34.54% of all aur packages will stop working at pacman 4.2 release. If you also count packages that are plain broken, duplications, outdated and so on, I would not be supriced if the total number is close to 50%,

Seriously, the aurpocalypse is going to be fun.

[fosskers]

When I read the following, I really hope the entire AUR content get wiped and starts again on a clean basis.

I would support that.

[dhouck]

Am I to understand that you don't want to implement customizepkg support, but you wouldn't object to its existing if somebody else were to write it? Or do you expect that it would cause too many headaches even if it was well written initially?

Also, what about a more generic hooks mechanism? ALPM will probably get hooks at some point, but it might be useful to have AUR, ABS, or building specific hooks. They could be used in general to implement customizepkg support, namcap support, etc. It seems like that could be a useful feature to consider when doing a redesign even without the recent conversation.

[fosskers]

I think it's a very good idea (thinking for future version of Aura as well, moved to other distros) to implement some sort of plugin/hooks system so that the build process can be highly customized.

[fosskers]

@dhouck I've made three proposals on how the plugin system could be implemented. What do you think?

[flamusdiu]

@fosskers

Option 3 gives the most flexibility though the CON (many deps) could really cause havoc with the build system.

Option 2 I think is good middle ground and works somewhat like PKGBUILDs now (different areas you can write and such.

Option 1 Too complexed really and honestly Option 2 can give just as much flexibility as 1.

[fosskers]

could really cause havoc with the build system.

How exactly? One other thing I thought of was that non-Haskell plugins wouldn't be forced to compute things via the Aura Monad (in particular, via the Settings held within it). That said, the settings could always be passed to the plugin in the JSON, and they would have to obey it.

...good middle ground...

Unfortunately I've never coded with Haskell actors before and I believe for me personally this would be the hardest to implement. Another pro that hasn't been added yet is the opposite of the above: Operations could be guaranteed to run through the Aura Monad.

Too complex really...

I don't know if I agree, but the whole "pre-built version users can't configure" thing is a big con, I think.

[bb010g]

Too complex really...

I don't know if I agree, but the whole "pre-built version users can't configure" thing is a big con, I think.

We could just do something like aura-contrib for all the good ones.

[fosskers]

That applies to any of the options.

[fosskers]

I'm considering going ahead with Option 1 for the following reasons:

• AuraConf.hs replaces pacman.conf, and becomes a place to define all relevant file paths (package cache, etc.) and build hooks, finally giving us a conf file.
• Distribution of hooks is simple via Hackage packages.
• Turning hooks on/off is just a matter of removing them from whatever hooks list they were a part of:

hooks :: AuraHooks
hooks = defaultHooks

becomes:

import Aura.Hooks.Customizepkg

hooks :: AuraHooks
hooks = defaultHooks { pkgbuildHooks = customizepkgHook : defaultPkgbuildHooks }

• Aura's default behaviour just becomes a matter of defining default hooks.
• Aura can be heavily customized for use by other Distros by ripping out all the default hooks and supplying new ones specific to that distro. The core of Aura then becomes a hook manager and dependency resolver.

[rbellamy]

If you're building for cross-distro support, then might I suggest to not call any hooks "default" but instead name them by distro, for instance, rather than defaultPkgbuildHooks, the name would be archPkgbuildHooks or rpmSpecHooks.

The idea that "AuraConf.hs replaces pacman.conf" violates DRY, and strikes me as a smell, unless archPkgbuildHooks parses pacman.conf and AuraConf.hs provides overriding behavior. I guess I'm saying that AurConf.hs should extend pacman.conf rather than replace it... but I have no idea how you would do that and adhere to the cross-distro principle, so maybe I'm talking out of both sides of my mouth.

Finally, the "Distribution of hooks is simple via Hackage packages" begs the question completely - so far, for me, the haskell-* packages via pacman are anything but simple, and have been fraught with issues.

[dhouck]

I agree with @rbellamy that the way Arch currently handles Haskell is pretty bad. When I first tried installing Aura it was completely impossible to do so using just the default repos and the AUR because not all of the necessary packages were provided. I could probably have done so with the Haskell-specific repos instead of using aura-bin, but I didn't want to use those because (to the best of my knowledge) I couldn't integrate them with ABS (and other people have other, more substantial reasons).

Option 1 seems to nullify most of the advantages of making aura-bin the default and/or of having a configuration file. If somebody uses the configuration file, they need to have all the dependencies for aura --recompile, which if I understand correctly would be most or all of the dependencies of Aura itself. Option 2 requires that some Haskell dependencies exist, but if I understand correctly there would be far fewer. I believe Option 3 is the only one that preserves all of the advantages of making aura-binthe default.

[bb010g]

If we're going for cross-distro, Option 3 would probably be the best choice. Note that 3 does not exclude 1, as 3's functionality could from from a 1 plugin. Have a plain-text config file plugin, and you're good for not recompiling with 3, yet doing 1 style if you want. (Also, I'm excited about cross-distro Aura. It sounds awesome!)

[dhouck]

@bb010g, that seems like by far the best solution.

It might also be worth mentioning that at some point in the future ALPM (and thus Aura) is likely to get pre/post install/uninstall/upgrade hooks (but this almost certainly won't cover makepkg), and these will look be in the .desktop/.ini/systemd-style (is there an actual name for this kind of config file?). The format for those hasn't yet been finalized. I don't know if it's a good idea to deliberately imitate this format or deliberately use a separate format, but it's something to consider.

[rbellamy]

@dhouck they're called units, right?

[fosskers]

Having Aura plugins written in other languages would open up a world of connection to other tools/libraries. I see that as a really good thing. There are no guarantees about the behaviour of those plugins, though.

@bb010g The idea with Option 1 is that it preserves type safety and the APIs for input/output to the hooks' functions would be very clear, as everything is Haskell. It would also allow the hooks to directly access Aura types, as well as Aura utility functions (IO, number parsing, error handling, etc).

--recompile is an issue. Probably the make-or-break one. As @dhouck points out, if we have aura-bin as the main package, that will leave most users with no options when it comes to custom configuration. We could offer "preconfigured bundles", but now I foresee that being a slipper slope of an AUR filled with custom aura-* packages.

Our priorities are:

• The main package aura being a prebuilt binary to avoid Haskell deps. (or building via cabal sandbox?)
• Configurable at any time.
• Plugins/Hooks that offer customized behaviour so as to allow Aura to become cross-distro.

Now, are all these possible with a Haskell-based solution (to have a nice API)? Does it not seem like Option 2 allows this?

[bb010g]

@fosskers I haven't dealt with actors before, and a quick Hackage check of actors shows a package that's only been updated twice. (I may be looking in the wrong place, though. If you're thinking of something different, please point me there.) I would prefer Option 1 as well, but that purity may be hard to maintain in other distros (and even Arch) where Haskell support is spotty or nonexistent. Option 3 would provide a way for it to be usable and expandable without much in the way of compilers. (Unless aura-bin came with ghc...) It's really the same issue XMonad has. They decided to go pure, but with heavy dependencies, which puts them at a bit of a disadvantage versus something like Awesome, which only needs Lua. The only problem I see with a 3 on 1 method is plugin fragmentation, but we can cross that bridge when we come there. It's probably a good idea, KISS-wise, to go with Option 1 first.

[fosskers]

I don't think we should we should stray from an all-Haskell solution because "Haskell isn't as popular". Something will only become popular if it's used, and to be honest propagation of Haskell is one of Aura's (minor) goals.

For option two the right packages would be something like concurrent or distributed. There's a really good book on the subject from O'Reilly that I've been meaning to get into. I'll have to anyway, since building of non-interdependent packages will be spread across cores (see here)

At the moment, I think Option 1 would be the simplest to implement.

[flamusdiu]

@fosskers I think staying with an all-Haskell is a good idea and should help increase the flexibility of the application.

Maybe I should try to pickup Haskell again. =)

[dhouck]

Sticking with all-Haskell might be a good idea, but I don't see how it could increase flexibility. It seems to me that it's more flexible to be able to write a plugin in shell, or Python, or C++, than to force people to use Haskell.

In either case, you said the hardest part about Option 2 would be that you didn't know some of the required concepts. If you need to research them for concurrent building anyway, I think Option 2 is better than Option 1 in pretty much every way. The only thing on the Pro side for Option 1 in the design document that doesn't fit with the current design of Option 2 is the ability to enable/disable plugins, which still seems possible to implement in Option 2 with either a non-compiled config file, symlinks in /etc like systemd, or any number of other options. Not listed in the design document is the difficulty of implementing, but I think that the easier logistics might compensate.

[fosskers]

We could release a Hackage package called aura-types that plugins would have to import and use to ensure type safety. I think we could "turn off" installed plugins by using a non-compiled config file.

Questions

• Are there any Haskell packages that offer a config file format/parser?
• Is it possible to pass Haskell data between Haskell processes that aren't copies of the same binary?

[flamusdiu]

@dhouck Couldn't there be a hook that works for third party scripts (kinda like what option 2) that really just runs that tool. I doubt it would be very safe though and how would aura know the hook ended? You would need to return something in a standardized way. Something like a external-build hook that just lists the scripts with the order and args. However, this would not be something I would enable by default and probably documented as unsafe due to the inability for aura to insure the safety of the scripts. Though, this is breaking using only Haskell.

[dhouck]

Yes; any hooks system that is powerful enough to let you run customizepkg is powerful enough to let you shoot yourself in the foot.

[fosskers]

Plugins written in other languages could be wrapped in Haskell and packaged as an Aura hook couldn't they? (+1 option 1/2)

[flamusdiu]

@dhouck I am aware. I actually like the idea behind customizepkg and what users could do with it but you could cause some fun issues with it.

[flamusdiu]

@fosskers I guess it would depends on where these hooks were saved. You would still need to setup basic Haskell build env to package the hook as a Haskell package as well. Where did you plan to store the hooks at? /etc/aura/hooks?

[fosskers]

For Option 1, the hooks would just be normal Haskell packages and thus in ghc's PATH when using aura --recompile.

For Option 2/3, the executables would have to sit somewhere central. /usr/local/aura?

[flamusdiu]

@fosskers gotcha. For Option 1: how would work if I had an external tool named mytool.sh package as a Haskell package? How would aura be able to run the tool and get something in return? (Not overly familiar with Haskell package and how to access them -- I was reading on this today though.)

[fosskers]

You could just make a hook as a haskell program, and it could do a shell call to your script. Error handling would be done by the outer hook and pass data back to Aura in the Aura Monad.

[fosskers]

We spoke in #235 about keeping flags consistent across distros so that using Aura is always the same, even if it's pulling packages from different sources and building things in different ways.

Luckily, -S and -A don't lose relevance. -S still stands for sync and -A can be thought of as "auxiliary" or "additional" for packages that are installed in other ways.

To recap other points:

• Aura's mission is now to become a cross-distro package manager. This is possible because most behaviour specific to the distros will be defined in Hooks. (PSA: Let's refer to them as Hooks from now on, instead of plugins)
• Use of libalpm will become specific to Arch-like distros.
• Aura becomes a Hook manager, dependency resolver (I have a great idea for this), package state backup manager, and package downloader (like what powerpill does right now).

[fosskers]

See the new mission statement.

[bb010g]

• Dependency management.
• Package downloading.
• Package-state backups/restoration.

Does this include unified querying, removal, and updating? It'd be annoying to have differing functionalities/interfaces for those. (Also, are -S and -A going to be built on the same base, just with different scopes?)

[fosskers]

Yeah, the way you do everything will be the same. The inner mechanics may differ. A -Si foo will produce the same output regardless of distro. The inner workings of -i may be unique, but its output will be passed to Aura and output by Aura in a standardized way.

...built on the same base...

What do you mean by "same base"?

[bb010g]

Same internal calls, just focusing on either main or aux. (Now that I think about it, it would be hard not to with the new hook structure, forget I asked.)

[fosskers]

I understand your original question now. And yeah, the hooks should take care of most of it.

[rmarquis]

dependency resolver (I have a great idea for this)

May I ask more details about this? From my perspective, the best dependency solver I came across is the SAT solver libsolv from openSUSE (used in libzypp/zypper, and also soon in Fedora with dnf).