Specification/Packaging Standards Concerns

[Llammissar]

I think this is a great idea, but I'm getting a strong "rubygems" vibe from it and that scares me because unless there are rigourous standards in place, packages turn out like gems and are nearly impossible for distros to package. (cf. http://blog.flameeyes.eu/2011/12/21/gems-make-it-a-battle-between-the-developer-and-the-packager). This is not a fate I'm fond of dealing with, and I have zero good memories of our large-scale rvm deployment at work, so I would like to offer what assistance and advice I might so as to ensure orbit doesn't fall into a miserable state. I'm not sure where or how far you're willing to take this conversation, so for now, I'd implore you to consider this document: http://dev.gentoo.org/~ulm/pms/head/pms.html if not skim the whole thing.

I've used a lot of package managers at this point, but I am now and will remain a Gentoo user because package management is properly standardised and pleasant to deal with. It's a lot like BSD ports, if you've used that, only improved in pretty much every way possible. There are a number of ideas in there that I would love to see cribbed or strongly encourage the adoption of, but the real takeaway for now is probably that there need to be guarantees built into the system to prevent the sorts of issues Diego deals with. Thanks!

Cheers, Wyatt

PS: "I'll probably explore the possibility of having spec and orb files written in D." Why not use rdmd? (TDPL, pg 2)

[jacob-carlborg]

It's not strange you get a Rubygems vibe, I've basically copied Rubygems and adopted it for D.

I've read the blog post and I can say this:

1. I don't understand why anyone would want to package a gem in the distro's package manager in the first place
2. There's a lot of complains about gems not being maintained and not compatible with each other and different versions of Ruby. I don't see how this can be easily solved. There are only complain and no solutions.

Aren't most packages in the distro's package managers supported by the distro and managed by someone from the distro community and possibly hired and paid by some company? They're adding patches to make sure the packages work on their distro and so on. Rubygems doesn't have that luxury and I'm pretty sure we will neither.

I see you have had problems with Rubygems. I on the other hand have had problems with Portage. Packages that don't compile and when they don't you're completely lost (at least I am). I also had the problem of not updating my server often enough. It ended with dependencies that couldn't be met. Package B depends on a newer version of Portage, can't install newer version of Portage because a newer version of Portage is required. Older packages are not available in the repositories anymore. Problems like that.

BTW, RVM isn't really made for deploying, it's made for primary for development. This makes it a bit harder to adapt for a global installation for instance.

I would much more prefer that you tell me if there are any specific problems you see and if you have any solutions for them.

I don't know if you seen this already but I've written down my ideas for a package manager:

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

This ideas started out just as ideas and I wrote them down so I didn't have to repeat myself on the newsgroup. But now I've started on this project the ideas can be considered the specification and will gradually become more detailed and complete.

I know about rdmd and use it all the time. If I decide D can be used for spec and orb files rdmd will most likely be used.

[Llammissar]

Oh wow, I'm sorry. I've been meaning to respond to this for a few weeks, but things haven't calmed down since I got back from my vacation (maintenance windows and migrations; ah, to have an intern!) Anyway, this is pretty long, so feel free to take it in chunks. First up, your initial response:

1. Wow, really? It's because any software on the system that isn't managed by the system package manager is an administrative liability. They install themselves wherever they like, overwriting whatever they well please, not paying any mind to what they might be breaking, modifying system PATHs and generally polluting the environment. This is or has been true of gem, pear, cpan, pypi, makefiles, even RPM at times. Between Gentoo and Debian, I think it's pretty clear there's demand for this sort of thing. And considering the distro is packaging your software to help you (rooting out QA bugs is very important for production environments), it doesn't seem unreasonable to "...hav[e] tests around, a clear repository, the ability to rebuild documentation and tests without bringing in a whole lot of gem-building tools – jewelwer, hoe, and so on – and most importantly code that works and tests green...." Does it?
2. Diego has been writing about just this for two years, now, and has explained his reasoning at every turn. http://blog.flameeyes.eu/2008/12/14/rubygems-cpan-and-other-languages sums up the idea pretty nicely and can be summarised as "Be like CPAN so we can wrap it with a PM module." If you don't want to get super-involved, would that be an acceptable middle ground?

I understand that you've had issues with Portage. I appreciate that; dependency resolution is a difficult problem and, depending on when you were using it, some kind-of-awful things might have happened (the libpng soname change was a particularly egregious example). But I don't think these experiences invalidate what we've learned about package management and what makes a good, even enjoyable, experience for working with it. More on this in a bit

Having taken the time to sit down and actually go over it with a fine toothed comb, here are my specific impressions. This may seem nit-picky at times, but all points I raise are born of experience. If I seem negative, apologies in advance. Please consider it a function of my passion for the subject; I take package management very seriously. ;)

Integration:

• Source Code Management: This is dangerous because it gives no guarantee that another entire source tree isn't included. Package files should be maintained separately from their source trees. You can see this problem up close if you attempt to package the mongo_mapper gem, among other things. Including an entire separate source tree can cause collisions, conflicts, and keeps old rotten code around. It's just a Bad Idea. Most importantly, we need to avoid this: http://blog.flameeyes.eu/files/bones-dependencies-graph.png That's insidious and should be horrifying for you as a developer.
• Packages in D files: Please don't do this. This causes far more issues than its worth. When you do this, you wind up with multiple mutually incompatible modules and it's terribly messy. It's a pain to deal with for everyone because as your package graph will end up fanning out at a terrifying rate. Moreover, it breaks the canonical D module system for zero effective gain. You cannot guarantee an end user will be using orbit to manage their D packages. You can guarantee that a D user will use D conventions in their code.
• DVM should be optional, not required: I wasn't actually aware RVM had been replicated for D. Given my history with it, I admit I'm not especially happy about this. RVM, as you mention, is a developer-oriented tool. I want to know: is there anyone who is NOT a release engineer that consciously keeps and uses multiple versions of DMD? If other development communities are any indication the answer is a firm "no" greater than 99% of the time. Multiple versions of the same package should only exist simultaneously when the interface changes between versions. GTK+ 1.2 and 2.6. In Gentoo, we call these slots; name them whatever ("rings"?), but the real important point is end users shouldn't have to even think about developer-oriented tools if they're not managing releases. -- If keeping multiple versions of any given package around is truly important, it would be best to look to eselect as an example of good practice. Granted, in this case, it would need to be a smarter tool because orbs may have dependency on other orbs of a particular version specification that would also need switching. Basic sanity checking that helps cut down on completely avoidable frivolous bugs of the "this isn't compatible with that version" nature.

Orb:

• Installing packages: No package spec should ever contain multiple packages. In fact, doing so is a clear violation of the entire principle of having a package manager in the first place. This is a point I cannot stress enough; the sort of thing that prevents projects from being packaged at all. It's slightly more work, but we will all be much better served to enforce this strictly. Orbs should contain dependency metadata; dependency resolution should be conservative. If orbs can contain multiple packages, then it creates a nasty source of bugs because what if the orbs you install have different versions of the same package included? Which gets used? These aren't things you can afford to leave to chance because it makes the whole language very unpleasant to use at all.
• This whole point is about UI and UX. Having used a lot of terminal applications over the years, the ones I always come back to tend to have sane defaults (except for find, but it's just too useful; I can't help it). Anyway... -- The default action any user wants from a package manager is to install a package. Generally they know the name (within reason) and aren't trying to remove things. Installing should be the natural assumption. This makes the tool pleasant to use. To install dwt, it should be sufficient to invoke: orbit dwt or similar. And, in absentia of any flags, it should select the latest stable version and install it. -- Version specification: a specific version will only rarely be necessary for ordinary users, but will likely be a rather common check for dependency resolution. Since it's primarily useful for your own internal dependency resolution, specifying it properly simplifies a lot of graph evaluation and renders the --version option superfluous as a fringe benefit: orbit dwt-0.3.1 to install that version. -- Selecting versions: I alluded to this above, but I would actually suggest a separately-named tool for this. This is where the management part of "package manager" comes in and it's a litmus many PMs fail utterly. Your tool needs to be able to show which packages have multiple versions (orbiter list), list available versions for those packages (orbiter dwt list), show which is selected (orbiter dwt show), and set one of the list (orbiter dwt set 2). This is a significantly different task from installing and uninstalling and should be treated as such to enforce the contextual relevance. And it doesn't ever hurt to make things more pleasant to use. Hitting one number rather than entering a version sounds like a smaller difference than it really ends up being, UX-wise, especially when you consider that it frees up your namespace for flags. Every keystroke really counts, even if you don't always acknowledge it overtly. -- Commands: In keeping with the theme above, Flags reduce typing and make day-to-day use more pleasant. The common actions you want in your package manager are installation (no flag), searching package names to see if a library has an orb (-s, --search), searching package descriptions for things you don't have a name for yet (-S, --searchdesc), displaying the dependency graph of a package and asking permission before continuing to install or remove (-a, --ask), updating a package or set of packages to newer versions (-u, --update), and removing a package you no longer need or want (-c, --clean (or --crater, if you're feeling cheeky)). And to be sure, this also implicates combinations such as orbit -au derelict where you want it to resolve the dependency tree and ask for your blessing before it updates.
• I have to admit this section has me a little worried. "build"? "upload"? I sincerely hope you're not planning on distributing binary packages. For a programming language, this is not good. It assumes the tests pass in the environment they'll see real use in and is incredibly unfriendly to distro packagers. Many people are content to use Gems without running the tests and it has lead to some terrific bugs. You cannot package something without running the tests. -- If you're only referring to the packaging of the sources and spec, I would still recommend putting that in a separate utility (orb-roller or orb-polisher or something) for the same reasons that version selection is a separate tool. The idea is to clutter your package manager with useful features, not releng features. Very few people care about releng but release engineers (which is why I'm here making an ass of myself). Package files:
• As you probably guessed by now, I'm going to say just go with orbspecs. Binary is bad. -- Put the metadata in the orbspec and fetch the source tarball when told to install; that makes things simpler for packaging and upstream. If you're dead set on supporting binary libraries, well, this isn't incompatible with that (though, on behalf of the end users, I ask that you not do that unless you have a good package-review infrastructure and are willing to maintain tracking for different architectures)
• Don't let things into the blessed repo without review and testing; that will poison the project. Use and abuse rsync; orbit --sync to grab a tree of orbspecs from a community-maintained repository. One may optionally add other repositories to a file and they will be considered at the time of dependency resolution (Gentoo's overlays). Orbspecs, in addition to the other metadata, also contain a URI for the package. wget the tarball and continue as normal.
• Using a programming language is the correct decision, though I strongly question the ubiquity of Ruby. For Windows and Linux, Perl and Python are much more likely to exist on any given machine; nearly everything has them. Another candidate might be Lua which, if not terribly common, is a very lightweight dependency. Orbspec Spec:
• Oh, nice. Yes, this will do nicely. I like it. Sure, should you choose to take my recommendations seriously, some of those fields will disappear, but on the whole this looks pretty good. -- Though rather than using a list of files, use digest files to store sha1 hashes for different tarball versions of a package. This way you can easily, automatically, verify the sources you fetch. In general, a properly-packaged tarball has only the necessary files and nothing more-- I'm not sure why gems can't seem to get that right. -- Keep track of the files that are installed on a per-package basis and you can check for collisions with other packages. Depending on your implementation details, this could be very desirable.

I hope that, with this examination it becomes clear that I'm not just trying to be contrarian. I strongly believe there is a lot to be learned from what Zac Medico and others have done with portage and its related tools while acknowledging that it has flaws and features that aren't necessarily going to apply to a language-specific PM. You've put yourself in an interesting place within the community because what you do could ultimately bring people scrambling to use D because it's enjoyable from end-to-end or ultimately hold many excellent programmers at bay because the tools don't encourage good community practice.

Best, Wyatt

On Sun, Dec 25, 2011 at 15:02, jacob-carlborg reply@reply.github.com wrote:

It's not strange you get a Rubygems vibe, I've basically copied Rubygems and adopted it for D.

I've read the blog post and I can say this:

1. I don't understand why anyone would want to package a gem in the distro's package manager in the first place
2. There's a lot of complains about gems not being maintained and not compatible with each other and different versions of Ruby. I don't see how this can be easily solved. There are only complain and no solutions.

Aren't most packages in the distro's package managers supported by the distro and managed by someone from the distro community and possibly hired and paid by some company? They're adding patches to make sure the packages work on their distro and so on. Rubygems doesn't have that luxury and I'm pretty sure we will neither.

I see you have had problems with Rubygems. I on the other hand have had problems with Portage. Packages that don't compile and when they don't you're completely lost (at least I am). I also had the problem of not updating my server often enough. It ended with dependencies that couldn't be met. Package B depends on a newer version of Portage, can't install newer version of Portage because a newer version of Portage is required. Older packages are not available in the repositories anymore. Problems like that.

BTW, RVM isn't really made for deploying, it's made for primary for development. This makes it a bit harder to adapt for a global installation for instance.

I would much more prefer that you tell me if there are any specific problems you see and if you have any solutions for them.

I don't know if you seen this already but I've written down my ideas for a package manager:

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

This ideas started out just as ideas and I wrote them down so I didn't have to repeat myself on the newsgroup. But now I've started on this project the ideas can be considered the specification and will gradually become more detailed and complete.

I know about rdmd and use it all the time. If I decide D can be used for spec and orb files rdmd will most likely be used.

---

Reply to this email directly or view it on GitHub: https://github.com/jacob-carlborg/orbit/issues/2#issuecomment-3271534

[jacob-carlborg]

Ok, that might be true. But I still think the package manager provided by the OS is different from a package manager like Rubygems.

As far as I understand it Diego likes CPAN because it uses a well established archive format, tarball or zip and a specific page structure. That is just what Orbit does as well: https://github.com/jacob-carlborg/orbit/wiki/Orb-Package

• Source Code Management: I'm not sure that I follow what you mean. But I don't think we will have the luxury of special package managers that build and maintain packages. It will be the developer that will package the software.
• Packages in D files I'm not entirely sure I understand
• DVM DVM is completely optional as you can see here: https://github.com/jacob-carlborg/orbit/wiki/orbspec%20specification Only two fields are required, so far. It seems like you mix up things here, compilers and packages. DVM is a great tool, specially since DMD breaks something in basically every release. If you want to build multiple packages you're most likely do need multiple versions of DMD.

About multiple versions of the same package. I would say that this is a key feature of a package manager. Installing a package, including its dependencies and supporting multiple versions of the same package are the key features of a package manager. Say, application A depends on library C, version 1. Application B depends on library C, version 2. How would that be solved if multiple versions of the same package can't be installed.

• Install packages A package spec only contains one package. Although a package can contain both an executable and a library, if that what concerns you.

Orbs do contain metadata: https://github.com/jacob-carlborg/orbit/wiki/orb%20package

• The default action I don't really like tools like this with a default action, other than displaying usage information. I have had some bad experience with this before. Take Rails 2 for example, to generate a new Rails application you just run rails foo. Say then that I want to display the usage information, I then run rails hep, "help" misspelled. Instead of complaining about an unrecognized command it will happily generate an new Rails application called "hep" and you can only hope it will not overwrite any existing folder.
• Version specification I quite often specify the exact version to install when using Rubygems. I don't see what's wrong with a --version option. I there is no --version option then the user have to figure out how to specify the version, should it be "package-version", "package_version" or something else.
• Selecting versions I don't see a reason to have a separate tool for listing version, it will just be confusing. Yet another tool one has to know about.
• Commands All these commands will be available. I have to start somewhere :). I'm not sure if orbit -au derelict will work, it depends on if the library I use can handle it.
• Build, upload "build" will build an orb package, i.e. it will put all necessary files in an orb package. "upload" will upload a given package to a repository. All this information is available here: https://github.com/jacob-carlborg/orbit/wiki/orb

About binary packages. Yes Orbit will support binary packages. This is one of the top two complains people have had with Orbit. Again, if you actually read the wiki pages you will see this: https://github.com/jacob-carlborg/orbit/wiki/orb%20package

• "If you're only referring to the packaging ..." I think Orbit will mostly be used by developers and it will be developers that create packages of their own software. I quite certain we won't have the luxury of release managers.
• "Put the metadata in the orbspec" Currently the orbspec IS the metadata. When an orb is packaged the tool will just copy the orbspec into the orb package. I don't see how binary packages are not compatible with this. it will just be an orb package containing a pre-compiled library/executable and the metadata.
• "Don't let things into the blessed..." I don't see how I can have time to test and review every single package.
• "Using a programming language..." I have to say first, Ruby is NOT a runtime dependency, I'm linking statically with libruby. Actually most people don't like Ruby as the language for orbspecs but instead want D.
• Orbspec Spec I just basically sketched that up based on Rubygems. Not entirely sure what's actually needed. Most likely some fields will be removed and some will be added.
• "Though rather than using..." I'm planning to add something similar like this, but probably not in the initial version. But I'm not sure what you mean, if I don't use a list of files, how till I know that files to put in an orb package?
• "Keep track of the files that..." I'm keeping track of the installed files.

This was a very long post, thanks for all the comments.

[apriori]

@Wyatts :

Please consider taking a look at cabal for haskell to understand what goals of Orbit will be. Btw., as a former packager for multiple distros I can tell, the amount of work duplicated because proper package management is not an integrated part of the language and its toolchain is not even funny. Every distro packages its own stuff, and the wast majority want the very same results without actually being able to provide those same results. As a result many developers suffer from their applications getting bad reputation, which was not caused by them, but by packagers which have no clue about the application in question and how to build it properly. While its not really the goal of Orbit to change that situation, Orbit is meant to provide convenience to developers to escape the usual "buildsystem hell" we got everywhere. Cabal on haskell for example is extremely good and makes developing extremely convenient... its just a natural wish to have something similar for D. About fearing that libs, headers, etc. installed by Orbit would conflict with any versions installed by a systems package manager.. well, one would need to solve that with either 1) proper include orders or 2) sandboxing.

[jacob-carlborg]

I do not plan to install Orbit packages in standard locations. I plan to install them into its own directory. Then maybe create symlinks or solve it some other way.