search

Abscissa / DPak

A currently design-phase reboot of DUB's package management
0 stars 0 forks source link

Crazy idea: Just use 0install under-the-hood? #1

Open Abscissa opened 5 years ago

Abscissa commented 5 years ago

Throwing this out there, as it only just occurred to me: There's so many package managers in existence, why reinvent the wheel? Why not just take an existing cross-platform language-agnostic package manager (like 0install) and just make a wrapper for it that suits our needs?

Naturally, that begs the question "Why have a wrapper at all instead of just using the existing package manager as-is?" Well, largely because I don't think for a second it would go over well to head to the DLang Forums and say "Hey everyone, let's just use XYZ existing package manager as the new official D package manager!" That'd never work. Plus, no matter what we do, we NEED to create some way to leverage the existing DUB-based code.dlang.org ecosystem. And likely there will be ways we'd want to tailor the experience for a D programmer's needs.

Another possibility would be to just simply support dependencies on 0install-based packages for binary tools that don't originate from D-land (In much the say way that 0install allows its own package's dependencies to be satisfied by packages already installed by most native OS package managers.)

quickfur commented 5 years ago

I wonder if we can write an automatic translation tool that makes packages on code.dlang.org available on 0install. It has to be automatic, 'cos nobody will be able to maintain it otherwise. But once we have it, it would allow us to test how 0install behaves w.r.t. dub packages. If it does what we want, or mostly what we want, then we could either just wrap it, or adapt its algorithms.

Abscissa commented 5 years ago

Well, some sort of automatic translation/interpretation of DUB packages will be needed anyway, whether 0install's involved or not. So that definitely sounds like a good approach for testing the waters. I would think automatic DUB -> 0install should be entirely possible, maybe even easy, since AIUI 0install is already designed to be language-agnostic (much like an OS package manager) and we can safely assume that pretty much any existing DUB package can be built via the command dub build. So I like it, sounds like as good first step. We can try that, see how it goes.

Abscissa commented 5 years ago

AIUI, Publishing a 0install package basically amounts to just posting an XML config file via http. So maybe a good approach would be to set up a vibe.d server that takes a dub package name and optional version through the URL, either scrapes code.dlang.org or just queries what it needs through DUB directly, and then spits out 0install's xml.

quickfur commented 5 years ago

Sounds good, should be pretty easy to setup once we have the translation code working. I think on-demand translation with caching is probably the best approach. Dub being the way it is, I'm not holding my breath to wait for it to work through the entire code.dlang.org.

Abscissa commented 5 years ago

Wow, ok, initially I had some worries. But I've been working an a cli dub->0install package converter (which can then be utilized by a vide-d frontend), and I've been studying the docs for the [0install config file]() (they call it a "feed" since they're implemented as XML either locally or over HTTP). The tool's not ready yet, but the more I dig into this, the more convinced I am that this is an entirely feasible approach.

Far as I can tell so far, 0install basically supports all of our requirements, plus a bunch of other niceties like package temper/integrity checks, allows dependencies (for example, make) to optionally be satisfied by already-installed OS packages (like pacman or apt-get, for example), and of course a whole pre-existing package ecosystem. For the most part, we just need:

A. This dub compatibility layer, and B. A 0install front-end and config format to make the whole thing more approachable, palatable and catered to the D audience. (And jettisons all the buildsystem and other baggage from dub). Call this "DPak" or whatever.

Both parts are really just little more than just a matter of translating things into 0install's feed format (which appears to be entirely capable and flexible enough for the job). And then right there, with just those two translation systems, A and B, and that's already probably like 90% of everything we'd need for a v1.0.

Fingers crossed for no unpleasant surprises, but I'm actually getting quite excited about this...

taylorh140 commented 5 years ago

I also think it would be silly to reinvent the wheel with package management. (there are so many). But shouldn't we see what options are available? What criteria are necessary functions for the package manager? 0install may be the best solution but it would be nice to build a list of contenders.

quickfur commented 5 years ago

I think it has already been laid out in the forum and also in the wiki here, but just so we're on the same page:

Abscissa commented 5 years ago

I also think it would be silly to reinvent the wheel with package management. (there are so many). But shouldn't we see what options are available? What criteria are necessary functions for the package manager? 0install may be the best solution but it would be nice to build a list of contenders.

@taylorh140: Good point. All the other ones I'm aware of I did have reasons for rejecting. However, that info shouldn't reside just in my own head. I've now posted details to the wiki. The criteria @quickfur listed is good, too.

quickfur commented 5 years ago

@Abscissa I glanced over the wiki page you wrote up. Very nice, and very detailed! I think the list of requirements at the end will serve us well in deciding what to use / not to use as backend, and also if we should decide to write our own (hopefully not -- I'm uncomfortable with the idea of having to implement a SAT solver by hand).

But anyway, on a side note (perhaps this should be a dub issue?): in your comparison notes of dub vs. 0install, you noted that every dub package has its own description file, and so dub has to potentially scan through all versions of a package in order to obtain sufficient information to resolve its dependency graph. This is a huge bottleneck!! I'll bet this is (one of?) the reason(s) dub is so annoyingly slow. A quick fix that immediately comes to mind is to have the code.dlang.org server automatically consolidate version and dependency information into a single resource (file) per package, ala 0install. Then dub will no longer have to query every version individually; it just has to download the single consolidated file and it will be able to build its dependency graph. I wonder if dub will perform better after this optimization.

Abscissa commented 5 years ago

@Abscissa I glanced over the wiki page you wrote up. Very nice, and very detailed! I think the list of requirements at the end will serve us well in deciding what to use / not to use as backend, and also if we should decide to write our own (hopefully not -- I'm uncomfortable with the idea of having to implement a SAT solver by hand).

From what I've been learning this past week or two, IIUC, the package managers that use SAT solvers often use third-party SAT-solver libs. But still, I wouldn't want to have to deal with converting the dependency problem into a SAT and back, either. Or figure out how to use a SAT solver lib. Or go anywhere near SAT stuff...

A quick fix that immediately comes to mind is to have the code.dlang.org server automatically consolidate version and dependency information into a single resource (file) per package, ala 0install. Then dub will no longer have to query every version individually; it just has to download the single consolidated file and it will be able to build its dependency graph.

Actually, just moments ago, I realized it already does that. For example: https://code.dlang.org/api/packages/vibe-d/info Run that through a json pretty-printer (such as right nya) and check the following nodes:

I'll adjust the wiki to clarify.