search

dlang / dub

Package and build management system for D
MIT License
672 stars 230 forks source link

System package managers (by example of Portage) #342

Open mleise opened 10 years ago

mleise commented 10 years ago

Dub has successfully become the de facto library manager for D now. To be fully accepted it needs to support some more uses though. One of them is to work together with the various system package management systems. Languages like Java are pretty much self-contained and can have their own package eco-system independent of the system. D libraries and programs on the other hand are native code and expected to behave exactly like the C code they link to and will be linked by. Dependencies may go either way C->D or D->C.

If looked at objectively there can only be one installed package database for a single system. Handling D packages with dub alone will only isolate the D eco-system from the rest of the system and hinder its wide spread acceptance on the same level as C/C++. Now the dozen or more Linux package managers are often very different, although they commonly can tell you what files you have installed from what package, which packages can be uninstalled because they are no longer needed as a dependency etc. Since I am using Gentoo and it has a very sophisticated package manager (Portage), I'll try to explain how I imagine it can work with dub...

First of all for Portage (and two other package managers based on the same specs) to be able to calculate system-wide dependencies it needs an ebuild (a package installation script, written in bash). So for every D package that is needed by some application someone would create such a script. This is no different from C. These scripts define what the dependencies are (as described in the previous issue about adding system dependencies to dub), where to fetch the source code from (since on Gentoo every package is compiled from source), what configuration option the package offers (typically short words like 'python' that map to ./configure options like --enable-python), and lists the commands to compile and install the package files. On top of that there are several environment variables that affect the "root" directory for installation or the exact C and C++ compiler flags to use. In general it is very much tailored towards the typical ./configure && make && sudo make install sequence, which is in fact run by default if the script doesn't override it.

As for the dependencies, it is already described in the other issue, but I'll repeat it here. Package names may differ between distributions and often several implementations of a package exist (ffmpeg/libav, OpenGL, ...). What I would like dub to establish is an own list of system package names (at least in theory, in practice only where ambiguities may ensue) shared by all dub projects. So in case of ffmpeg and libav, "ffmpeg" could be the package name dub uses. I would then create a translation table dub<->Gentoo that translates ffmpeg to virtual/ffmpeg, a dummy package that depends on either media-video/ffmpeg or media-video/libav. The same applies to OpenGL which is usually provided by the graphics card vendor. Version support in Portage is either '>', '>=', '<', '<=', '=' with a reference version or partial version and a wildcard, so you can specify version 2.4* of a library for example. The wildcard version may look superfluous, but often you want to depend on a certain API version (major.minor) and don't care about any bug fix revisions. So since ebuilds are scripts I can run arbitrary code to generate the list of dependencies including calling dub. Some command that outputs bash parsable text that describes the dependencies (name and version) would be ideal. JSON could become more complicated: http://stackoverflow.com/questions/1955505/parsing-json-with-sed-and-awk :smiley:

The source code is a different beast. Extensions exist for the ebuild system to check out source repositories. This will be handy for ~master versions. So what dub would need to tell me here is the repo URI and the version. I'll try to figure out from there if I can download a .zip for specific version, or need to fetch ~master. Either way dub would not just be passing information to the ebuild, not download the file(s) itself, as there are specific directories for them on a Gentoo system. Afterwards the source files are unpacked/copied to a temporary directory where they may be patched, configured and compiled.

So called USE-flags determine what features a package is built with. This ranges from compatibility hacks to optional linkage with external libraries. As mentioned they often map to ./configure options. The non-dub ebuild I wrote for GtkD has USE flags that map to GtkD's optional parts like bindings to Gtk's OpenGL or source code highlighter widgets. I consider this a nice to have. If dub has something like sub- or extra-packages inside a master package or exposes known version= identifiers, those would be candidates for use flags. In the case of GtkD, these optional features also pull in further system dependencies. It is unclear to me, what you plan to do about this or if GtkD would be split up under dub to have statically known dependencies for each part. In any case for many C libraries it is common to e.g. have different options for the SSL library they can be build for. If this trend continues in D, it would probably be a version= flag and good to see exposed by dub. All I would need is a simple list of distinct C identifiers like "openssl gnutls". But let's put that on the TODO list for some time.

The actual compilation is heavily influenced by environment variables for C/C++ libraries. Commonly used in C/C++ projects are DESTDIR and PREFIX for the installation path and compiler flags that already found some relatives that suggested themselves in a couple of D projects: CC - the C compiler to use (e.g. icc, gcc) -> DC (e.g. gdc, dmd, ldc2) CFLAGS - C compiler flags override -> DFLAGS (usually either dmd flags or rdmd flags if multiple compiler vendors are supported) LDFLAGS - additional linker flags as would be passed to gcc. Needs some adaption for the various D compilers, but it is manageable. It is expected from an ebuild to honor these flags, but it is clearly not the end of the world if the DFLAGS have no effect. Since dub approaches this entirely differently through premade configurations I see little hope of bringing the two worlds together. On the other hand I could introduce a DUBFLAGS variable to control dub behavoir when installing packages through Portage. The question here is, could I even use dub to a compile code inside a specific directory ignoring ~/.dub, which is possibly not even accessible under the special user account that the installation runs as?

The last important step would be installation. Usually "make install" takes care of that. Otherwise there are special commands in ebuilds to install documentation, libraries and executables (with the correct permission bits set) or configuration files. System wide installation directories can differ from system to system. A special case is lib/lib32/lib64 on systems that can run both 64-bit and 32-bit code. The compilation process needs to accept DFLAGS=-m32/64 so I can compile a library for one or both pointer-widths and either I have to tell dub, what the lib directory is for this compile. Alternatively dub could install to a fixed location like $DESTDIR/$PREFIX/lib and I would rename that to the actual directory. You might want to think about this in general as soon as desktop applications hit dub, which install configuration files, HTML documentation, desktop launchers, icons and more. If they are good applications, package maintainers for Linux distributions will want to well ... package them and files need to find their places, some of which must be customized for a particular distribution. Maybe we need to discuss this in a larger round of all current Linux package maintainers.

That's all for now, thanks for reading. I hope it is more clear now why I was hesitant to jump on the dub band wagon. I understand that you want to cover 99% of use cases eventually, which is great. For that you need to tackle system-wide installations somehow.

mihails-strasuns commented 10 years ago

Arch Linux prohibits pure source library packages in any official repositories. And I believe it is right thing to do. Job of dub is to help developer. Job on system package manager is to help system user. I see no point in merging these often conflicting goals.

bioinfornatics commented 10 years ago

They are many different problems:

lib should to go to:

conf file:

data file

Once tool which dub should to looks is distutil in python.

Pypi is like dub registry but their tools is more powerfull

$ python setup.py install --home
$ python setup.py install --prefix=/usr
$ python setup.py install --prefix=/usr/local
$ python setup.py install --prefix=/usr/ --libdir=/usr/lib64

I hope to see it one day in dub will be able to package tons of D tools. At te moment i can share only tool which developers provides another build tools.

So at this time all tool which use dub is only for D community and not to share to most people.

mihails-strasuns commented 10 years ago

I hope to see it one day in dub will be able to package tons of D tools. At te moment i can share only tool which developers provides another build tools.

I hope to never see it. dub should stay build and development tool. Not packagement tool. Not installation tool. Not distribution tool.

bioinfornatics commented 10 years ago

By example take derelict 3 which provide only dub as build system. How is possible to build it ? without a build system that is kinda hard and that is a small project.

Sorry @Dicebot i do not understand you. What is the problem with the python system? It works fine, have years as experience and really easy to use for all purpose

p0nce commented 10 years ago

This all stems from an ongoing IRC discussion with people experienced in distro package maintenance. Since I don't know the subject I'lll try to summarize:

bioinfornatics commented 10 years ago

@ponce something which generate a makefile is more a cmake like which is seem dub is not. But if dub devlopers want go to this way they can use MakefileForD as template file

I would like a way to set prefix libdir and default compiler

p0nce commented 10 years ago

@bioinfornatics I see DUB as similar to cmake in that it knows how to generate IDE builds. Like you said somewhere there is two parts in it, a build tool and a dependency resolver/installer.

s-ludwig commented 10 years ago

Regarding make files, see also #275. It should be quite easy to do by taking one of the existing generators and modifying it accordingly, so if someone wants to give it a shot...

mihails-strasuns commented 10 years ago

Sorry @Dicebot i do not understand you. What is the problem with the python system? It works fine, have years as experience and really easy to use for all purpose

Python is scripting language. It does not have any distinction between development dependencies and runtime dependencies. For native compiled language distinction is very practical - I don't want my users to ever care what tools and libraries I have used to develop my library / application.

Also it does not work good for Python either because at some point you are either forced to copy all dependency packages used from pip (resulting in huge amount of tiny package only tiny amount of people cares about) or bundle all stuff together resulting in dependency duplication. But it is common problem with scripting languages. D can do as good as C here.

markos commented 10 years ago

I cannot speak for other distros, but @mleise and @p0nce gave a good summary. For Debian, as I think I've told too many times, and am in danger of becoming repetitive and boring :), there are two things that are absolutely a no-go if I would ever want to package a D application/library using dub.

  1. There can be no network access from the build system during the build phase -this is for the autobuilders, some of the actual autobuilders are connected but it cannot be taken for granted.
  2. The build system is absolutely not allowed to tamper the existing/installed packages, it can do whatever it wants with the build directory - as a non privileged user, but nothing more. It must also be able install to a given directory -like DESTDIR as @mleise mentioned.
  3. A not-vital-but-nice-to-have feature would be to have dub list the dependencies via some simple command, both includes and libraries. That way it would be easy for the packaging person to deduct what has to be entered in the package description afterwards.

So, the way I see it from the Debian side, it's definitely not my wish to replace dub, no more than I can replace git, it serves a different purpose than a package manager. If I was developing a GUI library or a game, I would definitely use it, but if I want to package the library/game for a distro, I just can't do it. The arguments about other build systems, while not irrelevant, they really change nothing, even perl/python/ruby/etc packages share the same restrictions, it doesn't matter if it's a binary or an interpreted package, it has to follow policy otherwise it won't be allowed in the archive -nothing to do with me, it will just never get passed the ftpmasters/release team.

To conclude, @s-ludwig has already created a ticket for makefile generation. That and a dependency listing -maybe I'll go ahead and create the ticket myself (edit: there actually is one already #34) - would make it much easier for me and I believe others, to package D stuff for a distro, without getting in the way of dub, and vice versa.

bioinfornatics commented 10 years ago

@markos totally agree with you on fedora we have same problem if dub generate a makefile with var:

it will be a good start

I am happy to see some others packagers which said same thing about dub.

They are one year i had choose to stop D because no one care about my issue. I come back because too many fedora user want D and some libs. This job take to me many free time and i do this because D is a good language.

ximion commented 8 years ago

Hey :) I just found this report... We are hitting similar issues (reported https://github.com/dlang/dub/issues/838 ), and basically using dub for building stuff in Linux distributions is a pain at time.