search

dlang / dub

Package and build management system for D
MIT License
676 stars 227 forks source link

interaction with deimos / native dependencies feels clunky #852

Open wilzbach opened 8 years ago

wilzbach commented 8 years ago

I just report back my feedback as a user

I tried to install drepl which requires linenoise. There's a D-Wrapper via deimos for it, which however has no idea how to fetch linenoise My assumption was that D linenoise has linenoise has git submodule and adds the necessary buildsteps to it's preBuildCommands and adds the built library to its linked flags, s.t. all I need to type is dub.

Well sure if I get /usr/local/bin/ld: error: cannot find -llinenoise, I can install it, but I hope we all agree that's not the experience we aim too.

So what is your opinion is git submodules, preBuildCommands, linking flags the solution we should suggest to package authors?

MartinNowak commented 8 years ago

Deimos libraries are by definition header-only bindings for C libraries. Because binaries depend on your distribution, you'll have to install the necessary C libraries yourself.

wilzbach commented 8 years ago

Deimos libraries are by definition header-only bindings for C libraries. Because binaries depend on your distribution, you'll have to install the necessary C libraries yourself.

Yes I am aware of that, but my question was how can we make this process easier. As deimos shows, there's a whole bunch of them out there and dub run <my-package> should just work, otherwise it is frustrating for newcomers.

In this example linenoise is a tiny C library with zero dependencies that can be easily compiled on the fly. So is git submodules and preBuildCommands (calls e.g. cross-platform make) a good way to go?

mihails-strasuns commented 8 years ago

Support of 3d party C libraries can't be implemented without adding platform-specific modules to dub for each OS/distro out there. I am not aware of any language which currently does that because of how inherently challenging the problem is.

And no, building all those libraries manually is not a way to go. It will be unreliable (C libraries have their own build systems and there is no such thing as "cross-platform make"), slow (because, well, C libraries) and alien to host system (not linking to system-wide installation of libraries).

Would be good to print reminder from dub to install required deps if link dependencies are not found though.

wilzbach commented 8 years ago

I am not aware of any language which currently does that because of how inherently challenging the problem is.

Even NodeJs does this. They use node-gyp and you can just put your C/C++ code in the package and it will be compiled after the source is downloaded.

There are a couple of good articles out there, but this one is kinda famous "C++ and Node.js: An Unholy Combination….But Oh So Right"

The interface for package developer is really simple, mostly you just set an array of C/C++ source files that will be compiled then. As long as C/C++ is still the majority of libraries, that would be extremely helpful.

I am not aware of any language which currently does that because of how inherently challenging the problem is.

For Cargo they allow a user to specify a custom build script

mihails-strasuns commented 8 years ago

Even NodeJs does this. They use node-gyp and you can just put your C/C++ code in the package and it will be compiled after the source is downloaded.

For certain definitions of "does" - probably. In my book it is a good shining example of 100% broken attempt to do so. It requires to install additional platform-specific tools, still only works with specially adjusted C libraries despite that and always does source build even if library is already installed in the system. Last thing I want is that my D projects start silently doing bullshit like that in build background.

For Cargo they allow a user to specify a custom build script

That is also something different - a self-hosted build system as a library for defining mixed projects. Probably most similar to https://github.com/atilaneves/reggae in D world. Solving install/discovery of external libraries is still to be solved manually there AFAICS.

There are quite some tools (i.e. cmake) that do a decent job at finding and configuring already installed external dependencies but I am not aware of one that integrates with the system to prompt required installation. And that is crucial - otherwise you end in a situation when trying to compile vibe.d projects transitively causes compilation of whole libevent.

etcimon commented 8 years ago

This is one of the oldest issues and would require some readings before it gets brought up again short of repeating ourselves.

https://github.com/dlang/dub/issues/52

wilzbach commented 8 years ago

Solving install/discovery of external libraries is still to be solved manually there AFAICS.

Afaik you can have C code in your github repo for pure dependencies and they have special -sys packages for linking to system libraries. The latter part is similar to Deimos, but a lot smarter: "this will often probe the current system for libfoo before resorting to building from source."

Probably most similar to https://github.com/atilaneves/reggae in D world.

Pinging @atilaneves for ideas.

Btw I just saw that Gradle also allows C/Cpp builds.

atilaneves commented 8 years ago

If you want to compile C/C++ code in a dub project, I'd recommend having a git submodule with the dependencies and either:

  1. Use preBuildCommand with a shell script to build it. If it's a simple build this might be sufficient.
  2. Use reggae - you get the dub builds for free (modulo bugs) and can tell it to build the dependencies as well. You'll probably only need something like staticLibrary. This is the closest to the cargo solution.
wilzbach commented 8 years ago

Some updates from my site:

1) DEP5 tackles this problem 2) In R C/C++/Fortran are also built-in, see e.g. this tutorial

Use preBuildCommand with a shell script to build it. If it's a simple build this might be sufficient.

Yeah, but that won't work for Windows :/

Use reggae - you get the dub builds for free (modulo bugs) and can tell it to build the dependencies as well.

How would that work with getting reggae? So the preBuildCommand would be dub run reggae?

atilaneves commented 8 years ago

I didn't even know DEPS were a thing.

Yeah, but that won't work for Windows :/

preBuildCommand could run a D script instead of shell.

How would that work with getting reggae? So the preBuildCommand would be dub run reggae?

No, reggae understands dub builds and offers rules that build the targets defined in dub.json/dub.sdl. But... I haven't ported it to Windows yet. I was waiting on a dmd bug that only recently got fixed.

wilzbach commented 8 years ago

I started to read DEP5 more in depth and @s-ludwig if that's on the upcoming agenda - that would be really nice! And as you can see from the other opened issues (e.g. #873, #874, #841, #604, #444) - there is growing demand :)

In any case about DEP5, I would only have some minor comments:

jacob-carlborg commented 8 years ago

I can add that Rubygems (the Ruby package manager) also supports compiling C code. There's a few conventions, like where the C files are placed and which script to invoke to build the C code, except from that it's pretty open. Either a package contains all the C code, or it contains something like bindings, allowing to access the C code from Ruby. System libraries need to be installed manually. Usually fairly common libraries like libxml or libevent.

The only issues I've had with this is when Apple decides to change something (GCC -> Clang, location of SDK) and the gems have not been updated yet.

truedat101 commented 5 years ago

What is the status of this issue? There is is/was a great discussion here. Is DEP5 out of the gate?