Question: How to compile statically with all drivers included in libgphoto2.a/libgphoto_port2.a

MIvanchev commented 2 years ago

I'm working on something which requires a static build of libgphoto2 but it creates separate .a files for all the drivers. How can I include everything in the 2 main static libraries?

ndim commented 2 years ago

TLDR

At this time, we only support dynamic loading of drivers.

Reasoning

I have actually wanted to try building a single large library containing all the drivers for a long time, mainly for curiosity's sake. Back then, it did not work for some reason, and it has not been much of a priority: Dynamic loading of just the relevant driver makes a lot of sense, simplifies the tasks make needs to do during development, and dynamic driver loading has been working well for all supported operating systems for the last about 20 years.

The driver loading having been developed on and for ELF/Linux in the early 2000s, nobody ever ran into problems with circular linking dependencies, but those circular dependencies were IIRC a large part of the reason why 15 years ago, we could not just cross compile for Windows. No idea what changed here, but these days, (cross) compiling for and running on Windows appears to work.

It could be argued that with either modern RAM sizes or with modern OS's mmap()ing libraries and actually loading only the pages actually used, we should support statically linking all drivers into the libraries at least optionally and at least on the operating systems which support that, or (on the other extreme) maybe even switch to it.

Another use case for statically-everything I would consider valid is an embedded system where you want to avoid needing filesystem code for loading drivers. Just lacking a proper software distribution model and instead every application shipping statically linked with all its dependent libraries (Windows) is not a good reason to invest time in static-everything IMHO.

Why do you want static-everything? I would like to hear the argument.

Affected Areas

Anyway, here are some aspects to consider when adding support for statically adding all drivers to the ibrary files.

Statically linking everything into one library would need adaptations in (at least!) two places:

how we load drivers at runtime (code using )
how we link drivers at build time (libtool, dlopen, dlpreopen, etc.)

We are using those mechanisms in two places, mostly via code duplication in both C and buildsystem:

camera drivers called camlibs loaded by libgphoto2
port drivers called iolibs loaded by libgphoto2_port

Then we want driver loading to work on relatively different operating systems like

Linux,
BSD
MacOS
Windows (and DLLs are weird)

This is a non-trivial bit of work.

In case it helps you: It might be possible to relatively easily adapt the build system to allow static linking of an executable against libgphoto2_port and libgphoto2, while those statically linked libraries in turn dynamically load the respective drivers (iolibs and camlibs).

Plans Before You Brought This Up

Given that nobody uses libgphoto2_port but libgphoto2 and software which uses libgphoto2, I have been working on integrating the two relatively similar buildsystems for libgphoto2_port and libgphoto2 into one buildsystem (https://github.com/gphoto/libgphoto2/issues/708 https://github.com/gphoto/libgphoto2/pull/731). That is a first step towards moving the code from libgphoto2_port into the libgphoto2 library and either abolishing libgphoto2_port.{so,a,dll,dylib} or replace it with an empty dummy for compatibility, and then updating the buildsystem to support non-recursive make which would be a prerequisite for reasonably building a single library file which contains all drivers.

In order to avoid doing the same thing twice for both libraries, I would have started on the driver loading only after all the buildsystem and probably library code integration has been completed.

MIvanchev commented 2 years ago

Hey, thanks for taking your time to give me such a detailed response, I really appreciate this.

Why do you want static-everything? I would like to hear the argument.

It's very convenient for the user when they are obtaining a large software product depending on hundreds of stuff. Not all distros have package managers and even if they do stuff are often quite outdated or even unavailable. This drives the users mad because they often spend days trying to figure out what's missing. Worst case they have to install a compiler and build the software themselves. I am aware of the short-comings of static linking and my scenario is something slightly else which I'll reveal soon with my current project but believe me, the need for static linkage is quite real. Also so far gphoto2 is one of 2 libraries out of 70 which doesn't support it. It's not criticism btw, gphoto2 is an awesome software I use daily and this is just a minor thing I happen to need now :)

Now, I understand your position that it's risky to change the architecture given the number of users so I think I'll fork and adapt it to my needs because I do need it urgently and then we can see about merging back gradually if you're interested. From what I saw it comes down to:

make camera_init, camera_abilities, camera_id static and export pointers through a global structure _drv_funcs
building an array at compile time from all the _drv_funcs and using this instead of the list functions
something similar for the iolibs

Let's stay in touch regarding this.

hfiguiere commented 2 years ago

I hope you intend to also respect the LGPL license? Linking things statically doesn't help allowing user to use the software with a modified version.

MIvanchev commented 2 years ago

I didn't quite get it, but everything is open source and made clear to the user and nothing is hidden away because I love open source and the users do too, it's just a convenience measure :)

hfiguiere commented 2 years ago

It wasn't clear from the original report since you never mentionned your use case. So I just wanted to make sure.

MIvanchev commented 2 years ago

Yes, I didn't phrase it properly, sorry about that :)

MIvanchev commented 2 years ago

Hey, in case someone is interested, the project I needed a static gphoto for is https://github.com/MIvanchev/static-wine32.

RReverser commented 2 years ago

FWIW I did static linking (albeit, only of USB and PTP2, excluding all other drivers) for WebAssembly in Porting USB applications to the web. Part 2: gPhoto2, because WebAssembly doesn't have real dynamic linking yet.

To do that, I utilized Libtool's own preloading mechanism (which is a fancy name for statically linking + declaring libraries so that libtool can find it).

The changes necessary are relatively simple but somewhat repetitive and would need to be added to every driver. See the section Dealing with dynamic loading of the article for more details.

MIvanchev commented 2 years ago

Thank you so much for this, looks very useful :) Nice hacking!

MIvanchev commented 1 year ago

Hey, just wanted to report I now got libgphoto2 to compile statically. The (hacky) patch is here https://github.com/gphoto/libgphoto2/compare/master...MIvanchev:libgphoto2:v2.5.31-static-build?expand=1. It was surprisingly easy though I decided against using libtool. I mainly had to do some prefixing because of symbol collisions. It might be a good idea to port them over to libgphoto2 for the sake of cleaner code.

gphoto / libgphoto2