search

nh2 / static-haskell-nix

easily build most Haskell programs into fully static Linux executables
388 stars 36 forks source link

cannot build Haskell programs that depend on gtk+3 statically #50

Open cdepillabout opened 5 years ago

cdepillabout commented 5 years ago

I'm trying to build a Haskell executable (Termonad) statically that uses gi-gtk, which is built using haskell-gi and depends on gtk+3.

It appears to be failing when building systemd.

Termonad depdends on gi-gtk, which depends on gtk+3, which depends on at-spi2-atk, which depends on dbus, which depends on systemd.

I can include the output from the failed systemd build if it would help.

Alternatively, here is the derivation I am trying to build:

let
  survey = import ./survey { normalPkgs = import ./nixpkgs {}; };

  termonad-pkg =
    { mkDerivation, adjunctions, base, Cabal, cabal-doctest
    , classy-prelude, colour, constraints, containers, data-default
    , directory, distributive, doctest, dyre, filepath, focuslist
    , genvalidity-containers, genvalidity-hspec, gi-gdk, gi-gio
    , gi-glib, gi-gtk, gi-pango, gi-vte, gtk3, haskell-gi-base
    , hedgehog, inline-c, lens, libpcre2, mono-traversable
    , pretty-simple, QuickCheck, singletons, stdenv, tasty
    , tasty-hedgehog, tasty-hspec, template-haskell, text, vte_291
    , xml-conduit, xml-html-qq
    }:
    mkDerivation {
      pname = "termonad";
      version = "2.0.0.0";
      sha256 = "0rprqn5vcvhbqqg0grrmz0ijjpkrprza88la4mbdg6skb34fjsp0";
      isLibrary = true;
      isExecutable = true;
      enableSeparateDataOutput = true;
      setupHaskellDepends = [ base Cabal cabal-doctest ];
      libraryHaskellDepends = [
        adjunctions base classy-prelude colour constraints containers
        data-default directory distributive dyre filepath focuslist gi-gdk
        gi-gio gi-glib gi-gtk gi-pango gi-vte haskell-gi-base inline-c lens
        mono-traversable pretty-simple QuickCheck singletons text
        xml-conduit xml-html-qq
      ];
      libraryPkgconfigDepends = [ gtk3 libpcre2 vte_291 ];
      executableHaskellDepends = [ base ];
      testHaskellDepends = [
        base doctest genvalidity-containers genvalidity-hspec hedgehog lens
        QuickCheck tasty tasty-hedgehog tasty-hspec template-haskell
      ];
      homepage = "https://github.com/cdepillabout/termonad";
      description = "Terminal emulator configurable in Haskell";
      license = stdenv.lib.licenses.bsd3;
    };

  myHaskellPackages =
    survey.haskellPackages.override {
      overrides = self: super: {
        termonad = self.callPackage termonad-pkg {
          libpcre2 = survey.pkgs.pcre2;
          vte_291 = survey.pkgs.gnome3.vte;
        };

        gi-gtk = survey.pkgs.haskell.lib.unmarkBroken super.gi-gtk;

        gi-vte = survey.pkgs.haskell.lib.unmarkBroken super.gi-vte;
      };
    };
in

myHaskellPackages.termonad

Note the following:

It is possible this should just be an issue on the nixpkgs repo. If this is the case, I will move it there.

cdepillabout commented 5 years ago

I'm not super familiar with cross-compiling support in nixpkgs, and I don't really know the relationship between static-haskell-nix and the pkgsStatic overlay, but I also tried to build systemd with pkgsStatic and pkgsMusl (where ./nixpkgs/ below is the nixpkgs submodule in this repo, currently at commit b577340eb5b):

$ nix-build ./nixpkgs -A pkgsMusl.systemd
...
$ nix-build ./nixpkgs -A pkgsStatic.systemd
...

Neither of these worked. All the dependencies for pkgsMusl.systemd were able to be built, but systemd itself failed. A bunch of the dependencies for pkgsStatic.systemd were not able to be built.

cdepillabout commented 5 years ago

Oh, hmm, I found https://github.com/NixOS/nixpkgs/issues/61580, which makes it sound like systemd can't be built with musl.

cdepillabout commented 5 years ago

I made some progress with this. I was able to get everything building except for one system dependency, VTE:

$ nix-build build-termonad.nix
these derivations will be built:
  /nix/store/j4kwsxxzcx34c99zqxi7b9aaw01cm31d-vte-0.56.3.drv
  /nix/store/rjjaia4f6fajgd0szail634gdfbf8cwh-gi-gtk-3.0.27.drv
  /nix/store/rq0922i46774q7mx7d3zfxfyn7mzziqn-gi-vte-2.91.19.drv
  /nix/store/x1wmi30vyzjg7kq5fsj842nrx1i5qqm4-termonad-2.0.0.0.drv
building '/nix/store/j4kwsxxzcx34c99zqxi7b9aaw01cm31d-vte-0.56.3.drv'...
...
vte.cc:3587:53: warning: unused variable ‘wp_str’ [-Wunused-variable]
                                         char const* wp_str = g_unichar_isprint(c) ? c_buf : _vte_debug_sequence_to_string(c_buf, -1);
                                                     ^~~~~~
vte.cc: In member function ‘void vte::terminal::Terminal::expand_rectangle(cairo_rectangle_int_t&) const’:
vte.cc:8994:31: warning: variable ‘old_rect’ set but not used [-Wunused-but-set-variable]
         cairo_rectangle_int_t old_rect = rect;
                               ^~~~~~~~
  CXX      libvte_2_91_la-vtedraw.lo
vte.cc: In member function ‘void vte::terminal::Terminal::insert_char(gunichar, bool, bool)’:
vte.cc:2908:40: warning: missed loop optimization, the loop counter may overflow [-Wunsafe-loop-optimizations]
   while (cell && cell->attr.fragment() && col > 0)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
vte.cc: In member function ‘GString* vte::terminal::Terminal::get_text(vte::grid::row_t, vte::grid::column_t, vte::grid::row_t, vte::grid::column_t, bool, bool, GArray*)’:
vte.cc:6171:82: warning: missed loop optimization, the loop counter may overflow [-Wunsafe-loop-optimizations]
                                 while ((pcell = _vte_row_data_get (row_data, col))) {
                                                                                  ^
  CXX      libvte_2_91_la-vtegtk.lo
  CXX      libvte_2_91_la-vteregex.lo
vteregex.cc:109:32: warning: unknown option after ‘#pragma GCC diagnostic’ kind [-Wpragmas]
 #pragma GCC diagnostic ignored "-Wcast-function-type"
                                ^~~~~~~~~~~~~~~~~~~~~~
  CXX      libvte_2_91_la-vterowdata.lo
  CXX      libvte_2_91_la-vtespawn.lo
  CXX      libvte_2_91_la-vteseq.lo
  CXX      libvte_2_91_la-vtestream.lo
  CXX      libvte_2_91_la-vtetypes.lo
  CXX      libvte_2_91_la-vteunistr.lo
  CXX      libvte_2_91_la-vteutils.lo
  CXX      libvte_2_91_la-widget.lo
widget.cc: In member function ‘void vte::platform::Widget::dispose()’:
widget.cc:134:30: error: ‘W_EXITCODE’ was not declared in this scope
                 int status = W_EXITCODE(0, SIGKILL);
                              ^~~~~~~~~~
widget.cc:134:30: note: suggested alternative: ‘WEXITED’
                 int status = W_EXITCODE(0, SIGKILL);
                              ^~~~~~~~~~
                              WEXITED
make[4]: *** [Makefile:1810: libvte_2_91_la-widget.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: Leaving directory '/build/vte-0.56.3/src'
make[3]: *** [Makefile:2133: all-recursive] Error 1
make[3]: Leaving directory '/build/vte-0.56.3/src'
make[2]: *** [Makefile:1279: all] Error 2
make[2]: Leaving directory '/build/vte-0.56.3/src'
make[1]: *** [Makefile:578: all-recursive] Error 1
make[1]: Leaving directory '/build/vte-0.56.3'
make: *** [Makefile:485: all] Error 2
builder for '/nix/store/j4kwsxxzcx34c99zqxi7b9aaw01cm31d-vte-0.56.3.drv' failed with exit code 2

Here's the nix file I am using. It is basically the same as the above, with some additional overrides:

let
  gtk3NoCupsOverlay = self: super: {
    gtk3 = super.gtk3.override {
      cups = null;
      cupsSupport = false;
    };
  };

  dbusNoSystemdOverlay = self: super: {
    dbus = super.dbus.override {
      systemd = null;
    };
  };

  survey = import ./survey {
    normalPkgs = import ./nixpkgs {
      overlays = [
        dbusNoSystemdOverlay
        gtk3NoCupsOverlay
      ];
    };
  };

  termonad-pkg =
    { mkDerivation, adjunctions, base, Cabal, cabal-doctest
    , classy-prelude, colour, constraints, containers, data-default
    , directory, distributive, doctest, dyre, filepath, focuslist
    , genvalidity-containers, genvalidity-hspec, gi-gdk, gi-gio
    , gi-glib, gi-gtk, gi-pango, gi-vte, gtk3, haskell-gi-base
    , hedgehog, inline-c, lens, libpcre2, mono-traversable
    , pretty-simple, QuickCheck, singletons, stdenv, tasty
    , tasty-hedgehog, tasty-hspec, template-haskell, text, vte_291
    , xml-conduit, xml-html-qq
    }:
    mkDerivation {
      pname = "termonad";
      version = "2.0.0.0";
      sha256 = "0rprqn5vcvhbqqg0grrmz0ijjpkrprza88la4mbdg6skb34fjsp0";
      isLibrary = true;
      isExecutable = true;
      enableSeparateDataOutput = true;
      setupHaskellDepends = [ base Cabal cabal-doctest ];
      libraryHaskellDepends = [
        adjunctions base classy-prelude colour constraints containers
        data-default directory distributive dyre filepath focuslist gi-gdk
        gi-gio gi-glib gi-gtk gi-pango gi-vte haskell-gi-base inline-c lens
        mono-traversable pretty-simple QuickCheck singletons text
        xml-conduit xml-html-qq
      ];
      libraryPkgconfigDepends = [ gtk3 libpcre2 vte_291 ];
      executableHaskellDepends = [ base ];
      testHaskellDepends = [
        base doctest genvalidity-containers genvalidity-hspec hedgehog lens
        QuickCheck tasty tasty-hedgehog tasty-hspec template-haskell
      ];
      homepage = "https://github.com/cdepillabout/termonad";
      description = "Terminal emulator configurable in Haskell";
      license = stdenv.lib.licenses.bsd3;
    };

  myHaskellPackages =
    survey.haskellPackages.override {
      overrides = self: super: {
        termonad = self.callPackage termonad-pkg {
          libpcre2 = survey.pkgs.pcre2;
          vte_291 = survey.pkgs.gnome3.vte;
        };

        gi-gtk = survey.pkgs.haskell.lib.unmarkBroken super.gi-gtk;

        gi-vte = survey.pkgs.haskell.lib.unmarkBroken super.gi-vte;

        gtk3 = survey.pkgs.gtk3;
      };
    };
in

myHaskellPackages.termonad

This is pretty exciting! I'm surprised it was this easy to get gtk3 compiled statically!

@nh2 I'd be interested if you had any suggestions on how to get VTE compiled statically as well.

cdepillabout commented 5 years ago

Oh, looks like I was able to get VTE building as well with a small patch:

let
  gtk3NoCupsOverlay = self: super: {
    gtk3 = super.gtk3.override {
      cups = null;
      cupsSupport = false;
    };
  };

  dbusNoSystemdOverlay = self: super: {
    dbus = super.dbus.override {
      systemd = null;
    };
  };

  vteMuslPatchOverlay = self: super: {
    vteForMusl = super.gnome3.vte.overrideAttrs (oldAttrs: {
      patches = (oldAttrs.patches or []) ++ [
        (self.fetchpatch {
            name = "0001-Add-W_EXITCODE-macro-for-non-glibc-systems.patch";
            url = "https://gitlab.gnome.org/GNOME/vte/uploads/c334f767f5d605e0f30ecaa2a0e4d226/0001-Add-W_EXITCODE-macro-for-non-glibc-systems.patch";
            sha256 = "1ii9db9i5l3fy2alxz7bjfsgjs3lappnlx339dvxbi2141zknf5r";
        })
      ];
    });
  };

  survey = import ./survey {
    normalPkgs = import ./nixpkgs {
      overlays = [
        dbusNoSystemdOverlay
        gtk3NoCupsOverlay
        vteMuslPatchOverlay
      ];
    };
  };

  termonad-pkg =
    { mkDerivation, adjunctions, base, Cabal, cabal-doctest
    , classy-prelude, colour, constraints, containers, data-default
    , directory, distributive, doctest, dyre, filepath, focuslist
    , genvalidity-containers, genvalidity-hspec, gi-gdk, gi-gio
    , gi-glib, gi-gtk, gi-pango, gi-vte, gtk3, haskell-gi-base
    , hedgehog, inline-c, lens, libpcre2, mono-traversable
    , pretty-simple, QuickCheck, singletons, stdenv, tasty
    , tasty-hedgehog, tasty-hspec, template-haskell, text, vte_291
    , xml-conduit, xml-html-qq
    }:
    mkDerivation {
      pname = "termonad";
      version = "2.0.0.0";
      sha256 = "0rprqn5vcvhbqqg0grrmz0ijjpkrprza88la4mbdg6skb34fjsp0";
      isLibrary = true;
      isExecutable = true;
      enableSeparateDataOutput = true;
      setupHaskellDepends = [ base Cabal cabal-doctest ];
      libraryHaskellDepends = [
        adjunctions base classy-prelude colour constraints containers
        data-default directory distributive dyre filepath focuslist gi-gdk
        gi-gio gi-glib gi-gtk gi-pango gi-vte haskell-gi-base inline-c lens
        mono-traversable pretty-simple QuickCheck singletons text
        xml-conduit xml-html-qq
      ];
      libraryPkgconfigDepends = [ gtk3 libpcre2 vte_291 ];
      executableHaskellDepends = [ base ];
      testHaskellDepends = [
        base doctest genvalidity-containers genvalidity-hspec hedgehog lens
        QuickCheck tasty tasty-hedgehog tasty-hspec template-haskell
      ];
      homepage = "https://github.com/cdepillabout/termonad";
      description = "Terminal emulator configurable in Haskell";
      license = stdenv.lib.licenses.bsd3;
    };

  myHaskellPackages =
    survey.haskellPackages.override {
      overrides = self: super: {
        termonad = self.callPackage termonad-pkg {
          libpcre2 = survey.pkgs.pcre2;
          vte_291 = survey.pkgs.vteForMusl;
          gtk3 = survey.pkgs.gtk3;
        };

        gi-gtk = survey.pkgs.haskell.lib.unmarkBroken super.gi-gtk;

        gi-vte =
          (survey.pkgs.haskell.lib.unmarkBroken super.gi-vte).override {
            vte_291 = survey.pkgs.vteForMusl;
          };
      };
    };
in

myHaskellPackages.termonad

Running it:

$ nix-build build-termonad.nix
these derivations will be built:
  /nix/store/rjjaia4f6fajgd0szail634gdfbf8cwh-gi-gtk-3.0.27.drv
  /nix/store/njrrx5fxn17q37fmxjv5x6fqvfc8kpyc-gi-vte-2.91.19.drv
  /nix/store/a310ambi2z9xcr2d7lc4lml823jd9h7i-termonad-2.0.0.0.drv
building '/nix/store/rjjaia4f6fajgd0szail634gdfbf8cwh-gi-gtk-3.0.27.drv'...

gi-gtk and gi-vte aren't able to be built, but I am pretty sure this is also a problem when building dynamically:

$ cd nixpkgs/ # this is the nixpkgs submodule in this repo, currently at commit b577340eb5b
$ nix-build -A haskellPackages.gi-gtk
error: Package ‘gi-gtk-3.0.27’ in /home/illabout/git/static-haskell-nix/nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix:93780 is marked as broken, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)

However it looks like gi-gtk and gi-vte are no longer marked as broken in current nixpkgs master, so once the nixpkgs submodule in this repo is updated to the current version of nixpkgs, I imagine gi-gtk and gi-vte will be able to be built statically as well! I am hopeful that termonad will then be easy to build statically!

@nh2, I guess this issue can be closed, since I figured out most of the questions I had.

nh2 commented 4 years ago

Thanks for the detailed writeup!

I also want to try building static GUI applications soon, so I'm very grateful for this.

However it looks like gi-gtk and gi-vte are no longer marked as broken in current nixpkgs master, so once the nixpkgs submodule in this repo is updated to the current version of nixpkgs, I imagine gi-gtk and gi-vte will be able to be built statically as well!

That should be after #61 is done.

nh2 commented 4 years ago

looks like I was able to get VTE building as well with a small patch:

@cdepillabout Would you mind PRing this into nixpkgs, directly into the vte package?

You can check for musl using the bool stdenv.hostPlatform.isMusl.

cdepillabout commented 4 years ago

@nh2 I've sent 4 PRs to nixpkgs that are needed to get vte (and gtk3) to compile with pkgsMusl. Would you be able to do a review of these?

I wasn't sure exactly what was the best way to do the check for pkgsMusl.

nh2 commented 4 years ago

Awesome, very appreciated!

nh2 commented 4 years ago

@cdepillabout There is another obstacle we have to take on:

When moving from autotools to Meson, gtk3 lost the ability to build static libraries:

https://gitlab.gnome.org/GNOME/gtk/issues/2248

CC @flokli

nh2 commented 4 years ago

When moving from autotools to Meson, gtk3 lost the ability to build static libraries:

My PR to fix it for gtk4 got merged: https://gitlab.gnome.org/GNOME/gtk/merge_requests/1172#note_648846

Now we should probably backport it to gtk3 upstream.

nh2 commented 4 years ago

Now we should probably backport it to gtk3 upstream.

I've done it in https://gitlab.gnome.org/GNOME/gtk/merge_requests/1196.

nh2 commented 4 years ago

I've done it in https://gitlab.gnome.org/GNOME/gtk/merge_requests/1196.

Merged.

nh2 commented 4 years ago

Another upstream PR: https://gitlab.gnome.org/GNOME/at-spi2-atk/merge_requests/19

Also very relevant: meson 0.52.0

https://github.com/NixOS/nixpkgs/pull/70650#issuecomment-558421025

nh2 commented 4 years ago

Behold tens of hours of effort:

image

(Edit: For a fancy button example scroll down.)

ldd /nix/store/bv21vj7xa6az6fhkg385523rv96xdmnh-meson-tutorial-gtk-0.0.1/bin/demo-gtk
    not a dynamic executable

This is a static GTK app written in C built with meson.

Some first sizes evaluation (plain strip, gzip -9, xz -6):

 21M demo-gtk-unstripped
 18M demo-gtk-stripped
8.2M demo-gtk-unstripped.gz
7.3M demo-gtk-stripped.gz
6.2M demo-gtk-unstripped.xz
5.6M demo-gtk-stripped.xz

nm -g shows that all gtk functions seem to be present, so this doesn't seem to do any unused-code elimination yet.

cdepillabout commented 4 years ago

@nh2 Great work!

I'm really impressed how much work went in to this (including upstreaming all the required fixes)!

nh2 commented 3 years ago

This is a static GTK app written in C built with meson.

For some more reproducibility, I just checked that this still works, and pushed the following branches:

I also made the demo app way cooler, it now has a button that actually does something to show that it can do more than rendering empty windows:

image

Binaries: https://github.com/nh2/static-haskell-nix/releases/tag/c-static-gtk3-apps-button-example-2020-11-23

nh2 commented 3 years ago

geekosaur on #haskell IRC gave me a tip that building UIs with glade and loading the corresponding XML file might cause GTK to dlopen() stuff, I should check how that behaves in static exes.

nh2 commented 3 years ago

might cause GTK to dlopen() stuff

Indeed, loading glade files does not work yet; I made a demo glade app glade-example-main.c (must be run from its directory to find the .glade file) and it errors with:

(demo-glade:8872): GModule-CRITICAL **: 22:19:02.953: g_module_symbol: assertion 'module != NULL' failed

(demo-glade:8872): GModule-CRITICAL **: 22:19:02.986: g_module_close: assertion 'module != NULL' failed
Dynamic loading not supported
Failed to load module: /nix/store/88gpkpcfjbgihn3fl8b8vk5ggfs8wn73-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so
Dynamic loading not supported
Failed to load module: /nix/store/d6l7xwbdm23xgds5vafzibw57790zw71-glib-networking-2.64.3/lib/gio/modules/libgiolibproxy.so
Dynamic loading not supported
Failed to load module: /nix/store/d6l7xwbdm23xgds5vafzibw57790zw71-glib-networking-2.64.3/lib/gio/modules/libgiognutls.so
Dynamic loading not supported
Failed to load module: /nix/store/d6l7xwbdm23xgds5vafzibw57790zw71-glib-networking-2.64.3/lib/gio/modules/libgiognomeproxy.so
Dynamic loading not supported
Failed to load module: /nix/store/bkjpypri81svkgq5rdfd4mdn33ic1pja-gvfs-1.44.1/lib/gio/modules/libgioremote-volume-monitor.so
Dynamic loading not supported
Failed to load module: /nix/store/bkjpypri81svkgq5rdfd4mdn33ic1pja-gvfs-1.44.1/lib/gio/modules/libgvfsdbus.so

(demo-glade:8872): GModule-CRITICAL **: 22:19:03.183: g_module_symbol: assertion 'module != NULL' failed

(demo-glade:8872): Gtk-ERROR **: 22:19:03.186: gtk_builder_connect_signals() requires working GModule

Need to check whether those disappear if you specify those libs as direct build dependencies of the static exe.