dhall-lang / dhall-lang

Maintainable configuration files
https://dhall-lang.org
BSD 3-Clause "New" or "Revised" License
4.23k stars 174 forks source link

Create packages for `dhall` and `dhall-json` for various Linux distros #192

Open Gabriella439 opened 6 years ago

Gabriella439 commented 6 years ago

Several people have requested an easier way of installing Dhall-related packages on Linux distributions, so any help packaging them for various Linux distributions is welcome

Gabriella439 commented 6 years ago

Alright, so I took an initial stab at creating an RPM for dhall-haskell using Nix. Apparently Nixpkgs has infrastructure for doing this correctly and the only issue I ran into is that at least one dependency (insert-ordered-containers) has not yet been packaged for Fedora yet.

The low-level utility that Nixpkgs uses for building RPMs is located here:

https://github.com/NixOS/nixpkgs/blob/dab1b67f9aacc388332867fb2ee9490828e6ae45/pkgs/build-support/release/rpm-build.nix

Here's an example of how to use it (in this case it's being used to create an RPM for Nix):

https://github.com/NixOS/nix/blob/1b34b69b45106e5bfdbdc0201d3ff4dcd36632f0/release.nix#L330-L342

The default Fedora releases supported by Nixpkgs are located here:

https://github.com/NixOS/nixpkgs/blob/db00e457aca5e765957005ec9e1a41400e2bcb70/pkgs/build-support/vm/default.nix#L739-L770

In this case it's just Fedora 26 and Fedora 27 (without updates). However, it's very easy to support newer releases of Fedora (like Fedora 28) and also releases plus backported updates.

For example, here is the latest draft of what I tested:

let
  pkgs = import <nixpkgs> { };

  fedora-28-updates-x86_64 =
    let version = "28";
    in rec {
      name = "fedora-${version}-updates-x86_64";
      fullName = "Fedora ${version} (x86_64)";
      packagesLists =
        [ (pkgs.fetchurl rec {
            url = "mirror://fedora/linux/releases/${version}/Everything/x86_64/os/repodata/${sha256}-primary.xml.gz";
            sha256 = "9659dfc44de50563682658fd41f2ab0da60e5657e1cc4f598b50d39fb3436e0c";
          })
          (pkgs.fetchurl rec {
            url = "mirror://fedora/linux/updates/${version}/Everything/x86_64/repodata/${sha256}-primary.xml.gz";
            sha256 = "ba89db3f59c6710fa1fd9a087940ed390b3fbcaa36cf577ab6c0c450f43b8a78";
          })
        ];
      urlPrefixes =
        [ "mirror://fedora/linux/releases/${version}/Everything/x86_64/os"
          "mirror://fedora/linux/updates/${version}/Everything/x86_64/"
        ];
      archs = ["noarch" "x86_64"];
      packages = pkgs.vmTools.commonFedoraPackages ++ [ "cronie" "util-linux" ];
      unifiedSystemDir = true;
    };
in
  pkgs.releaseTools.rpmBuild {
    name = "dhall-rpm";
    src = ./dhall-1.15.1.tar.gz;
    diskImage = pkgs.vmTools.makeImageFromRPMDist (fedora-28-updates-x86_64 // {
      extraPackages = [
        "chrpath"
        "ghc-Cabal-devel"
        "ghc-Diff-devel"
        "ghc-ansi-terminal-devel"
        "ghc-bytestring-devel"
        "ghc-case-insensitive-devel"
        "ghc-containers-devel"
        "ghc-contravariant-devel"
        "ghc-cryptonite-devel"
        "ghc-directory-devel"
        "ghc-exceptions-devel"
        "ghc-filepath-devel"
        "ghc-haskeline-devel"
        "ghc-http-client-devel"
        "ghc-http-client-tls-devel"
        "ghc-insert-ordered-containers-devel"
        "ghc-lens-family-core-devel"
        "ghc-megaparsec-devel"
        "ghc-memory-devel"
        "ghc-mtl-devel"
        "ghc-optparse-applicative-devel"
        "ghc-parsers-devel"
        "ghc-prettyprinter-ansi-terminal-devel"
        "ghc-prettyprinter-devel"
        "ghc-repline-devel"
        "ghc-rpm-macros"
        "ghc-scientific-devel"
        "ghc-template-haskell-devel"
        "ghc-text-devel"
        "ghc-transformers-devel"
        "ghc-unordered-containers-devel"
        "ghc-vector-devel"
      ];
    });
  }

The tarball I created by first running cabal-rpm spec . inside the dhall-haskell project to generate a ghc-dhall.spec file and then manually compressing the project. This can definitely be automated using Nix and I just haven't gotten around to it. The generated spec file looks like this:

# generated by cabal-rpm-0.12.4
# https://fedoraproject.org/wiki/Packaging:Haskell

%global pkg_name dhall
%global pkgver %{pkg_name}-%{version}

%bcond_with tests

Name:           ghc-%{pkg_name}
Version:        1.15.1
Release:        1%{?dist}
Summary:        A configuration language guaranteed to terminate

License:        BSD
Url:            https://hackage.haskell.org/package/%{pkg_name}
Source0:        https://hackage.haskell.org/package/%{pkgver}/%{pkgver}.tar.gz

BuildRequires:  ghc-Cabal-devel
BuildRequires:  ghc-rpm-macros
# Begin cabal-rpm deps:
BuildRequires:  chrpath
BuildRequires:  ghc-Diff-devel
BuildRequires:  ghc-ansi-terminal-devel
BuildRequires:  ghc-bytestring-devel
BuildRequires:  ghc-case-insensitive-devel
BuildRequires:  ghc-containers-devel
BuildRequires:  ghc-contravariant-devel
BuildRequires:  ghc-cryptonite-devel
BuildRequires:  ghc-directory-devel
BuildRequires:  ghc-exceptions-devel
BuildRequires:  ghc-filepath-devel
BuildRequires:  ghc-haskeline-devel
BuildRequires:  ghc-http-client-devel
BuildRequires:  ghc-http-client-tls-devel
BuildRequires:  ghc-insert-ordered-containers-devel
BuildRequires:  ghc-lens-family-core-devel
BuildRequires:  ghc-megaparsec-devel
BuildRequires:  ghc-memory-devel
BuildRequires:  ghc-mtl-devel
BuildRequires:  ghc-optparse-applicative-devel
BuildRequires:  ghc-parsers-devel
BuildRequires:  ghc-prettyprinter-ansi-terminal-devel
BuildRequires:  ghc-prettyprinter-devel
BuildRequires:  ghc-repline-devel
BuildRequires:  ghc-scientific-devel
BuildRequires:  ghc-template-haskell-devel
BuildRequires:  ghc-text-devel
BuildRequires:  ghc-transformers-devel
BuildRequires:  ghc-unordered-containers-devel
BuildRequires:  ghc-vector-devel
%if %{with tests}
BuildRequires:  ghc-deepseq-devel
BuildRequires:  ghc-doctest-devel
BuildRequires:  ghc-mockery-devel
BuildRequires:  ghc-tasty-devel
BuildRequires:  ghc-tasty-hunit-devel
%endif
# End cabal-rpm deps

%description
Dhall is an explicitly typed configuration language that is not Turing
complete. Despite being Turing incomplete, Dhall is a real programming language
with a type-checker and evaluator.

Use this library to parse, type-check, evaluate, and pretty-print the Dhall
configuration language. This package also includes an executable which
type-checks a Dhall file and reduces the file to a fully evaluated normal form.

Read "Dhall.Tutorial" to learn how to use this library.

%package devel
Summary:        Haskell %{pkg_name} library development files
Provides:       %{name}-static = %{version}-%{release}
Provides:       %{name}-doc = %{version}-%{release}
%if %{defined ghc_version}
Requires:       ghc-compiler = %{ghc_version}
Requires(post): ghc-compiler = %{ghc_version}
Requires(postun): ghc-compiler = %{ghc_version}
%endif
Requires:       %{name}%{?_isa} = %{version}-%{release}

%description devel
This package provides the Haskell %{pkg_name} library development files.

%prep
%setup -q -n %{pkgver}

%build
%ghc_lib_build

%install
%ghc_lib_install
%ghc_fix_rpath %{pkgver}

%check
%cabal_test

%post devel
%ghc_pkg_recache

%postun devel
%ghc_pkg_recache

%files -f %{name}.files
%license LICENSE

%files devel -f %{name}-devel.files
%doc CHANGELOG.md README.md doctest examples
%{_bindir}/%{pkg_name}

%changelog
* Sun Jul 15 2018 Fedora Haskell SIG <haskell@lists.fedoraproject.org> - 1.15.1-1
- spec file generated by cabal-rpm-0.12.4

The way you get the hash for a given release or update is just to visit the relevant URL and find the name of the file ending with primary.xml, such as in these two directories for the Fedora 28 release and updates, respectively:

The way I got the set of extraPackages was that I just built without them and then the failed build told me which dependencies were missing. This could probably be automatically computed from the Haskell derivation with a little bit of automation, but it's not too hard to manually maintain in the short term.

When I built the above derivation I got the following error:

$ nix build dhall-rpm.nix
builder for '/nix/store/0jwf0hfp5frjwhpbgwa7z2pnak4j4j1n-fedora-28-updates-x86_64.nix.drv' failed with exit code 255; last 10 log lines:
  >>> ghc-hourglass-devel
  >>> ghc-x509-devel
  >>> ghc-asn1-parse-devel
  >>> ghc-pem-devel
  >>> ghc-x509-store-devel
  >>> ghc-x509-validation-devel
  >>> ghc-x509-system-devel
  >>> ghc-http-client-tls
  >>> ghc-insert-ordered-containers-devel
  package ghc-insert-ordered-containers-devel doesn't exist at /nix/store/wv6n8g4v01iz9lpzx81qnvfkwkwx148n-rpm-closure.pl line 135.
[0 built (1 failed), 0.0 MiB DL]

... and I confirmed that insert-ordered-containers has not yet been packaged for Fedora:

http://rpmfind.net/linux/rpm2html/search.php?query=ghc-insert-ordered-containers&submit=Search+...&system=&arch=x86_64

... so the next step is I'm going to ask the Fedora Haskell SIG if I need to do that myself or if they can do that for me.

In parallel, I'm also going to try to repeat the same process for Debian since Nix has similar support for building Debian packages, too.

Gabriella439 commented 6 years ago

I just learned that there is an effort in Nixpkgs to build completely statically linked Haskell executables here:

https://github.com/NixOS/nixpkgs/issues/43795

If that works then we could also serve completely static binaries that would work on basically any Linux distribution

nh2 commented 6 years ago

@Gabriel439 I have just succeeded building dhall statically for Linux after cherry-picking some tinfo fixes.

It's 14 MB in size.

Here is a binary if you want to try it: dhall-musl-static-1.15.1-unofficial.tar.gz

Once I have committed my heap of local changes, I'll ping you on how to build it from source as well.

nh2 commented 6 years ago

Once I have committed my heap of local changes, I'll ping you on how to build it from source as well.

@Gabriel439 Done https://github.com/nh2/static-haskell-nix/blob/ef283274ce193f713082591dd462f4bd3fb4dd1f/survey/default.nix#L117

Build via

NIX_PATH=nixpkgs=https://github.com/nh2/nixpkgs/archive/925aac04f4ca58aceb83beef18cb7dae0715421b.tar.gz nix-build --no-link survey/default.nix -A haskellPackages.dhall

Profpatsch commented 6 years ago

It's 14 MB in size.

There must be some unnecessary parts that can be stripped in there.

nh2 commented 6 years ago

There must be some unnecessary parts that can be stripped in there.

Not sure, the dynamically linked version is of similar size:

% du -h /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall
7.5M    /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall

% ldd /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall | grep -o '=> .*' | grep -o ' .*.so[^ ]*' | xargs du -hL --total
1.4M    /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libm.so.6
1.5M    /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6
100K    /nix/store/r43dk927l97n78rff7hnvsq49f3szkg6-zlib-1.2.11/lib/libz.so.1
44K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/librt.so.1
16K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libutil.so.1
20K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libdl.so.2
624K    /nix/store/8qfd8gx0j3yzamkrbrfz5kc00h4cqd1q-gmp-6.1.2/lib/libgmp.so.10
2.0M    /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libc.so.6
136K    /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
160K    /lib64/ld-linux-x86-64.so.2
104K    /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libgcc_s.so.1
6.0M    total

So 13.5 MB in total. Both static and dynamic versions are stripped.

Profpatsch commented 6 years ago

1.5M /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6

huh?

7.5M /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall

The problem is that strip can’t do much with Haskell binaries[citation needed].

f-f commented 6 years ago

Googling around about this I read that passing -split-objs to GHC might help

Gabriella439 commented 6 years ago

I think 14 MB static binary is still great

Gabriella439 commented 6 years ago

dhall and dhall-json are now available as tarballs containing static executables

Links to their latest build here:

Links directly to the latest tarballs here:

I'll also include these tarballs in the GitHub release page for each new version

joneshf commented 6 years ago

This is awesome!

Can we utilize the releases of hydra to find specific versions easily? Like https://hydra.dhall-lang.org/build/2356 and https://hydra.dhall-lang.org/build/2383 are both dhall-json-1.2.1. Which one should you choose? It's not clear which of these builds is the real version 1.2.1, and which is really master. Are either of those builds the real version 1.2.1?

I think if we have that, we can also close this issue: https://github.com/dhall-lang/dhall-haskell/issues/235

nh2 commented 6 years ago

The problem is that strip can’t do much with Haskell binaries[citation needed].

@Profpatsch Can you elaborate? From what I have seen, the core feature of strip (removing function labels for debugging) seems to work and does shrink Haskell executables; if you disable stripping you usually find it quite a bit larger also for Haskell. Is there more it could do?

1.5M /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6

huh?

@Profpatsch That's a good question. No diea what part of Dhall or the build process pulls in libstdc++.

It would be great to find out, maybe we can fix this and thus reduce the dynamic closure of dhall a bit more. For the static version, it will most likely already be eliminated because that one only links in functions that are actually used.

Googling around about this I read that passing -split-objs to GHC might help

@f-f The binaries produced here already do that, as -split-sections, the successor to -split-objs is already enabled by default in nixpkgs.

nh2 commented 6 years ago

Dhall seems to somehow directly depend on libstdc++ in the dynamic nix build, it's not even via some other dependency:

% lddtree /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall
dhall => /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall (interpreter => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/ld-linux-x86-64.so.2)
    libm.so.6 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libm.so.6
        ld-linux-x86-64.so.2 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/ld-linux-x86-64.so.2
    libstdc++.so.6 => /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6
        libgcc_s.so.1 => /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libgcc_s.so.1
    libz.so.1 => /nix/store/r43dk927l97n78rff7hnvsq49f3szkg6-zlib-1.2.11/lib/libz.so.1
    librt.so.1 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/librt.so.1
    libutil.so.1 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libutil.so.1
    libdl.so.2 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libdl.so.2
    libgmp.so.10 => /nix/store/8qfd8gx0j3yzamkrbrfz5kc00h4cqd1q-gmp-6.1.2/lib/libgmp.so.10
    libc.so.6 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libc.so.6
    libpthread.so.0 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
Gabriella439 commented 6 years ago

@joneshf: In theory the build you would choose is for the revision of master labeled Version X.Y.Z. In practice, I wouldn't use Hydra for retrieving historic versions because I have it configured to only keep GC roots for the last build (to conserve space). Hydra does have a feature to explicitly keep a build and create a GC root for it but I don't plan to use it.

What I do plan to do is create releases on GitHub and include the static tarball in each release. That will allow GitHub to host the tarball and provide a reliable URL for downloading old releases. For example, I just created a release for dhall-1.16.1:

https://github.com/dhall-lang/dhall-haskell/releases/tag/1.16.1

.... and included the tarball with it, so now you can reliably retrieve it using:

https://github.com/dhall-lang/dhall-haskell/releases/download/1.16.1/dhall-1.16.1-x86_64-linux.tar.bz2

I'm also going to update dhall-{json,nix,bash,text} to generate static tarballs and cut new releases with them so that you can also get static tarballs for them, too.

Profpatsch commented 6 years ago

@nh2 use nix why-depends (from nix 2.0)

nh2 commented 6 years ago

@Profpatsch Doesn't seem to output too much interesting stuff we don't know yet:

% nix why-depends /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib
/nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2
╚═══bin/dhall: …p-glibc-2.26-131/lib:/nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib.XXXXXXXXXXXXX…
    => /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib

It shows the chain of dependencies (we already know that, it's a direct dependency), but we're not sure why this direct dependency exists.

joneshf commented 6 years ago

@Gabriel439 That's perfect! Thanks!

nh2 commented 5 years ago

It looks like there's a size reduction coming up for static dhall: https://github.com/NixOS/nixpkgs/issues/43795#issuecomment-493712831

dhall went from 14 MB to 9.1 MB

ocharles commented 5 years ago

We distribute static executables - is that enough to close this issue? What's the remaining work?

Gabriella439 commented 5 years ago

@ocharles: I'd keep this open as the scope of this is to add the various Dhall packages as an installable package in various Linux package repositories (i.e. for Arch/Debian/etc.). This would most likely not use the static executables but instead use whatever idiom they use for packaging a Haskell library.

indolering commented 5 years ago

In past projects I've worked on we used the Open Build Service to generate packages for Suse, RedHat, CentOS, Fedora, Debian, Ubuntu, and Arch. It's pretty easy, you create a template which can build it locally or just download a binary from GitHub. You can use an API or WebHooks to integrate everything with your CI system. Free for FOSS projects.

Gabriella439 commented 5 years ago

@indolering: The last time I attempted to do this the issue wasn't building the package(s). Nix already supports utilities for creating RPMs/DEBs/etc. The problem I ran into was some of my dependencies were not yet packaged (such as insert-ordered-containers at the time, although we no longer depend on that).

The second issue is that we'd like to not just create package files but actually upstream the packages into each respective distribution.

indolering commented 4 years ago

Nix already supports utilities for creating RPMs/DEBs/etc.

Does it create a simple binary or will the RPM/Deb/Arch file produced use the native package manager to handle dependencies?

@Gabriel439 OBS will rebuild and test packages whenever dependencies are updated. It would also be useful for those that want feature updates past when the upstream distro accepts backports.

Gabriella439 commented 4 years ago

@indolering: The Nix build does the following:

That said, it would be way easier for me if we upstreamed maintenance of the dhall package to Fedora instead doing it ourselves.

indolering commented 4 years ago

@Gabriel439 That's impressive! OBS would still do all of that but as a free CI service and provide a repo for ~50 distro/version combinations. These are handy when you need features that haven't yet been/won't be back ported by the upstream distro. That being said, OBS can be pretty slow and their security isn't the best.

Gabriella439 commented 4 years ago

@indolering: Really, the non-trivial bit here is who is responsible for maintaining the RPM. I currently do not feel qualified or equipped to do so. If it were just a matter of authoring the initial RPM I would have just put up a feature bounty for doing so a long time ago.

l1x commented 3 years ago

@indolering: The last time I attempted to do this the issue wasn't building the package(s). Nix already supports utilities for creating RPMs/DEBs/etc. The problem I ran into was some of my dependencies were not yet packaged (such as insert-ordered-containers at the time, although we no longer depend on that).

The second issue is that we'd like to not just create package files but actually upstream the packages into each respective distribution.

What would it take to have a package in Debian? I wanted to use Dhall on RPI4 but on Github there are only AMD64 builds and no ARM64, Nix is not an option and using a source build fails (requires more than 4G ram). What do you need to have an upstream DEB?

TristanCacqueray commented 3 years ago

@l1x a Debian developer would need to create the package (and one for each of the missing dependencies).

This has been done for Fedora where you can now dnf install dhall through the official repository. For the record, here is the initial request: https://bugzilla.redhat.com/show_bug.cgi?id=1871697 , and the spec file is now maintained here: https://src.fedoraproject.org/rpms/dhall/tree/rawhide

l1x commented 3 years ago

@l1x a Debian developer would need to create the package (and one for each of the missing dependencies).

This has been done for Fedora where you can now dnf install dhall through the official repository. For the record, here is the initial request: https://bugzilla.redhat.com/show_bug.cgi?id=1871697 , and the spec file is now maintained here: https://src.fedoraproject.org/rpms/dhall/tree/rawhide

Ok, I am going to dig into the Debian package