Open Gabriella439 opened 6 years ago
Alright, so I took an initial stab at creating an RPM for dhall-haskell
using Nix. Apparently Nixpkgs has infrastructure for doing this correctly and the only issue I ran into is that at least one dependency (insert-ordered-containers
) has not yet been packaged for Fedora yet.
The low-level utility that Nixpkgs uses for building RPMs is located here:
Here's an example of how to use it (in this case it's being used to create an RPM for Nix):
https://github.com/NixOS/nix/blob/1b34b69b45106e5bfdbdc0201d3ff4dcd36632f0/release.nix#L330-L342
The default Fedora releases supported by Nixpkgs are located here:
In this case it's just Fedora 26 and Fedora 27 (without updates). However, it's very easy to support newer releases of Fedora (like Fedora 28) and also releases plus backported updates.
For example, here is the latest draft of what I tested:
let
pkgs = import <nixpkgs> { };
fedora-28-updates-x86_64 =
let version = "28";
in rec {
name = "fedora-${version}-updates-x86_64";
fullName = "Fedora ${version} (x86_64)";
packagesLists =
[ (pkgs.fetchurl rec {
url = "mirror://fedora/linux/releases/${version}/Everything/x86_64/os/repodata/${sha256}-primary.xml.gz";
sha256 = "9659dfc44de50563682658fd41f2ab0da60e5657e1cc4f598b50d39fb3436e0c";
})
(pkgs.fetchurl rec {
url = "mirror://fedora/linux/updates/${version}/Everything/x86_64/repodata/${sha256}-primary.xml.gz";
sha256 = "ba89db3f59c6710fa1fd9a087940ed390b3fbcaa36cf577ab6c0c450f43b8a78";
})
];
urlPrefixes =
[ "mirror://fedora/linux/releases/${version}/Everything/x86_64/os"
"mirror://fedora/linux/updates/${version}/Everything/x86_64/"
];
archs = ["noarch" "x86_64"];
packages = pkgs.vmTools.commonFedoraPackages ++ [ "cronie" "util-linux" ];
unifiedSystemDir = true;
};
in
pkgs.releaseTools.rpmBuild {
name = "dhall-rpm";
src = ./dhall-1.15.1.tar.gz;
diskImage = pkgs.vmTools.makeImageFromRPMDist (fedora-28-updates-x86_64 // {
extraPackages = [
"chrpath"
"ghc-Cabal-devel"
"ghc-Diff-devel"
"ghc-ansi-terminal-devel"
"ghc-bytestring-devel"
"ghc-case-insensitive-devel"
"ghc-containers-devel"
"ghc-contravariant-devel"
"ghc-cryptonite-devel"
"ghc-directory-devel"
"ghc-exceptions-devel"
"ghc-filepath-devel"
"ghc-haskeline-devel"
"ghc-http-client-devel"
"ghc-http-client-tls-devel"
"ghc-insert-ordered-containers-devel"
"ghc-lens-family-core-devel"
"ghc-megaparsec-devel"
"ghc-memory-devel"
"ghc-mtl-devel"
"ghc-optparse-applicative-devel"
"ghc-parsers-devel"
"ghc-prettyprinter-ansi-terminal-devel"
"ghc-prettyprinter-devel"
"ghc-repline-devel"
"ghc-rpm-macros"
"ghc-scientific-devel"
"ghc-template-haskell-devel"
"ghc-text-devel"
"ghc-transformers-devel"
"ghc-unordered-containers-devel"
"ghc-vector-devel"
];
});
}
The tarball I created by first running cabal-rpm spec .
inside the dhall-haskell
project to generate a ghc-dhall.spec
file and then manually compressing the project. This can definitely be automated using Nix and I just haven't gotten around to it. The generated spec file looks like this:
# generated by cabal-rpm-0.12.4
# https://fedoraproject.org/wiki/Packaging:Haskell
%global pkg_name dhall
%global pkgver %{pkg_name}-%{version}
%bcond_with tests
Name: ghc-%{pkg_name}
Version: 1.15.1
Release: 1%{?dist}
Summary: A configuration language guaranteed to terminate
License: BSD
Url: https://hackage.haskell.org/package/%{pkg_name}
Source0: https://hackage.haskell.org/package/%{pkgver}/%{pkgver}.tar.gz
BuildRequires: ghc-Cabal-devel
BuildRequires: ghc-rpm-macros
# Begin cabal-rpm deps:
BuildRequires: chrpath
BuildRequires: ghc-Diff-devel
BuildRequires: ghc-ansi-terminal-devel
BuildRequires: ghc-bytestring-devel
BuildRequires: ghc-case-insensitive-devel
BuildRequires: ghc-containers-devel
BuildRequires: ghc-contravariant-devel
BuildRequires: ghc-cryptonite-devel
BuildRequires: ghc-directory-devel
BuildRequires: ghc-exceptions-devel
BuildRequires: ghc-filepath-devel
BuildRequires: ghc-haskeline-devel
BuildRequires: ghc-http-client-devel
BuildRequires: ghc-http-client-tls-devel
BuildRequires: ghc-insert-ordered-containers-devel
BuildRequires: ghc-lens-family-core-devel
BuildRequires: ghc-megaparsec-devel
BuildRequires: ghc-memory-devel
BuildRequires: ghc-mtl-devel
BuildRequires: ghc-optparse-applicative-devel
BuildRequires: ghc-parsers-devel
BuildRequires: ghc-prettyprinter-ansi-terminal-devel
BuildRequires: ghc-prettyprinter-devel
BuildRequires: ghc-repline-devel
BuildRequires: ghc-scientific-devel
BuildRequires: ghc-template-haskell-devel
BuildRequires: ghc-text-devel
BuildRequires: ghc-transformers-devel
BuildRequires: ghc-unordered-containers-devel
BuildRequires: ghc-vector-devel
%if %{with tests}
BuildRequires: ghc-deepseq-devel
BuildRequires: ghc-doctest-devel
BuildRequires: ghc-mockery-devel
BuildRequires: ghc-tasty-devel
BuildRequires: ghc-tasty-hunit-devel
%endif
# End cabal-rpm deps
%description
Dhall is an explicitly typed configuration language that is not Turing
complete. Despite being Turing incomplete, Dhall is a real programming language
with a type-checker and evaluator.
Use this library to parse, type-check, evaluate, and pretty-print the Dhall
configuration language. This package also includes an executable which
type-checks a Dhall file and reduces the file to a fully evaluated normal form.
Read "Dhall.Tutorial" to learn how to use this library.
%package devel
Summary: Haskell %{pkg_name} library development files
Provides: %{name}-static = %{version}-%{release}
Provides: %{name}-doc = %{version}-%{release}
%if %{defined ghc_version}
Requires: ghc-compiler = %{ghc_version}
Requires(post): ghc-compiler = %{ghc_version}
Requires(postun): ghc-compiler = %{ghc_version}
%endif
Requires: %{name}%{?_isa} = %{version}-%{release}
%description devel
This package provides the Haskell %{pkg_name} library development files.
%prep
%setup -q -n %{pkgver}
%build
%ghc_lib_build
%install
%ghc_lib_install
%ghc_fix_rpath %{pkgver}
%check
%cabal_test
%post devel
%ghc_pkg_recache
%postun devel
%ghc_pkg_recache
%files -f %{name}.files
%license LICENSE
%files devel -f %{name}-devel.files
%doc CHANGELOG.md README.md doctest examples
%{_bindir}/%{pkg_name}
%changelog
* Sun Jul 15 2018 Fedora Haskell SIG <haskell@lists.fedoraproject.org> - 1.15.1-1
- spec file generated by cabal-rpm-0.12.4
The way you get the hash for a given release or update is just to visit the relevant URL and find the name of the file ending with primary.xml
, such as in these two directories for the Fedora 28 release and updates, respectively:
The way I got the set of extraPackages
was that I just built without them and then the failed build told me which dependencies were missing. This could probably be automatically computed from the Haskell derivation with a little bit of automation, but it's not too hard to manually maintain in the short term.
When I built the above derivation I got the following error:
$ nix build dhall-rpm.nix
builder for '/nix/store/0jwf0hfp5frjwhpbgwa7z2pnak4j4j1n-fedora-28-updates-x86_64.nix.drv' failed with exit code 255; last 10 log lines:
>>> ghc-hourglass-devel
>>> ghc-x509-devel
>>> ghc-asn1-parse-devel
>>> ghc-pem-devel
>>> ghc-x509-store-devel
>>> ghc-x509-validation-devel
>>> ghc-x509-system-devel
>>> ghc-http-client-tls
>>> ghc-insert-ordered-containers-devel
package ghc-insert-ordered-containers-devel doesn't exist at /nix/store/wv6n8g4v01iz9lpzx81qnvfkwkwx148n-rpm-closure.pl line 135.
[0 built (1 failed), 0.0 MiB DL]
... and I confirmed that insert-ordered-containers
has not yet been packaged for Fedora:
... so the next step is I'm going to ask the Fedora Haskell SIG if I need to do that myself or if they can do that for me.
In parallel, I'm also going to try to repeat the same process for Debian since Nix has similar support for building Debian packages, too.
I just learned that there is an effort in Nixpkgs to build completely statically linked Haskell executables here:
https://github.com/NixOS/nixpkgs/issues/43795
If that works then we could also serve completely static binaries that would work on basically any Linux distribution
@Gabriel439 I have just succeeded building dhall
statically for Linux after cherry-picking some tinfo
fixes.
It's 14 MB in size.
Here is a binary if you want to try it: dhall-musl-static-1.15.1-unofficial.tar.gz
Once I have committed my heap of local changes, I'll ping you on how to build it from source as well.
Once I have committed my heap of local changes, I'll ping you on how to build it from source as well.
@Gabriel439 Done https://github.com/nh2/static-haskell-nix/blob/ef283274ce193f713082591dd462f4bd3fb4dd1f/survey/default.nix#L117
Build via
NIX_PATH=nixpkgs=https://github.com/nh2/nixpkgs/archive/925aac04f4ca58aceb83beef18cb7dae0715421b.tar.gz nix-build --no-link survey/default.nix -A haskellPackages.dhall
It's 14 MB in size.
There must be some unnecessary parts that can be stripped in there.
There must be some unnecessary parts that can be stripped in there.
Not sure, the dynamically linked version is of similar size:
% du -h /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall
7.5M /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall
% ldd /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall | grep -o '=> .*' | grep -o ' .*.so[^ ]*' | xargs du -hL --total
1.4M /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libm.so.6
1.5M /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6
100K /nix/store/r43dk927l97n78rff7hnvsq49f3szkg6-zlib-1.2.11/lib/libz.so.1
44K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/librt.so.1
16K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libutil.so.1
20K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libdl.so.2
624K /nix/store/8qfd8gx0j3yzamkrbrfz5kc00h4cqd1q-gmp-6.1.2/lib/libgmp.so.10
2.0M /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libc.so.6
136K /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
160K /lib64/ld-linux-x86-64.so.2
104K /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libgcc_s.so.1
6.0M total
So 13.5 MB in total. Both static and dynamic versions are strip
ped.
1.5M /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6
huh?
7.5M /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall
The problem is that strip
can’t do much with Haskell binaries[citation needed].
Googling around about this I read that passing -split-objs
to GHC might help
I think 14 MB static binary is still great
dhall
and dhall-json
are now available as tarballs containing static executables
Links to their latest build here:
Links directly to the latest tarballs here:
I'll also include these tarballs in the GitHub release page for each new version
This is awesome!
Can we utilize the releases of hydra to find specific versions easily? Like https://hydra.dhall-lang.org/build/2356 and https://hydra.dhall-lang.org/build/2383 are both dhall-json-1.2.1
. Which one should you choose? It's not clear which of these builds is the real version 1.2.1, and which is really master. Are either of those builds the real version 1.2.1?
I think if we have that, we can also close this issue: https://github.com/dhall-lang/dhall-haskell/issues/235
The problem is that strip can’t do much with Haskell binaries[citation needed].
@Profpatsch Can you elaborate? From what I have seen, the core feature of strip
(removing function labels for debugging) seems to work and does shrink Haskell executables; if you disable stripping you usually find it quite a bit larger also for Haskell. Is there more it could do?
1.5M /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6
huh?
@Profpatsch That's a good question. No diea what part of Dhall or the build process pulls in libstdc++
.
It would be great to find out, maybe we can fix this and thus reduce the dynamic closure of dhall a bit more. For the static version, it will most likely already be eliminated because that one only links in functions that are actually used.
Googling around about this I read that passing
-split-objs
to GHC might help
@f-f The binaries produced here already do that, as -split-sections
, the successor to -split-objs
is already enabled by default in nixpkgs.
Dhall seems to somehow directly depend on libstdc++
in the dynamic nix build, it's not even via some other dependency:
% lddtree /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall
dhall => /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall (interpreter => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/ld-linux-x86-64.so.2)
libm.so.6 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libm.so.6
ld-linux-x86-64.so.2 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/ld-linux-x86-64.so.2
libstdc++.so.6 => /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libstdc++.so.6
libgcc_s.so.1 => /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib/libgcc_s.so.1
libz.so.1 => /nix/store/r43dk927l97n78rff7hnvsq49f3szkg6-zlib-1.2.11/lib/libz.so.1
librt.so.1 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/librt.so.1
libutil.so.1 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libutil.so.1
libdl.so.2 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libdl.so.2
libgmp.so.10 => /nix/store/8qfd8gx0j3yzamkrbrfz5kc00h4cqd1q-gmp-6.1.2/lib/libgmp.so.10
libc.so.6 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libc.so.6
libpthread.so.0 => /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
@joneshf: In theory the build you would choose is for the revision of master
labeled Version X.Y.Z
. In practice, I wouldn't use Hydra for retrieving historic versions because I have it configured to only keep GC roots for the last build (to conserve space). Hydra does have a feature to explicitly keep a build and create a GC root for it but I don't plan to use it.
What I do plan to do is create releases on GitHub and include the static tarball in each release. That will allow GitHub to host the tarball and provide a reliable URL for downloading old releases. For example, I just created a release for dhall-1.16.1
:
https://github.com/dhall-lang/dhall-haskell/releases/tag/1.16.1
.... and included the tarball with it, so now you can reliably retrieve it using:
I'm also going to update dhall-{json,nix,bash,text}
to generate static tarballs and cut new releases with them so that you can also get static tarballs for them, too.
@nh2 use nix why-depends
(from nix 2.0
)
@Profpatsch Doesn't seem to output too much interesting stuff we don't know yet:
% nix why-depends /nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2/bin/dhall /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib
/nix/store/2cmvn0lp5jxqc3s7cxv0ws2rc6h67jqs-dhall-1.8.2
╚═══bin/dhall: …p-glibc-2.26-131/lib:/nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib/lib.XXXXXXXXXXXXX…
=> /nix/store/4zd34747fz0ggzzasy4icgn3lmy89pra-gcc-7.3.0-lib
It shows the chain of dependencies (we already know that, it's a direct dependency), but we're not sure why this direct dependency exists.
@Gabriel439 That's perfect! Thanks!
It looks like there's a size reduction coming up for static dhall
: https://github.com/NixOS/nixpkgs/issues/43795#issuecomment-493712831
dhall
went from 14 MB to 9.1 MB
We distribute static executables - is that enough to close this issue? What's the remaining work?
@ocharles: I'd keep this open as the scope of this is to add the various Dhall packages as an installable package in various Linux package repositories (i.e. for Arch/Debian/etc.). This would most likely not use the static executables but instead use whatever idiom they use for packaging a Haskell library.
In past projects I've worked on we used the Open Build Service to generate packages for Suse, RedHat, CentOS, Fedora, Debian, Ubuntu, and Arch. It's pretty easy, you create a template which can build it locally or just download a binary from GitHub. You can use an API or WebHooks to integrate everything with your CI system. Free for FOSS projects.
@indolering: The last time I attempted to do this the issue wasn't building the package(s). Nix already supports utilities for creating RPMs/DEBs/etc. The problem I ran into was some of my dependencies were not yet packaged (such as insert-ordered-containers
at the time, although we no longer depend on that).
The second issue is that we'd like to not just create package files but actually upstream the packages into each respective distribution.
Nix already supports utilities for creating RPMs/DEBs/etc.
Does it create a simple binary or will the RPM/Deb/Arch file produced use the native package manager to handle dependencies?
@Gabriel439 OBS will rebuild and test packages whenever dependencies are updated. It would also be useful for those that want feature updates past when the upstream distro accepts backports.
@indolering: The Nix build does the following:
rpmbuild -ta
on a TAR archive containing the source and a specification file to create source and binary RPMsThat said, it would be way easier for me if we upstreamed maintenance of the dhall
package to Fedora instead doing it ourselves.
@Gabriel439 That's impressive! OBS would still do all of that but as a free CI service and provide a repo for ~50 distro/version combinations. These are handy when you need features that haven't yet been/won't be back ported by the upstream distro. That being said, OBS can be pretty slow and their security isn't the best.
@indolering: Really, the non-trivial bit here is who is responsible for maintaining the RPM. I currently do not feel qualified or equipped to do so. If it were just a matter of authoring the initial RPM I would have just put up a feature bounty for doing so a long time ago.
@indolering: The last time I attempted to do this the issue wasn't building the package(s). Nix already supports utilities for creating RPMs/DEBs/etc. The problem I ran into was some of my dependencies were not yet packaged (such as
insert-ordered-containers
at the time, although we no longer depend on that).The second issue is that we'd like to not just create package files but actually upstream the packages into each respective distribution.
What would it take to have a package in Debian? I wanted to use Dhall on RPI4 but on Github there are only AMD64 builds and no ARM64, Nix is not an option and using a source build fails (requires more than 4G ram). What do you need to have an upstream DEB?
@l1x a Debian developer would need to create the package (and one for each of the missing dependencies).
This has been done for Fedora where you can now dnf install dhall
through the official repository. For the record, here is the initial request: https://bugzilla.redhat.com/show_bug.cgi?id=1871697 , and the spec file is now maintained here: https://src.fedoraproject.org/rpms/dhall/tree/rawhide
@l1x a Debian developer would need to create the package (and one for each of the missing dependencies).
This has been done for Fedora where you can now
dnf install dhall
through the official repository. For the record, here is the initial request: https://bugzilla.redhat.com/show_bug.cgi?id=1871697 , and the spec file is now maintained here: https://src.fedoraproject.org/rpms/dhall/tree/rawhide
Ok, I am going to dig into the Debian package
Several people have requested an easier way of installing Dhall-related packages on Linux distributions, so any help packaging them for various Linux distributions is welcome