Open vbrandl opened 4 years ago
It potentially auto-detects whether openssl is available at build time. You may need a "crateOverrides" that adds openssl to the buildInputs. And /or add the corresponding feature if there is one.
Usually, nix detects of there are references to build time dependencies in the output and adds them as runtime dependencies.
https support seems to be the default :
features = {
"default" = [ "ssh" "https" "ssh_key_from_memory" ];
"https" = [ "libgit2-sys/https" "openssl-sys" "openssl-probe" ];
"ssh" = [ "libgit2-sys/ssh" ];
"ssh_key_from_memory" = [ "libgit2-sys/ssh_key_from_memory" ];
"vendored-openssl" = [ "openssl-sys/vendored" ];
};
I'm not quite sure how to do this (sorry if the question is stupid...)
I added a default.nix
for the build and tried adding openssl
as a dependency to both git2
and libgit2-sys
(I also tested only one of them at a time):
cargoNix = callPackage ./Cargo.nix {
defaultCrateOverrides = pkgs.defaultCrateOverrides // {
git2 = attrs: {
buildInputs = [ pkgs.openssl ];
};
libgit2-sys = attrs: {
buildInputs = [ pkgs.openssl pkgs.cacert ];
};
};
};
When only adding the dependency to git2
, the error persists.
When adding the dependency to libgit2-sys
, this results in a compile error for libgit2-sys
which doesn't really help me:
error occurred: Command "gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-I" "/build/libgit2-sys-0.10.0/target/build/libgit2-sys.out/include" "-I" "libgit2/src" "-I" "libgit2/deps/http-parser" "-I" "libgit2/deps/pcre" "-fvisibility=hidden" "-DGIT_REGEX_BUILTIN=1" "-DHAVE_STDINT_H=1" "-DHAVE_MEMMOVE=1" "-DNO_RECURSE=1" "-DNEWLINE=10" "-DPOSIX_MALLOC_THRESHOLD=10" "-DLINK_SIZE=2" "-DPARENS_NEST_LIMIT=250" "-DMATCH_LIMIT=10000000" "-DMATCH_LIMIT_RECURSION=MATCH_LIMIT" "-DMAX_NAME_SIZE=32" "-DMAX_NAME_COUNT=10000" "-DSHA1DC_NO_STANDARD_INCLUDES=1" "-DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"common.h\"" "-DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"common.h\"" "-o" "/build/libgit2-sys-0.10.0/target/build/libgit2-sys.out/build/libgit2/src/zstream.o" "-c" "libgit2/src/zstream.c" with args "gcc" did not execute successfully (status code exit code: 1).
The error is probably further above, try nix log
.
This is not really the place for general nix advice. Maybe you can ask in IRC or user forums?
If you find a solution, you might post back here or create a pull request for defaultCrateOverrides in nix pkgs
Turns out the compile error wasn't a problem in defaultCrateOverrides
but I just forgot to add zlib
as a build dependency.
But now that the crate builds using my overrides, I get the same runtime error as in the beginning...
I haven't used crate2nix
and don't know for sure, but for what it's worth this does work with rustPlatform.buildRustPackage
inside Nix, which suggests to me it's an issue (somewhere) in the generated crate2nix
dependency specification for cacert
or openssl
:
https://github.com/bhipple/nixpkgs/commit/cc035068f25fd1dc2506b7b038ed2281539450c0
$ nix-build -A hits-of-code
/nix/store/3fgv4l77ip6ydafd2c9fqjvlphmwa0cp-hits-of-code-0.11.5
$ .result/bin/hoc
2020-02-16T11:05:09.880810039-05:00 INFO actix_server::builder - Starting 4 workers
2020-02-16T11:05:09.880988584-05:00 INFO actix_server::builder - Starting "actix-web-service-0.0.0.0:8080" service on 0.0.0.0:8080
The error only occurs when a git clone or pull is performed. In your case that means requesting http://0.0.0.0:8080/view/github/vbrandl/hoc or some other repo. That's why I created the testcase where the error occurs directly when executing the binary.
It still appears to work with my package:
λ brh ~ → /nix/store/3fgv4l77ip6ydafd2c9fqjvlphmwa0cp-hits-of-code-0.11.5/bin/hoc
2020-02-16T12:35:03.322505396-05:00 INFO actix_server::builder - Starting 4 workers
2020-02-16T12:35:03.322878178-05:00 INFO actix_server::builder - Starting "actix-web-service-0.0.0.0:8080" service on 0.0.0.0:8080
2020-02-16T12:35:10.602161387-05:00 INFO hoc - Cloning https://github.com/vbrandl/hoc for the first time
2020-02-16T12:35:11.135063113-05:00 INFO hoc - Creating cache for ./repos/github.com/vbrandl/hoc
2020-02-16T12:35:11.293053764-05:00 INFO actix_web::middleware::logger - 127.0.0.1:35218 "GET /view/github/vbrandl/hoc HTTP/1.1" 200 1720 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" 1.033479
2020-02-16T12:35:11.323393030-05:00 INFO actix_web::middleware::logger - 127.0.0.1:35218 "GET /tacit-css.min.css HTTP/1.1" 200 6597 "http://0.0.0.0:8080/view/github/vbrandl/hoc" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" 0.000071
2020-02-16T12:35:11.413120138-05:00 INFO actix_web::middleware::logger - 127.0.0.1:35218 "GET /favicon.ico HTTP/1.1" 200 983 "http://0.0.0.0:8080/view/github/vbrandl/hoc" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36" 0.000188
which produces this:
That's strange. So it seems to be a problem with my store? I'm using Arch Linux and installed nix separately. Are you on NixOS?
Earlier today you mentioned sandboxed builds. How can I configure a build to run sandboxed?
Can you run a git checkout of https://github.com/bhipple/nixpkgs/commit/cc035068f25fd1dc2506b7b038ed2281539450c0, then nix-build -A hits-of-code
, then try that binary? That'll determine whether it's your Arch/Nix installation vs. whether it's a crate2nix issue.
For me the crate2nix version you have as a WIP does not work either.
Your expression works fine. So it seems to be an issue with the expression generated by crate2nix?
Thank you for all the analysis! I just glanced at it but I came by to say that I appreciate that, even though it might take a while until I catch up with it.
BTW, buildRustPackage should be avoided because it does not work well with caching. It is a fixed output derivation that depends on the rust tool chain so even if the tool chain changes, nix will still think that it has the same hash and not rebuild it if it is in the cache.
That creates super annoying extra work for nixpkgs maintainers such as https://gitlab.muc.ccc.de/txpower/nixpkgs/commit/062210bdff8cd2ea3508bc0d7f3341eee913a261
Obviously, that is not satisfying advice if the alternatives are not working for you.
That's the reason why I wanted to use crate2nix
in the first place. I also tried naersk which promises an easy way to build rust packages and even does some caching by building the whole dependency tree first and only then the crate itself. But building a Docker image using the naersk
expression included all dependencies in the image which makes it unusable.
I'll use crate2nix
for my project as soon as this is fixed, for the meantime I'll stick with my Dockerfile
. If you can need any help from a bloody amateur in nix to fix this issue, just hit me up :)
Anyway thanks for your help @bhipple and @kolloch .
@nmattia: Maybe you can help @vbrandl to build slimmer docker images with naersk?
I'll have a look at this at some time as well.
@vbrandl have you see (or were you involved in) https://github.com/nmattia/naersk/pull/74 ? Also, nixpkgs has the great buildImage
which generates very lightweight docker images.
I'll try to get https://github.com/nmattia/naersk/pull/74 in as quickly as possible.
I haven't seen the PR but I saw the corresponding issue (https://github.com/nmattia/naersk/issues/71).
buildImage
doesn't really give me any advantage over a regular Dockerfile
, does it?
Anyway my current setup does work just fine, I just want to give nix a try, so no hurries
@vbrandl FYI https://github.com/nmattia/naersk/issues/71 was closed. buildImage
does give you the advantage that it's not as "linear" as a Dockerfile
(you can import multiple packages without having to specify one command at a time) and it produces very, very tiny images.
Yeah I followed the issue and PRs. I'm currently building my Docker image using naersk and will compare it to the image I got from building the Dockerfile
Edit: The image is actually 12 MB (36 MB vs 48 MB) larger than my multi-stage Dockerfile linking against musl and using alpine in the last stage. While this is an increase by 1/3 in size, it's not really significant for images that small. It might be interesting for a larger project. Also the hand-rolled Dockerfile required quite a bit of optimization and hackery to be that small and cachable, so I guess nix is worth the trade off (also I see myself getting fanboyish hypes from nix :D)
BTW, I'd love musl support
I have a project that depends on OpenSSL (indirectly through
git2
) (repo here). It builds fine but at runtime I get an error from libgit2 which indicates that indicates that OpenSSL is missing at runtime (https://github.com/libgit2/libgit2/issues/4147 or https://github.com/ropensci/git2r/issues/410).Is it possible to define nix packages that are required at runtime when running the binary? Or should this already happen and is a bug? I can try to create a smaller testcase to reproduce the issue.
Update:
Here is a minimal example to reproduce the issue: https://github.com/vbrandl/crate2nix-git2
Cargo.nix
was created using the current HEAD (a76b8d3dfb63271dc82fba1743482ac648ca1f39)Compiling and running directly with
cargo run
works fine on my system.But building the nix package using
nix-build Cargo.nix --attr rootCrate.build
and running./result/bin/crate2nix-git2
gives me the following error: