search

tadfisher / gradle2nix

Generate Nix expressions which build Gradle-based projects.
MIT License
124 stars 59 forks source link

Mismatching hashes and dependency versions for gradle2nix itself #26

Open mschwaig opened 3 years ago

mschwaig commented 3 years ago

I just tried to build gradle2nix with the command suggested in the README.md:

nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz"

which always seems to give me a mismatching hash for different random parts of jackson.

For example with the latest nix docker image:

$ nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz"
waiting for locks or build slots...
hash mismatch in fixed-output derivation '/nix/store/q4apxx0rdz72axxhi7c4gcghsf0vmj3d-jackson-core-2.11.0.rc1.pom':
  wanted: sha256:1jx8piydld4km6rhlxxzwwx70277nr75g7zbvkbg47zd2sm9ljk6
  got:    sha256:182jz7hf4b5ywphziw0647kl7r2y0fm49h9xnpv8w9vrsq27bqbc
cannot build derivation '/nix/store/q987m8dns0cbsrhz47xj1xdxh89kx9ja-jackson-core-2.11.0.rc1.pom.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/f6i3wy1p7q71gy68yb3qkbii2vpiwk5f-gradle2nix-gradle-plugin-env.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8b19gkvn40l7l1valmyjdy89glrbf95c-init.gradle.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/cmnmm9sg2l6d7jwbizj2jvsys2kjpfhh-gradle2nix-1.0.0-rc2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/cmnmm9sg2l6d7jwbizj2jvsys2kjpfhh-gradle2nix-1.0.0-rc2.drv' failed

And with my local machine running unstableNix:

$ nix build -f gradle2nix.nix
warning: unknown setting 'extra-sandbox-paths'
warning: unknown setting 'extra-sandbox-paths'
error: --- Error --- nix-daemon
hash mismatch in fixed-output derivation '/nix/store/rfmm7n5sqi52sbaaklvjqmk3dwjc7z0b-jackson-parent-2.11.pom.drv':
  specified: sha256-wbChPYz+Y7QLD66EWLmUYxae0MYPf8lj7z8FJXFQy+g=
     got:    sha256-YXSwNaEPVu8DtZPoLCjfnlMFVcJRiAFd1x+wu3k65RY=
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/4falvcj3mk9k2fwl1ygikzn4lm0gx9bz-jackson-parent-2.11.pom.drv' failed to build
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/nxlbrgh6k96l24gvmyqp8iq8194xvqhv-gradle2nix-gradle-plugin-env.drv' failed to build
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/043gm5jm164nq82a5fiwljcg0pyxx55w-init.gradle.drv' failed to build
error: --- Error ------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/s05zvjxgzs75gp8nl7syqy9flymm8nm5-gradle2nix-1.0.0-rc2.drv' failed to build

Looking into that a bit I saw that the jackson version in fixtures/basic/basic-kotlin-project/kotlin/build.gradle.kts also does not match the jackson version in gradle-env.json which looks wrong to me and leads me to think the mismatch might simply be because some version change got lost in either one of those files.

I have not looked into this further or actually gotten gradle2nix to build yet.

tadfisher commented 3 years ago

This is strange, I'm not sure why jackson is being pulled in at all. Let me investigate.

mschwaig commented 3 years ago

With the changes I made in #27 I was able to bypass the issue for now and and to then successfully apply gradle2nix to one of Gradle's builtin kotlin hello world projects and package it as a flake. 👍

Thanks for making that possible.

tadfisher commented 3 years ago

Does this work with the latest commit on master? That version of jackson is actually required by a plugin dependency.

stites commented 3 years ago

This is still broken in master, as far as I know, but not with jackson. For me I see this with javax.servlet-api:

✦ ❯ nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz"
warning: unknown setting 'extra-sandbox-paths'                                                                                                           
warning: unknown setting 'extra-sandbox-paths'
error: --- Error --- nix-daemon
hash mismatch in fixed-output derivation '/nix/store/m4mwk5idw1xahkdp5a6k82i4bhrs8nqx-javax.servlet-api-3.1.0.pom.drv':
  specified: sha256-sxEJ4i6j8t8a15VUMucYo13vUK5sGWmANK+ooM+ekGk=
     got:    sha256-b60q4KOgDBRbpXdNZxlmeCkcwgC9Cb0GJu8R/yC03Bo=
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/fbchzvf4413820ymwh4hvjambvr31k48-javax.servlet-api-3.1.0.pom.drv' failed to build
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/7chxzznq0ini5clc4imgxwkkqpgl2jkh-gradle2nix-gradle-project-env.drv' failed to build
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/kgcqdafk9dp0zn9r44l86sfq8vgykh0r-init.gradle.drv' failed to build
error: --- Error --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/a59vn6cradqc9p7zc6cpwzkrxq5iac2n-gradle2nix-1.0.0-rc2.drv' failed to build
mschwaig commented 3 years ago

Looks like I'm seeing the same mismatching hashes as @stites right now. Thanks for confirming that this does not only happen for me.

$ nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz"
warning: unknown setting 'extra-sandbox-paths'
warning: unknown setting 'extra-sandbox-paths'
error: --- Error --- nix-daemon
hash mismatch in fixed-output derivation '/nix/store/ali7yir0h9z7dbsy8xz95ish9vsbbms4-javax.servlet-api-3.1.0.pom.drv':
  specified: sha256-sxEJ4i6j8t8a15VUMucYo13vUK5sGWmANK+ooM+ekGk=
     got:    sha256-b60q4KOgDBRbpXdNZxlmeCkcwgC9Cb0GJu8R/yC03Bo=
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/v6flb992d2qlnc3rxfj0g905xbiys9qd-javax.servlet-api-3.1.0.pom.drv' failed to build
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/pxn87jkhn4y4ry4gyvpg11lz768nc4dj-gradle2nix-gradle-project-env.drv' failed to build
error: --- Error --- nix-daemon
1 dependencies of derivation '/nix/store/lbijb7hr77dm086wckrbwlj7qvasy7fw-init.gradle.drv' failed to build
error: --- Error ------------------------------------------------------------------------------------- nix
1 dependencies of derivation '/nix/store/z1apk82sx0hpp00j6rd2mkq6q8mklqdg-gradle2nix-1.0.0-rc2.drv' failed to build
tadfisher commented 3 years ago

Ah, once I run nix-collect-garbage to clear out my existing dependency derivations, I see this too. Investigating.

stites commented 3 years ago

@tadfisher see #28

mschwaig commented 3 years ago

28 is basically the same thing that I did in #27, but for a different dependency.

Since gradle2nix depends on gradle-env.json and gradle-env.nix it always indirectly depends on a previous version of itself. The hashes inside gradle-env.json naturally are the part of that that can very easily break.

Maybe it would be helpful to have a flag inside gradle-env.nix or somewhere that can disable checking the hashes, so that people can get to a working version of gradle2nix in a first step and then generate working hashes again in a second step in order to bootstrap independently of the provided hashes.

That does not help really fix any specific problem, but I think it would be useful for users to recover from things like this and I think it might have other uses as well.

tadfisher commented 3 years ago

@mschwaig That's not really a viable solution; if gradle2nix generates bad hashes for itself then it will do so for any project you run it on, making it a non-usable tool.

Anyway, what's going on is that we are hashing the artifact in the Gradle cache, which is apparently not the same as the artifact from the source repo. I'm not sure what Gradle is doing to alter these files (perhaps it's regenerating metadata) but the fix is to hash the artifact as we resolve it ourselves. I'll have a PR up shortly.

tadfisher commented 3 years ago

Ah, that file is being served with CRLF (Windows) line separators, and Gradle is saving it to the cache with LF (Unix) line separators.

I've moved hash generation to our own resolvers, but this is extremely slow as we have to redownload every dependency. It might be worth doing this only for POM artifacts just to mitigate this issue, or somehow normalizing POMs at fetch time.

tadfisher commented 3 years ago

This is the root cause, FWIW: https://github.com/gradle/gradle/issues/9072

tadfisher commented 3 years ago

@stites @mschwaig Try it now?

mschwaig commented 3 years ago

gradle2build builds successfully now, thanks.

stites commented 3 years ago

Works for me as well! Thanks!

ghost commented 3 years ago

Still (or again?) having this issue with the following command:

nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz"

The output is the following:

downloading 'https://github.com/tadfisher/gradle2nix/archive/master.tar.gz'
waiting for locks or build slots...
hash mismatch in fixed-output derivation '/nix/store/9w04ibi53kngj4k2fsv0ll0i19j6lnwr-jackson-bom-2.12.0.pom':
  wanted: sha256:082nadlh7mz44qikkm231188isdjm6v7hbxd57b3a5yxdjvdm1iq
  got:    sha256:0f8vg5jnyjcdxmcqm7m3iqn5mk2x08qvnp6iz17ml631pzyv4rzk
cannot build derivation '/nix/store/3pmb6d32z43qm8jbypwv3acnaw059ymm-jackson-bom-2.12.0.pom.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/16676hx9j24xqarfqs2wyk7j2y3bkvq0-gradle2nix-gradle-plugin-env.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/z4cgi26nwqfvd9psdvv6jzjcg61gpvwc-init.gradle.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/k7mc4nm19azwg72zvc7yjppdm57d7d65-gradle2nix-1.0.0-rc2.drv': 1 dependencies couldn't be built
[137 built (1 failed), 0.7 MiB DL]
error: build of '/nix/store/k7mc4nm19azwg72zvc7yjppdm57d7d65-gradle2nix-1.0.0-rc2.drv' failed
jeeger commented 3 years ago

Yep, same problem here: 

building '/nix/store/ypl5afl0g6iadvbhhilnjnb7yx5l7zi2-jackson-annotations-2.12.0.pom.drv'...

trying https://plugins.gradle.org/m2/com/fasterxml/jackson/core/jackson-annotations/2.12.0/jackson-annotations-2.12.0.pom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
direnv: ([/usr/bin/direnv export zsh]) is taking a while to execute. Use CTRL-C to give up.
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  6038  100  6038    0     0  12423      0 --:--:-- --:--:-- --:--:-- 57504
hash mismatch in fixed-output derivation '/nix/store/xwmnfx609qakriq27zvshrgkld9xi6qf-jackson-annotations-2.12.0.pom':
  wanted: sha256:1x8yrrnpfniphk9faylr9c1j69wkz6ff02mzimq6q1myk56hfbm8
  got:    sha256:15h29c2zy764j34x653gn7r56v7sbh4ynqmphay42h6a0cva0l2w
cannot build derivation '/nix/store/cgrzkb1jfcsp48hmcbdkn44ndbrsy0bm-jackson-annotations-2.12.0.pom.drv': 1 dependencies couldn't be built
building '/nix/store/3lx6sklaqvnbifbvx4qcmmmnaggd312a-jackson-base-2.12.0.pom.drv'...
cannot build derivation '/nix/store/2mgylfnqmm7254qji1wfs9cf7iqknbzx-gradle2nix-gradle-plugin-env.drv': 1 dependencies couldn't be built
building '/nix/store/mhmpnhwijkayj6sz0khf9j3i9n46fj1c-javalin-3.12.0.jar.drv'...
cannot build derivation '/nix/store/fvwmp444i6chz0fxi0pd62xp82qcni7m-init.gradle.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/x2x0w5nyi0n0298hgbjj13g4vq4sy529-gradle2nix-1.0.0-rc2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/x2x0w5nyi0n0298hgbjj13g4vq4sy529-gradle2nix-1.0.0-rc2.drv' failed
tadfisher commented 3 years ago

This issue is caused by me; when I hack on gradle2nix, it's pulling hashes from my local gradle cache, which normalizes certain artifacts such as POM files. It's time to add some CI to avoid this.

mschwaig commented 3 years ago

This is the .gitlab-ci.yml file that I used for reproducing this issue inside GitLab CI, in case it is helpful.

image: nixos/nix:latest

default:
  tags:
    - docker-images

stages:
  - test

before_script:
  - nix-channel --add https://nixos.org/channels/nixos-unstable nixpkgs
  - nix-channel --update

dockertest:
  stage: test
  script:
    - nix build -f "https://github.com/tadfisher/gradle2nix/archive/master.tar.gz"

To test for that kind of issue it's actually nice that starts from a docker image with nothing relevant in /nix/store each time.

spencerpogo commented 2 years ago

I'm unable to run this tool due to this error.

❯ nix run github:tadfisher/gradle2nix -- --help
error: hash mismatch in fixed-output derivation '/nix/store/6hhf9myxx3wxl38k89l0psm1p6b7006y-jackson-parent-2.12.pom.drv':
         specified: sha256-YqocFnmt4J8XPb8bbDLTXFXnWAAjj9XkjxOqQzfAh1s=
            got:    sha256-Z749r6r4zhcX56fefbkrpDPfkjtpekDJdDpUzJ/FNjc=
Pythoner6 commented 9 months ago

Hmm, I'm getting the opposite error right now:

error: hash mismatch in fixed-output derivation '/nix/store/a56mz008i9cnfmxrqszlfcvb3ysjq79l-jackson-parent-2.12.pom.drv':
         specified: sha256-Z749r6r4zhcX56fefbkrpDPfkjtpekDJdDpUzJ/FNjc=
            got:    sha256-YqocFnmt4J8XPb8bbDLTXFXnWAAjj9XkjxOqQzfAh1s=