Packaging Gradle-based applications

[bobvanderlinden]

Issue description

I have a (closed) application that I'd like to package and place on my NixOS server. The application can be built using Gradle, however since Gradle downloads its dependencies online it is unable to be build using Nix as-is.

It seemed all application that are based on Java/Ant/Maven/Gradle are being packaged based on binaries, instead of the source. Is this the recommended way to use Java-based applications on Nix?

While trying to find a solution, I have found mvn2nix-maven-plugin, but it was unclear how to use it for other projects. Especially the ones built with Gradle.

[spacekitteh]

Seconding this. I'd really like to figure this out.

[NeQuissimus]

There is #20444 :-)

[spacekitteh]

Aye, but Gradle is needed for Android apps

[auntieNeo]

I am especially interested in building Android apps with Gradle. Recent apps created with Android studio no longer work with ant.

I wonder if androidenv can be extended to work with the Gradle wrapper. Or maybe that's not a viable approach.

[bobvanderlinden]

Gradle 4.8 was just released. It includes locking of versions, which might be useful for Nix:

https://docs.gradle.org/current/release-notes.html#locking-of-dynamic-dependencies

[tadfisher]

@bobvanderlinden Very cool, a gradle2nix tool is feasible now.

[bobvanderlinden]

The lock file only includes the groupId, artifactId and the version. See https://docs.gradle.org/current/userguide/dependency_locking.html#_lock_state_location_and_format

We still need a way to know the URLs for these packages.

[tadfisher]

@bobvanderlinden That's fine. What I'd do is make a Gradle plugin that emits a Nix configuration for a task, and apply it using gradle --init-script. This would actually work without dependency locking.

[bobvanderlinden]

Oh wow. I didn't know Gradle could do that. Nice.

https://docs.gradle.org/current/userguide/init_scripts.html#sec:basic_usage

Register build listeners. External tools that wish to listen to Gradle events might find this useful.

Though I cannot find how to hook into the downloader.

Another useful plugin is one that downloads all dependencies:

https://github.com/mdietrichstein/gradle-offline-dependencies-plugin

[lheckemann]

https://docs.gradle.org/4.8/release-notes.html#locking-of-dynamic-dependencies Fixed link for earlier comment

[tadfisher]

So there's a couple of approaches we can take with this.

We could write a gradle2nix tool which simply outputs the set of artifacts including URLs and hashes. I've started on a plugin that can be applied from an init script, and it can successfully dump artifact information (groupId, artifactId, version, url, sha256) from a Gradle configuration. However, this is only easily doable for Maven repositories; dumping artifacts fetched from Ivy repositories require using internal APIs which undergo lots of churn. This comes down to the fact that Ivy repos do not expose their artifactPatterns, ivyPatterns, and m2compatible fields, which is easily worked around but makes me uneasy.

Another issue with a Gradle plugin solution is that Gradle does not expose any information about dependency resolution, at least not without extreme hacking. The naive implementation, then, would provide a set of source URLs for each artifact, as we have no way of knowing which repository Gradle actually fetched an artifact from without using internal APIs. This is all doable, but ugly. Note that the gradle-offline-dependencies-plugin linked earlier uses said internal APIs only to fetch artifacts into a local repository, and still must use internal APIs to do so (and broke in 4.9); it does not attempt to determine the source URLs/patterns for repositories or artifacts.

So I've used another approach recently in https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/security/jd-gui/default.nix, which uses Gradle itself to fetch dependencies, then copies those dependencies from its cache into an offline Maven repository. The entire repository is hashed. This was actually pulled from the mucommander derivation which applied this hack a long time ago to package an application built with Gradle.

This approach is far from pure, however, as it cannot be used with sandboxing. So I propose creating a fetchGradleDependencies function which can be used in a srcs attribute, and a buildGradlePackage function which wraps mkDerivation and appends a fetchGradleDependencies call to buildInputs. This should let us fetch and create the offline repository in sandbox builds.

[deliciouslytyped]

Any news here? Edit: continued: Ghidra will probably use gradle for it's build system if and when it gets released. At the least, it seems to be used for building extensions (and basically the entire app is a framework of extensions).

[tadfisher]

Here's a POC for a gradle2nix tool: https://github.com/tadfisher/gradle2nix

It should be able to build itself with the included default.nix. Use it with gradle2nix in a project root.

There are a few pitfalls that I'm working out:

• It builds against Gradle 5.3, but there are a couple of APIs that it uses that aren't available on older Gradle versions, so it only supports Gradle 5.3+ right now.
• It only supports Maven dependencies. Ivy support is slightly less trivial but I will add it after writing tests.
• It resolves plugin marker dependencies and plugin repositories with internal APIs, which might be unavoidable.
• It's currently a gradle plugin in order to resolve plugins. However, this requires application via an init script. It could potentially resolve plugins as a settings plugin, and the project dependency resolution could be split out into a project plugin.
• I haven't looked into resolving buildSrc dependencies or overriding repos for buildSrc projects.

[tadfisher]

I've reworked gradle2nix to use the Gradle Tooling API, so the implementation is a bit cleaner.

• It's now compatible with Gradle 4.4 and up, with a test suite for regressions.
• I've added some rudimentary support for included builds and buildSrc, but I haven't gotten around to testing it.

[deliciouslytyped]

@tadfisher This looks pretty good, I'm not intimately familiar with gradle, how do I run this if I want to package something?

[tadfisher]

You should be able to build the gradle2nix application using the included default.nix script, then just run [result]/bin/gradle2nix in a Gradle project directory. This will generate a gradle-env.json and gradle-env.nix pair. You can then import the Nix expression and use it to build the project. See default.nix in the gradle2nix project itself for an example.

[NorfairKing]

+1

[deliciouslytyped]

For later reference, here is an example of something using Gradle without any extra fancy tooling (I think) https://github.com/NixOS/nixpkgs/pull/72306

[tadfisher]

@deliciouslytyped Unfortunately, that solution will break whenever one of its SNAPSHOT dependencies publishes a new version. In general, you can't make a fixed-output derivation from the dependencies fetched for a build, because Gradle allows dynamic dependency versions and can resolve different artifacts from run to run. Therefore, you need tooling to extract the resolved artifact sources in order to make the build reproducible.

[ghost]

I made an attempt to use gradle2nix in nixpkgs: #77422

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[NorfairKing]

"still important to me"

[stale[bot]]

I marked this as stale due to inactivity. → More info

[doronbehar]

Still important.

On Jan 4 2021, at 10:27 pm, stale[bot] notifications@github.com wrote:

I marked this as stale due to inactivity. → More info (https://link.getmailspring.com/link/6FC0A521-CE59-4179-8F0A-83E5AD7B5045@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2FNixOS%2Fnixpkgs%2Fblob%2Fmaster%2F.github%2FSTALE-BOT.md&recipient=cmVwbHkrQUNUNUlNNFBKV1RDUFNTVzZDU1U3VE41NzVOVDNFVkJOSEhBVUJKNlRNQHJlcGx5LmdpdGh1Yi5jb20%3D) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub (https://link.getmailspring.com/link/6FC0A521-CE59-4179-8F0A-83E5AD7B5045@getmailspring.com/1?redirect=https%3A%2F%2Fgithub.com%2FNixOS%2Fnixpkgs%2Fissues%2F17342%23issuecomment-754199590&recipient=cmVwbHkrQUNUNUlNNFBKV1RDUFNTVzZDU1U3VE41NzVOVDNFVkJOSEhBVUJKNlRNQHJlcGx5LmdpdGh1Yi5jb20%3D), or unsubscribe (https://link.getmailspring.com/link/6FC0A521-CE59-4179-8F0A-83E5AD7B5045@getmailspring.com/2?redirect=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACT5IMYOTNU5REXC4TIOX53SYIQD3ANCNFSM4CK5GFQA&recipient=cmVwbHkrQUNUNUlNNFBKV1RDUFNTVzZDU1U3VE41NzVOVDNFVkJOSEhBVUJKNlRNQHJlcGx5LmdpdGh1Yi5jb20%3D).

[gytis-ivaskevicius]

Welp, this is the moment I have been training my whole life. Ahem... It is still important!!

Edit: Apparently I am blind, either that or I do not understand meaning of the word "removed". Sorry for ping

[randomnetcat]

Unfortunately, that solution will break whenever one of its SNAPSHOT dependencies publishes a new version. In general, you can't make a fixed-output derivation from the dependencies fetched for a build, because Gradle allows dynamic dependency versions and can resolve different artifacts from run to run. Therefore, you need tooling to extract the resolved artifact sources in order to make the build reproducible.

As of Gradle 6.2, Gradle can verify hashes and signatures of artifacts and metadata. See https://docs.gradle.org/current/userguide/dependency_verification.html

[bobvanderlinden]

Some updates:

https://github.com/NixOS/nixpkgs/pull/272380 is an effort to have a setup hook available for gradle projects in nixpkgs.

https://github.com/NixOS/GSoC/issues/19 is a project proposal for Google Summer of Code to create better gradle tooling.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/buildgradleapplication-a-simpler-approach-to-packaging-gradle-projects/46012/3

[tadfisher]

gradle2nix v2 is available for testing, with a novel approach to extracting artifact information by leveraging Gradle's module cache. I have successfully compiled Kotlin from source with it, which is no small feat. Please give it a try, and I intend to contribute this to nixpkgs if all goes well.