search

Infinidoge / nix-minecraft

An attempt to better support Minecraft-related content for the Nix ecosystem
MIT License
246 stars 26 forks source link

Forge support #15

Open babariviere opened 1 year ago

babariviere commented 1 year ago

Hello, is there any plan to support the Forge loader? Link: https://files.minecraftforge.net/net/minecraftforge/forge/

I am currently trying to fetch the libraries for the Forge loader to potentially add it to this repo. I have this simple script based on the fabric/update.py one.

Source of update.py ```python #!/usr/bin/env nix-shell #!nix-shell -i python3.10 -p python310Packages.requests import json import subprocess import requests from pathlib import Path def versiontuple(v): return tuple(map(int, (v.partition("+")[0].split(".")))) ENDPOINT = "https://raw.githubusercontent.com/PrismLauncher/meta-upstream/master/forge" def get_versions(): print("Fetching all versions") data = requests.get(f"{ENDPOINT}/promotions_slim.json").json() versions = [] for key, loader_version in data["promos"].items(): mc = key.split("-")[0] versions.append((mc, loader_version)) return versions def fetch_version(game_version, loader_version): """ Return the server json for a given game and loader version """ print(f"Fetch {ENDPOINT}/version_manifests/{game_version}-{loader_version}.json") return requests.get( f"{ENDPOINT}/version_manifests/{game_version}-{loader_version}.json" ).json() def gen_locks(version, libraries): """ Return the lock information for a given server json, returned in the format { "mainClass": string, "libraries": [ {"name": string, "url": string, "sha256": string}, ... ] } """ print(version) ret = {"mainClass": version["mainClass"], "libraries": []} for library in version["libraries"]: name, url = library["name"], library["downloads"]["artifact"]["url"] if not name in libraries or any(not v for k, v in libraries[name].items()): print(f"- - - Fetching library {name}") ldir, lname, lversion = name.split(":") lfilename = f"{lname}-{lversion}.zip" # lurl = f"{url}{ldir.replace('.', '/')}/{lname}/{lversion}/{lname}-{lversion}.jar" if url == "": url = f"https://maven.minecraftforge.net/{ldir.replace('.','/')}/{lname}/{lversion}/{lname}-{lversion}.jar" lhash = subprocess.run( ["nix-prefetch-url", url], capture_output=True, encoding="UTF-8" ).stdout.rstrip("\n") libraries[name] = {"name": lfilename, "url": url, "sha256": lhash} else: pass # print(f"- - - Using cached library {name}") ret["libraries"].append(name) return ret def main(versions, libraries, locks, lib_locks): """ Fetch the relevant information and update the lockfiles. `versions` and `libraries` are data from the existing files, while `locks` and `lib_locks` are file objects to be written to """ all_versions = get_versions() print("Starting fetch") try: for (game_version, loader_version) in all_versions: print(f"- Loader: {loader_version}") print(f"- - Game: {game_version}") if not versions.get(loader_version, None): versions[loader_version] = {} if not versions[loader_version].get(game_version, None): data = {} try: data = fetch_version(game_version, loader_version) except json.JSONDecodeError: versions.pop(loader_version, None) continue versions[loader_version][game_version] = gen_locks(data, libraries) else: print(f"- - Game: {game_version}: Already locked") except KeyboardInterrupt: print("Cancelled fetching, writing and exiting") json.dump(versions, locks, indent=2) json.dump(libraries, lib_locks, indent=2) locks.write("\n") lib_locks.write("\n") if __name__ == "__main__": folder = Path(__file__).parent lo = folder / "locks.json" li = folder / "libraries.json" lo.touch() li.touch() with ( open(lo, "r") as locks, open(li, "r") as lib_locks, ): versions = {} if lo.stat().st_size == 0 else json.load(locks) libraries = {} if li.stat().st_size == 0 else json.load(lib_locks) with ( open(lo, "w") as locks, open(li, "w") as lib_locks, ): main(versions, libraries, locks, lib_locks) ```

What I got with running it, after tweaking the loader.nix to change Main-Class, is this:

Error: Could not find or load main class net.minecraftforge.server.ServerMain
Caused by: java.lang.ClassNotFoundException: net.minecraftforge.server.ServerMain

Looking furthermore, it looks like forge doesn't expose the "base" library containing the boot code for the server.

I can try continuing, but I am no Java/Minecraft expert, so I will gladly take any help or a working solution!

Infinidoge commented 1 year ago

The main reason why I haven't added Forge support myself is just that I plain don't use it, but I would certainly accept a PR adding it.

The other reason why was because Forge as a project is... pretty hostile to people getting Forge through anything besides themself, because it would broadly bypass the Adfly links they use, etc.

babariviere commented 1 year ago

I totally understand and agree on the hostility of forge. Unfortunately, some mods don't make the migration to fabric / quilt.

I hope something like patchworks will make it easy to use forge mods.

MagicRB commented 1 year ago

I took a stab at this myself, specifically i wanted to run a modpack from FTB, so that needs modpachs.ch which is a pain in the ass. But anyway that step is fine and can be done through a FOD which is not ideal but what can you do. The part that is problematic is that the Forge installation step is horribly unreproducible so even an FOD wont help here. I wanted to skip that step and defer it to runtime but then I ran into what i believe is a Nix bug so I'm taking a break.

Infinidoge commented 1 year ago

Yeah... it's certainly a pain. Something that is theoretically possible would be to reimplement all of the steps that the Forge installer takes, but that is certainly not an ideal solution.

Faeranne commented 8 months ago

I don't suppose anyone else has tackled this problem recently? I'm attempting to package up a 1.18 server (abandon mods so I can't go with neoforge), and can get everything pulled for proper hashing and pre-setup, except for the mappings file. Forge installer refuses, even with all the offline and debug flags, to keep the existing mapping file, and instead just dies because nix won't let it download stuff. I'm almost tempted to make a script that pre-builds and tars the entire libraries folder, then just shoving that into the nix store manually to pickup in the install phase, but since tar files store user ids and timestamps, it's not guaranteed to be reproducible. So far forge install for server seems reproducible as per sha256 hashes between builds, but I've seen suggestions that that isn't the case?

Infinidoge commented 8 months ago

You could possibly use a Fixed-Output Derivation for that, and it might work. Ideally the installer wouldn't need to be involved at all.

Faeranne commented 7 months ago

I believe (at least for 1.18) the installer patches the official minecraft server to make the final forge jar, so the installer does have to be involved at some point. I'm still relatively new to nix, so I'm not 100% on terminology, but I get the impression my current process of running the installer, then adding the libraries folder to the nix-store is a "Fixed-Point Derivation"? I would make a pr with that involved, but as it turns out, the libraries folder can change if Microsoft decides to randomly change the bindings file that Forge relies on. If the mappings file doesn't change, the the folder has the same hash, but I have already seen 4 different results in the past 2 weeks. And since Forge doesn't seem keen on fixing 1.18.2 with the recent installer patch that makes the bindings something that can be pre-cached, I think this might be a lost cause here. Neo-Forge crew seems mildly on-board with making this work, but Lex has already made it clear that anything proposed on his fork is gonna be turned down outright.

kira-bruneau commented 3 months ago

I recently tried packaging a forge modpack myself and I kept running into same the reproducibility problem with the mapping file: libraries/net/minecraft/server/<version>/server-<version>-srg.jar.

I ran a diff on the files between two successive builds, and only the timestamps appear to differ across the entries of the archive. The unzipped contents are identical. Maybe we could just unzip & rezip it to normalize that file?

How is this mapping file used though? It seems like it'd only be useful in development, maybe we could just remove it - unless it's used for some kind of runtime reflection?

@Faeranne

I'm still relatively new to nix, so I'm not 100% on terminology, but I get the impression my current process of running the installer, then adding the libraries folder to the nix-store is a "Fixed-Point Derivation"?

A fixed output derivation is just a derivation that constructs an output directly using the hash of its contents rather than with a hash of its inputs. Here's an example using the forge installer:

{ pkgs, ... }:

{
  forge = pkgs.runCommand "forge-1.16.5-36.2.34"
    {
      nativeBuildInputs = with pkgs; [
        cacert
        curl
        jre_headless
      ];

      outputHashMode = "recursive";

      # This hash is not reprodubile
      outputHash = "sha256-jrDk5/ZVilMZukhhHjksUHGGISOHjWyuJSPWNJuhVpM=";
    }
    ''
      mkdir -p "$out"
      curl https://maven.minecraftforge.net/net/minecraftforge/forge/1.16.5-36.2.34/forge-1.16.5-36.2.34-installer.jar -o ./installer.jar
      java -jar ./installer.jar --installServer "$out"
    '';
}

When you use a fixed output derivation, it loosens the build sandbox to allow networking. It can do this because a successful build will always be guaranteed to have the same hash.

What you're doing already is kind of similar to what a fixed output derivation can do, but doesn't actually use a derivation to automate the process.

kira-bruneau commented 3 months ago

Ooooo!! I was able to make it reproducible using strip-nondeterminism:

{ pkgs, ... }:

{
  forge = pkgs.runCommand "forge-1.16.5-36.2.34"
    {
      nativeBuildInputs = with pkgs; [
        cacert
        curl
        jre_headless
        strip-nondeterminism
      ];

      outputHashMode = "recursive";
      outputHash = "sha256-jccxyIEU6KZGOQpLi6zf5rBXzFQ76mXdb9+cLTNLkVo=";
    }
    ''
      mkdir -p "$out"
      curl https://maven.minecraftforge.net/net/minecraftforge/forge/1.16.5-36.2.34/forge-1.16.5-36.2.34-installer.jar -o ./installer.jar
      java -jar ./installer.jar --installServer "$out"
      strip-nondeterminism --type jar "$out/libraries/net/minecraft/server/1.16.5-20210115.111550/server-1.16.5-20210115.111550-srg.jar"
    '';
}

Now would just need to generalize this across all forge installer versions.

Infinidoge commented 3 months ago

I don't like it because it feels icky and brittle to my Nix perfectionism, but if it works it works! Certainly better than it not working at all.

Would definitely welcome a PR, even if it just packages the recommended Forge version for each major version. A better and more complete packaging effort for snapshots and whatnot can come later.

Faeranne commented 3 months ago

(I have since spent a shocking amount of time leanring what I can about nix, and yet still stumble on terms, fyi.) Just want to point out that this will require --impure when using flakes, and breaks hermetic seal due to the network calls, which is generally considered unacceptable in nixos.

Having talked with a few others about this, the consensus I've heard is that since Forge does not provide a way to make the calls deterministic, either a systemd unit or an activation script is prefered for nixos, while non-nixos builds are just kinda out of luck. not as elegant or as nix-like, but apparently files potentially being different inside the store is considered far worse.

It might make sense long term to look into building the jar from source using gradle instead of relying on a broken installer. (from source is also preferred in general in nix). I don't have a current need for Forge, so I'll leave it to others for the time being to figure out what that takes.

Infinidoge commented 3 months ago

This wouldn't require --impure because it is a Fixed-Output Derivation. You also have to explicitly opt-into breaking the networking sandbox per-derivation, last I checked, which doesn't apply here.

It's hacky, and has a high chance of having other non-reproducible parts in it, but it isn't impure.

Faeranne commented 3 months ago

ah, nix's convoluted error messages caught me again. I had attempted to use this myself just now, and ended up with an error that read similar to the network error when doing non-fixed output derivations. what actually happened is that the output is apparently different on my system, resulting in a different hash. Best guess is the CDN issue I was running into is still alive. Attempting to correct the hash locally and then re-deploy this to one of my servers still results in a different hash.

Infinidoge commented 3 months ago

I imagine it isn't truly reproducible, which is definitely an issue.

MagicRB commented 3 months ago

The way i ended up doing this for enigmatica 6 is running the installer fully in the runit run script (service start for systemd jargon), not ideal but works. And having the hashes change broke so mamy things for my infra this was the only way.

kira-bruneau commented 3 months ago

@Faeranne Were you trying with the same version of forge? (forge-1.16.5-36.2.34) I just tried again and I'm still getting sha256-jccxyIEU6KZGOQpLi6zf5rBXzFQ76mXdb9+cLTNLkVo= as the output hash. If so, I'm curious where the diff is then.

pedorich-n commented 3 months ago

FYI: the newer forge version produces a differently structured output and, in my case, I didn't need the strip-nondeterminism at all. Thanks, @kira-bruneau for the code!

Server 1.20.1-47.3.1 ```nix { pkgs, ... }: let minecraftVersion = "1.20.1"; forgeVersion = "47.3.1"; version = "${minecraftVersion}-${forgeVersion}"; in pkgs.runCommandNoCC "forge-${version}" { inherit version; nativeBuildInputs = with pkgs; [ cacert curl jre_headless ]; outputHashMode = "recursive"; outputHash = "sha256-DRzLUVL56wnl2SBemSmXCYtHysI42yYB8WF7GEFnMjA="; } '' mkdir -p "$out" curl https://maven.minecraftforge.net/net/minecraftforge/forge/${version}/forge-${version}-installer.jar -o ./installer.jar java -jar ./installer.jar --installServer "$out" '' ```

This new derivation produces the following directory structure:

/nix/store/8kq0w5w8pqg1i4lhb8v34f4xynmmw94q-forge-1.20.1-47.3.1
├── libraries
├── run.bat
├── run.sh
└── user_jvm_args.txt

Where run.sh is:

java @user_jvm_args.txt @libraries/net/minecraftforge/forge/1.20.1-47.3.1/unix_args.txt "$@"

and unix_args.txt contains a bunch of references to relative libraries and launches cpw.mods.bootstraplauncher.BootstrapLauncher. Because of these relative paths, it doesn't work out of the box, but I was able to replace them with full ones to make it work with nix-minecraft and turn this run.sh into another Nix derivation.

Runnable server ```nix { pkgs, forge, ... }: pkgs.stdenvNoCC.mkDerivation { pname = "minecraft-server"; version = "forge-${forge.version}"; meta.mainProgram = "server"; dontUnpack = true; dontConfigure = true; # TODO: don't place the file in /bin buildPhase = '' mkdir -p $out/bin cp "${forge}/libraries/net/minecraftforge/forge/${forge.version}/unix_args.txt" "$out/bin/unix_args.txt" ''; installPhase = '' cat <<\EOF >>$out/bin/server ${pkgs.jre_headless}/bin/java "$@" "@${builtins.placeholder "out"}/bin/unix_args.txt" nogui EOF chmod +x $out/bin/server ''; fixupPhase = '' substituteInPlace $out/bin/unix_args.txt \ --replace-fail "libraries" "${forge}/libraries" ''; } ```

It feels very brittle but works with this specific combination of versions. Maybe we could come up with a better way?

Faeranne commented 3 months ago

@kira-bruneau I was using 1.18.2, so def a different version. I did some tracing and the issue was apparently that the file had not propagated across the cdn fully (guess they modified the 1.18 mapping recently)? I now get the same results regardless of location, but this does confirm they keep updating mapping periodically.

@pedorich-n 1.20 also supports using cached files completely, so everything can be statically pulled ahead of time and the Fixed Output Derivation is no longer needed. They just never backported the installer, so previous versions still ignore the caching flag specifically for the mappings.

Infinidoge commented 3 months ago

Every time I think about this issue I always wish Forge had a proper API that listed these things like Fabric does. It would make things so much easier.

Faeranne commented 3 months ago

actually, thinking on it, the project is lgpl, so it might make sense to make a fork of just the installer with the caching patch backported. Doing that would reduce any maintenance to just bug fixes, instead of chasing any hash changes, since forge does not update versions pre-1.19.2 (and soon pre-1.20). Post-1.19.2 has the installer cache patch added, so it should all be rock solid.

Faeranne commented 3 months ago

I may have found a workaround. It looks like the installer uses an "installer_profile.json" file to decide what to download, and that is standard since version 2.0 of the installer. this file even includes file hashes, so in theory this could be used for fetching the needed files ahead of time, and then use the latest 2.0 installer to do the actual "install" with the offline flag.

Faeranne commented 3 months ago

Alright, I got it working in theory. The main culprit was the DOWNLOAD_MOJMAPS task, which up until recently completely ignored the --offline flag. I have now found a way to remove that, and to gather everything into a predictable format.

The only outstanding issue is that 1.18.2 and before use a manifest.mf file in the installer, making patching it externally tricky, and would require multiple steps to extract the required installer_profile.json and versions.json files, patch the installer_profile.json to remove the download task, use those to fetch the files by hash, and then follow up with re-injecting those into the installer jar and do the final execute. This naturally causes an Import From Derivation case, which while hermetic, is not considered good, right?

Do we provide the installer_profile.json and versions.json here (and possibly the installer.jar for convience, since it is lgpl licensed?), or do we go with the IFD case?

The only other thing is that the version json files from mojang are not stable, and are the key source of the issue (they sometimes update the mapping files? Cant tell why, and I'm not enough of a java expert to understand what is changing). Do we consider that safe to store here, or do we extract the pieces we need and re-store them in a new format (we just need the libraries and server.jar and server-mappings.txt paths.)

Faeranne commented 3 months ago

Ok, nvm. I was moving a mile a minute and missed that the version json is locked, if you grab it from the per-defined piston path, and not from the base versions.json resource. That resolves that issue, and makes the process pretty straightforward. gonna attempt a pr for 1.18.2 - 40.2.21 as a test. If all goes well, we should be able to generate the version json files needed for everything pretty easily.

Faeranne commented 3 months ago

I need to call it a night, but there is a bunch of notes in commit f4e4514 on my fork. I can open a pull request to start tracking that if desired, or finish it up tomorrow and open the pr then.

Some notes to keep here just in-case.

Forge uses 3 formats for distribution:

Right now I'm just working on update.py to try and make some useful lock files, and since it really needs all the libraries for forge to be happy, I'm just going ahead and grabbing the minecraft libraries at the same time and locking them. That probably should happen in build-support, but the library paths matter, and I am not sure what the normal process for converting the name into a path is, so I'm tossing the provided path into the lock files too, since that seems guaranteed to work. If there's a preferred way to gather the libraries that can be used for the install to, please feel free to point me in that direction and I'll make it happen! :)

Piecuuu commented 1 month ago

The other reason why was because Forge as a project is... pretty hostile to people getting Forge through anything besides themself, because it would broadly bypass the Adfly links they use, etc.

Right, then, I propose adding NeoForged instead of Forge. It's a fork of MinecraftForge. More on that here.

Infinidoge commented 1 month ago

The biggest reason why I want Forge support personally is for hosting servers for older packs, which NeoForge doesn't help with. I do agree that NeoForge would be good to have for newer versions instead of Forge, though.

Faeranne commented 1 month ago

Since neoforge is a fork, it should be a few extra lines to enable, once forge is done. Main thing has been sorting between all the different install methods forge has used, and sorting out the script to build the modern forge instances, which neoforge also uses.

Szymzal commented 1 month ago

Hi, I tried also to get working Forge. I did make working one specific version of Forge: 1.20.1-47.3.1. I used some of @Faeranne work.

If someone wants to look and make use of it: https://github.com/Szymzal/dotfiles/tree/ffeaf5fcb6229ecdeb412f51f05d0707acab2da2/pkgs/forge-servers

The only thing is that there is no automatic lock generation, so you would need to create manually one to include different version of Forge

Faeranne commented 1 month ago

yeah, the automatic lock has been a nightmare, and may be something that has to be dropped for Forge, due to it's rather inconsistent history.