Feature request: provide `nixd` package that works with stable `nix`

hab25 commented 1 year ago

Please provide a nixd version that links against nixVersions.stable libraries so that users can keep using stable nix while using nixd and still enjoying accurate language service.

Since nixd's functionality is closely tied to nix, providing an attribute set structured similarly to nixVersions is probably a good idea, i.e., the attribute pkgs.nixd should link against pkgs.nix and pkgs should have a new attribute

nixdForNixVersions = { 
  "2_15" = nixdDerivationThatLinksAgainstNix_2_15; 
  "2_16" = nixdDerivationThatLinksAgainstNix_2_16;
};

(this paragraph is for the context of nixpkgs; the equivalent for nixd's flake.nix is trivially similar).

Given that providing these implies testing them (even if manually), if doing this for current nixVersions.stable (i.e. nix_2_15) would be too much work, delaying it until nixVersions.stable == nix_2_16 is an option.

If more context is needed:

There are vaild reasons to prefer stable, here's an example problem that affected unstable users but not stable users https://github.com/nix-community/home-manager/issues/3734#issuecomment-1458999735 .

Note that all of the nixpkgs hyperlinks in this post are pinned to a single very-recent commit of branch nixos-unstable.

nixVersions.stable is especially important because nix == nixVersions.stable; see https://github.com/NixOS/nixpkgs/blob/684c17c429c42515bafb3ad775d2a710947f3d67/pkgs/top-level/all-packages.nix#L39892

nixd is using nixVersions.unstable, i.e.,nix_2_16; see https://github.com/NixOS/nixpkgs/blob/684c17c429c42515bafb3ad775d2a710947f3d67/pkgs/tools/package-management/nix/default.nix#L208

This is true for both:

nixpkgs; see https://github.com/NixOS/nixpkgs/blob/684c17c429c42515bafb3ad775d2a710947f3d67/pkgs/top-level/all-packages.nix#L18122
nixd's flake.nix; see https://github.com/nix-community/nixd/blob/b835f4e677e2daf27239b8cf3535beeba2625ed9/flake.nix#L15

hab25 commented 1 year ago

It would also be nice if the nix of the contemporary stable nixos branch could be supported moving forward. E.g., currently, the contemporary stable nixos branch is nixos-23.05 and the nix of that is on version 2.13.3.

inclyc commented 1 year ago

Does nixd actually conflict with stable versions?

Please provide a nixd version that links against nixVersions.stable libraries so that users can keep using stable nix while using nixd and still enjoying accurate language service.

Cannot nixd users use the stable version of nix at the same time? e.g.

{
  environmentPackages = with pkgs; [
    nix
    nixd
  ];
}

I have asked some question at the Nix Language matrix room. Looks like the databases in nix is backward-compatible.

This is non-trivial

The C++ libraries provided by NixOS/nix are not ABI-compatible, which means you need to maintain forks of these versions, not in a single repository. Due to this reason, there is no trivial way to provide a series of nixd versions linking with the nix library.

hab25 commented 1 year ago

I expressed my original concern poorly, this is the key point:

If nixd is linking against a nix version that is not the same as the user's, the eval-related features won't properly emulate the user's nix.

nixd's README.md also values this:

https://github.com/nix-community/nixd/blob/b835f4e677e2daf27239b8cf3535beeba2625ed9/README.md?plain=1#L57

This was key word in my original post:

and still enjoying accurate language service

Examples where this can be problematic for a `nixVersions.stable` user

Example: I was using
```
assert builtins.nixVersion == "2.15.1"; /* ... */
```
in my nix file [^1]. nixd, which links against nix_2_16, signaled an eval error due to the failed assertion and was unable to provide its more powerful eval-related features. However, my nix evaluated the file successfully.

[^1]: Why? Because I'm using flakes, which is an experimental feature, and, from https://github.com/NixOS/nix/blob/85d0eb63165e7d7f441fe3dd94bb548a40502e52/doc/manual/src/contributing/experimental-features.md?plain=1#L5, "Experimental features are considered unstable, which means that they can be changed or removed at any time." The assert helps me prevent causing breakage related to this when running e.g. nix build --recreate-lock-file

If the previous example is too contrived (even though it is a real story), another example, from the 2.16 release notes:

The function builtins.replaceStrings is now lazy in the value of its second argument to. That is, to is only evaluated when its corresponding pattern in from is matched in the string s.

So if to throws, nixd might not signal an error in situations where the user's nix_2_15 would.

Addressing some of your points

Does nixd actually conflict with stable versions?

Cannot nixd users use the stable version of nix at the same time?

Looks like the databases in nix is backward-compatible

From the third quote (good to know; thanks), I'm guessing that it probably does not conflict and users can use both at the same time, but this was not my original concern.
This is non-trivial

If it is too much work maintaining nixd against multiple nix versions, is nixVersions.unstable the best target? I assume that nixd also likes some stability given that that it is not following master: https://github.com/nix-community/nixd/blob/b835f4e677e2daf27239b8cf3535beeba2625ed9/flake.nix#L3

(Of course this is for you to decide, but my preference would be what I mentioned in my previous comment:

the nix of the contemporary stable nixos branch

I recently encountered some segfaults and other errors with nix 2.15.1 and switching to 2.13.3 solved them all)
which means you need to maintain forks of these versions, not in a single repository

I've never done any serious C++ development, and have never worked with meson, but I don't understand this need for forks. I think you could easily refer to different versions by pinning each version and mapping it to a separate output, no? E.g. with flakes:
```
{
inputs = {
  nix-2-15-1 = "github:nixos/nix?ref=2.15.1";  
  nix-2-16-0 = "github:nixos/nix?ref=2.16.0";
};
outputs = /* here you apply any patches and then pass the appropriate store paths to your multiple meson builds */
}
```

Tangent:

The C++ libraries provided by NixOS/nix are not ABI-compatible,

Since it is not ABI compatible, I would guess that the library maintainers don't attempt to keep their APIs stable either; is this guess correct? If so, then nixd will have to deal with potential breakages on every nix upgrade, correct?

Maybe this is why we don't see language servers for Rust taking the approach that clangd, ccls and nixd have taken of linking directly against language libraries: Rust is under heavy development and is a large language. Comparatively, Nix is a much smaller language and I expect that C/C++ are ABI compatible and/or have a much lower rate of change than Rust.

inclyc commented 1 year ago

Examples where this can be problematic

I agree that this is a good point. I'm concerning about it may require a lot of work to resolve the difference, see comments below.

About the implementation details

is nixVersions.unstable the best target?

Actually, no. I pinned nix version to 2.16, and an explicit PR is required for bumping nix version. For example: https://github.com/nix-community/nixd/pull/86

I think you could easily refer to different versions by pinning each version and mapping it to a separate output, no?

No. See the changed files in #86, we need to change the source code between nix versions.

Since it is not ABI compatible, I would guess that the library maintainers don't attempt to keep their APIs stable either; is this guess correct?

Yes.

If so, then nixd will have to deal with potential breakages on every nix upgrade, correct?

Yes. As mentioned above, #86 is a good example for "deal with potential breakages".

Maybe this is why we don't see language servers for Rust taking the approach that clangd

No, the llvm C++ API/ABI is also unstable. But the clang authors explicitly designed clang for reusability. This is not true for Nix/Rust.

hab25 commented 1 year ago

Thank you for all the explanations.

I pinned nix version to 2.16

What will be the strategy for deciding when to bump the nix version will be moving forward? (it may be worthwhile to document this)

Also, feel free to close this issue, e.g. as not planned.

inclyc commented 1 year ago

What will be the strategy for deciding when to bump the nix version will be moving forward? (it may be worthwhile to document this.

There is currently no clear "strategy" for now. It might be updated according to the official release cycle, or maintain branches for different versions. This is much similar to Minecraft mods[^mods] :).

Also, feel free to close this issue, e.g. as not planned.

Ask questions anytime!

[^mods]: Mojang's Minecraft is not designed for "modding", so mod authors must maintain a series of version targeting specific minecraft version.

eli-schwartz commented 1 year ago

The C++ libraries provided by NixOS/nix are not ABI-compatible, which means you need to maintain forks of these versions, not in a single repository. Due to this reason, there is no trivial way to provide a series of nixd versions linking with the nix library.

No. See the changed files in #86, we need to change the source code between nix versions.

This is pretty simple to deal with and many projects do so. Simply keep both versions of the source code and gate them behind #ifdef macros generated in the buildsystem setup depending on the version of the nix dependency that was found.

If you choose to do this you'll have to maintain both branches of source code when adding new features that touch on the code in question, but that's significantly less complicated than maintaining both branches except in a fork.

nix-community / nixd