Allow flakes to refer to other flakes by relative path

[edolstra]

E.g.

{
  inputs.foo = { url = ./../foo; };
}

This is primarily useful when you have multiple flakes in the same repository.

However this means that we can't have a lock file entry for the foo input.

[roberth]

A lock file shouldn't be necessary as long as the input is going to be committed to the same repository.

It only becomes a problem when the reference points outside the repository. Doing is useful for testing a change in a dependency, but a warning seems appropriate.

[zimbatm]

It would be good to disallow .. in the relative paths to encourage composing things reliably.

[kloenk]

Can we only do one lockfile for this, like cargo workspaces does it?

[roberth]

@zimbatm Disallowing .. in relative paths forces the flakes to form a tree rather than a DAG. Suppose you have a repo with flakes with dependencies A -> P -> X and A -> Q -> X, you'd have to duplicate flake X for flakes P and Q. .. is also useful for ad-hoc testing. For example, you may want to use this feature temporarily to iterate on a dependency that's outside the repo.

@Kloenk I think you get half of this behavior for free if you don't invoke nix flake commands on the dependency flakes. The other half is that if you do invoke it, it can use the parent's lockfile. In cargo this seems to be implicit. Another example is Maven, where you reference the parent pom explicitly. If we require an explicit reference to the parent flake, we preserve the ability to opt out of the single lockfile, which can be important for monorepos. I also like that it reinforces that the flake isn't quite independent. For example, you can't just copy its files and expect nix flake commands to work the same outside the context of the parent. This is true whether the parent reference is explicit or not, but if it's explicit, you'll get an error message to "manage" that expectation; a useful message that's easy to resolve.

[stephank]

So, I’m considering if this is something I want/need. In my case, I have a repository of nixosConfiguration flakes in subdirectories (each a project with one or more machines), and a shared lib. There’re currently hacks in place to make the lib stuff work, because it technically lives outside any of the flake contexts.

• Workspaces sound like a different piece of functionality (pinning all flake dependencies as a whole), and in my case, I want separate lockfiles (per project).

• At the same time, workspaces could be a more sensible way to declare a toplevel than ‘discover git/vcs root’, if we want logic to prevent/warn escaping a certain context.

• My current setup would require .. to work, because lib is outside the flake context.

[MagicRB]

This can sort of be simulated with https://github.com/edolstra/flake-compat, you must make your build impure, but even though it sounds bad, it isn't that bad. If you know your build is pure, you're fine, if you aren't sure, you can temporarily disable the sub-flakes and test each one separately build nix build-ing it. To get this to work with nixos-rebuild you have to edit the script and add "--impure" to extraBuildFlags. Or add an option into the switch

oh and all the sub-flakes have their own lock-files, therefore its necessary to update them recursively, I'll write a script for that.

[MagicRB]

@stephank

* My current setup would require `..` to work, because `lib` is outside the flake context.

perhaps something like this could work, though I have no clue how to actually implement it

{
  inputs = {
    your-lib = {
      path = ./your-lib; # different from url to distinguish it
    };

    machine-1 = {
      path = ./machine-1;
      pass-on = your-lib; # I don't really know what `your-lib` should be, it could be a string, but that could cause confusion as registry references look the same. Also, I already saw something like this somewhere, can't remember what was the option, if you know then please tell me.
    };
  };
}

{
  inputs = {
    your-lib = {
      follows = "your-lib"; # this is the option used for referencing flake registries, I'm sure this time
    };
  };
}

[jaen]

I just want to add another two cents to the bandwagon – I have two use cases that keep me wanting to reach for flakes with a relative path (and possibly some kind of arguments, but that's a separate discussion).

One is the already mentioned idea to factor out common code into a lib flake shared across some nixosConfiguration flakes and possibly version things like nixpkgs on a per-configuration basis. Another is using nix to build a multi-language repo I have for work, where each sub-project could be a flake I then bring together.

I'm not sure if it's the right instinct to try reaching out for things like that, as I've been using NixOS mostly for Terraform/Ansible angle of reproducible configuration and I'm not sure how that fits in the package building conventions Nix has – maybe there are other options that are better suited to those use-cases I'm not aware of.

[MagicRB]

I'd add that sub-flakes are a useful tool to "partition" your repo, for example my dotfiles (EDIT: I moved those packages to my systems repo) themselves provide custom packages I created and I propagate those up from "half-flakes", flakes whose inputs are declared at the toplevel flake, but look like separate flakes and mostly behave the same way. Just call the outputs lambda with the unified inputs and self

[jaen]

Another semi-related thing I've bumped against just now is that after a few tries to use things like submodules or impure imports via flake-compat (dunno if that's kosher, like I'm said, I'm new at this) I bumped into enough issues that I decided to just split it out as separate flakes.

Unfortunately this now incurs the change->commit->push->update-flake penalty to make a simple change which is kind of annoying, is there plans for some way to develop flakes alongside a la yarn link (basically, substitutes a package with a local copy, so the changes you make are visible to the package that depends on it) for flakes. If not, would it make sense to open an issue for it?

[MagicRB]

impure imports via flake-compat (dunno if that's kosher, like I'm said, I'm new at this)

definitely not

Unfortunately this now incurs the change->commit->push->update-flake penalty

look at my dotfiles, I had the same issue and I solved it by those half-flakes

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/5

[jorsn]

Currently, relative paths work already with the path fetcher. However, this comment looks like the restrictions are just not implemented. flake-compat doesn't allow relative paths, because builtins.path needs an absolute path string or a path (e.g. ./. + "/lkjsfl") as attribute path in the argument set.

Edit: What is the problem with a lock file?

[BBBSnowball]

basically, substitutes a package with a local copy, so the changes you make are visible to the package that depends on it

nix flake update --override-input ... might do what you want. If you use it with path:/something, it will record the hash of that path so you have to run the command again after changing any file in /something. This should remove the "commit->push" part but you still have to update the flake.

This was still too much overhead in my edit-test-loop. Therefore, I have merged most of my repos and use a similar approach as MagicRB.

I hope relative paths with ".." will remain possible for path: and I will switch to plain .. if this becomes possible. I use (misuse?) them for these purposes:

1. I have a repo with modules and configurations (nixosConfigurations) for several hosts. Later, I wanted to change the nixpkgs input for one of them. I did that with a sub-flake (in the same repo) that uses the main flake via path:../... The sub-flake has its own lockfile.
2. A friend wanted to use a package from my flake. He is using nixos-unstable while I'm forcing a specific version of nixos-20.09 in the flake. Therefore, any package from my flake will download lot's of dependencies for him. I have created a sub-flake that sets inputs.nixpkgs.url = "nixpkgs" and doesn't have a lock file. That way, he can add ?dir=nolock to the flake url and all is well.

[nrdxp]

Currently this already works with inputs.subflake.url = "path:./adirectory", but if a lock file is not already generated, it can fail if your not in the directory with the flake.nix. Once the lock file exists, you can reference the flake from anywhere and it works as expected.

Perhaps a more elegant solution to this, which would also solve the above mentioned issue, would be to pass a special self reference to the flake inputs, in a similar manner as is passed to outputs. Then you could reference a subflake as inputs.subflake.url = "self:adirectory".

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/possibility-for-subflakes/11589/1

[DieracDelta]

This issue with path is that you cannot mark submodules = true;. This complicates things if you want to build from a local checkout of a GitHub repo and all its submodules.

[zhaofengli]

The current path:../sub-dir-of-same-repo solution doesn't actually work as it comes with a giant catch: Everything, including .gitignore'd files, in that directory is copied. This is unacceptable when you are developing and your flake directory contains loads of build artifacts. In the current implementation you will need to cargo clean or manually move the offending directories out from the input flake in order to not copy lots (in my case, gigabytes) of data into the Nix store.

[roberth]

@zhaofengli does git+file:../sub-dir-of-same-repo work?

[zhaofengli]

@roberth

Thanks for the suggestion. I tried the following and got:

• git+file:../sub-dir -> fetching Git repository 'file:../sub-dir'ssh: Could not resolve hostname file: Name or service not known
• git+file://../sub-dir -> error: file:// URL 'git+file://../sub-dir' has unexpected authority '..'
• git+file:/..?dir=sub-dir -> error: file:// URL 'git+file://..?dir=sub-dir' has unexpected authority '..'
• git+file:///home/zhaofeng/repo?dir=sub-dir -> Works, but it's not really scalable when you have multiple flakes and collaborate with other people

[edolstra]

Closing, you can have inputs like inputs.foo.url = "path:../bar" now.

[winston0410]

@edolstra can we? I have an input inputs.hostname.url = "path:../hostname.nix", and I got this:

error: relative path '../hostname.nix' points outside of its parent's store path '/nix/store/w5ymrwy0igygh8swkm449kb0f7kx1k3v-source'

[roberth]

Relative paths currently rely on a source with the correct hash to be already present in the store. This is fragile but won't be noticed by the flake author, because creating the lockfile adds locally. See #5437

[therealpxc]

This is an annoyance for my config, as I use subflakes to provide flake interfaces for non-flake inputs to my top-level flake.nix, to make things a little cleaner there and to pave the way for sending the flake interfaces for those inputs upstream.

This issue is definitely noticeable for flake authors if you don't go into the subdirectories for your subflakes and run nix flake lock every time you change those things (which you probably don't want to bother to do during development).

Since adding an in-repo flake input by relative path sometimes doesn't work (depending on whether that subflake is already in the Nix store), I think this issue should be reopened.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/locally-excluding-nix-flakes-when-using-nix-independenly-of-upstream/16480/19

[schuelermine]

However this means that we can't have a lock file entry for the foo input.

This seems untrue if the directory is a git repository.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/relative-path-support-for-nix-flakes/18795/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/relative-path-support-for-nix-flakes/18795/2

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/workflows-with-flakes/19223/2

[galenhuntington]

The above discussion indicates to me that this issue has not been solved and should be reopened.

[aakropotkin]

Over the last year this behavior really changed how I write Nix code.

In practice if I have a repo with multiple parts where multiple flakes would make sense, I move definitions out of the flake into individual files. This ends up becoming the classic overlay.nix and default.nix structure with the exception that I pull locked info from locks directly into default args.

Its honestly not how I'd like to organize things and really wished that relative paths "just worked", but I've also read the internals enough to know that I don't want to dedicate time to an RFC and PR which would quickly become a nightmare.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/get-flake-builtins-getflake-without-the-restrictions/17662/14

[bgamari]

I agree that this should be reopened. The status quo of needing to nix flake lock --update-input ... every time a sub-flake is modified is far from ideal, rendering such configurations extremely painful to use. This pushes the user away from taking advantage of clear opportunities for modularity and, at least in my case, acts as a deterrent from using flakes.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/set-up-hibernation-supporting-separate-encrypted-swap-partition/26412/1

[blaggacao]

Leaning out of the window on a design, here: but what's the point of locking rel-paths, at all? They are within the same atomic inputs boundary of (toplevel) self — whether dirty or not.

Is resolution of this being complicated by some code paths that would need to be special cased and are difficult to untangle?

Is it lack of good problem statement and/or understanding? Although this comment actually pinpoints the issue.

Is there anything I can help to mover this towards resolution? A writeup maybe? A visualization? I'm being bitten again in CI. But this time, I'd like to refuse to work around.

[roberth]

I believe @edolstra has solved this on #6530 but not on master yet. We're working to incorporate changes step by step. I can't tell whether the relevant change is independent enough to make a cherry-pick productive. Perhaps you could weigh in on this @edolstra?

[blaggacao]

I actually did try a cherry pick of this. 😊 But it seems it depends on a readFlake routine and that isn't there yet.

So in the meantime, we're back to a workaround again. I submitted https://github.com/edolstra/flake-compat/pull/55

Thanks for the great work im the Nix Team! I appreciate your work and that you'll get back to this when the time is right. It's good to know someone has already backlogged this. Thank you!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/improving-flakes/12831/61

[Atry]

I believe @edolstra has solved this on https://github.com/NixOS/nix/pull/6530 but not on master yet. We're working to incorporate changes step by step. I can't tell whether the relevant change is independent enough to make a cherry-pick productive. Perhaps you could weigh in on this @edolstra?

I tried #6530 and I got the error:

error: expected a string but got a path at /home/Atry/my-project/examples/my-test/flake.nix:2:3

Given:

{
  inputs.my-project = {
    url = ./../..;
  };
}

[roberth]

It might work with a relative path in a string, but the fact that a path expression does not work is a horrible UX bug.

[ck3mp3r]

I've used relative paths with some success... only issue is the flake.lock changes every time I work on another machine where the checkout directory path doesn't match what was used on a machine before... it then moans about using relative paths. A nix flake update solves it, but you keep playing catchup on various machines...

Any eta on supporting relative paths without having the parent directories having to be the same? I am using multiple flakes in one github project, so for composability this would be great.

[blaggacao]

@ck3mp3r https://github.com/edolstra/flake-compat/pull/55

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-06-02-nix-team-meeting-minutes-59/28666/1

[blaggacao]

Another more thorough work around for the meantime: https://github.com/divnix/call-flake

[colemickens]

"Painful" is a light way to describe this bug, especially since folks are doing this whole flake.system.nix thing with flake-utils.

[2xsaiko]

flake.system.nix

What's that? The only occurrence I can find of that in the flake-utils repo (or internet search) is your issue.

[colemickens]

Sorry, I should have included more details, and gotten the name right. See here for more details: https://github.com/nix-systems/nix-systems

For now, every single CI job has to re-run nix flake lock --update-input systems, which at least unblocks me, but doesn't feel right, nor do I want to maintain yet another workaround for something semi-broken in Nix.

(edit: the last line comes off more entitled than I mean it to, good to acknowledge flakes is still experimental, and I'm happy to have what's here.)

[mikepurvis]

I wasn't able to get even the trivial case of inputs.thing.url = "path:./thing" working on Hydra. It was doing the expected thing locally, but then on Hydra (with Nix 2.13.3):

in job ‘<redacted>.x86_64-linux’:
error: cannot fetch input 'path:./thing?lastModified=1&narHash=sha256-tk7BEyl8BW4OSrT%2fRCAhYBzJx3LQJxPcV1e2MFRl7kM=' because it uses a relative path

Is there some trivial error I'm making with this? I'd really like this setup to work as being able to pass an --override-input thing <other-thing> provides some limited parameterization that helps my workflow.

[eclairevoyant]

@mikepurvis see https://github.com/NixOS/nix/issues/3978#issuecomment-952418478

[r-vdp]

@mikepurvis in addition to @eclairevoyant's response, the comment just above should help to unblock you: https://github.com/NixOS/nix/issues/3978#issuecomment-1585001299

If you run nix flake lock with the right arguments, hydra should create the needed store path and then be able to evaluate the flake.

It's an ugly work-around until this issue gets resolved.