access source code outside of cells

keidrych commented 6 months ago

Some projects require isolating configuration from the source code. DVCS options (i.e. subtrees etc) aside STD should have an ability to split the location of NIX from source

If I'm not mistaken, STD can work in the following ways:

Nested src

flake.nix
/cells/app/runnable.nix
/cells/app/src/

`src = ./src;' in runnable.nix

Flaked src

flake.nix
/cells/app/runnable.nix

src = inputs.app + ./app; in runnable.nix

While this could work if every source app had a flake.nix & the parent flake.nix managing STD knew about these inputs, its a lot of overhead to manage and too easy to accidentally edit the wrong flake.nix when using nix flake commands
Alternatively, a symlink from ./app to ./cells/app/src/ could be used, but it forces the developers to work in a deeply nested tree and complicates the benefits of horizontal & vertical integration tooling of Nix (i.e. keep the slowly changing parts away from the main activity or source code)

Suggestion/need:

flake.nix
/cellls/app/runnable.nix
/app/src/

src = inputs.self + ./app/src in runnable.nix (this allows use of excellent flake src filtering tooling such as: numtide's nix-filter or src = ../../app/src; in runnable.nix, can work with std.incl

At first glance, you may think a symlink from /app to /cells/app would be suitable, but this will not work due to the way NIX implements directory linking via symlinkJoin https://ertt.ca/blog/2022/01-12-nix-symlinkJoin-nodePackages/. A post step could be included in STD to do something similar?

The issue, seems to be in flake.nix

std.grow.cellsFrom = ./cells

Effectively ignores everything outside of ./cells/, meaning any other option to access source code from the flake.nix root or inputs.self is impure

You can verify this by going to the NIX store & seeing that beyond the inputs.self store, STD in its derivation is tracking only

If you try to add the source path to grow STD complains about the additional path not being cellBlock compatible

What STD needs to do (for increased flexibility)

Add the ability to include a list of paths to bring into STD in addition to Cells this could be as simple as adding something like

std.include
std.raw
std.extraPaths

Naturally, this would need to be available in STD (normal) & STD (Flake-Parts) versions

whs-dot-hk commented 6 months ago

Hi, there is incl and layers of soil for that

blaggacao commented 6 months ago

could work if every source app

I'm not sure if this is the core premis of the argument, but if so, there's flake = false; on flake inputs with which any non-flake, and for that matter non-nix source can be fetched and used the same way (src = inputs.app).

In between the lines I also read that inputs.self might come in handy, at times. It simply refers to a non-flake reference of the owns repository source (top level). I.e. inputs.self + /otherfolder/subfolder/app.

keidrych commented 6 months ago

Hi, there is incl and layers of soil for that

I'm using the flake-parts integration of STD, I could be mistaken but for flake-parts + STD layers of soil are unavailable? incl functionally doesn't filter the flake & source of as effectively as nix-filter. I have verified this via time of both approaches.

could work if every source app

I'm not sure if this is the core premis of the argument, but if so, there's flake = false; on flake inputs with which any non-flake, and for that matter non-nix source can be fetched and used the same way (src = inputs.app).

It is not about the sources containing a flake.nix or not in this instance, the sources are out of tree and inaccessible to STD as std only pulls into the store what is under the <cells> directory everything else is inaccessible to STD.

i.e. in app.nix: src = ../../. will attempt to pull in the nix store, not other parts of the original source directory with the flake.nix that called STD.

In between the lines I also read that inputs.self might come in handy, at times. It simply refers to a non-flake reference of the owns repository source (top level). I.e. inputs.self + /otherfolder/subfolder/app.

This would be great if it worked, but at the moment unless otherfolder is under a STD cell it is inaccessible to STD (at least with the flake-parts integration)

blaggacao commented 6 months ago

I'm using the flake-parts integration of STD, I could be mistaken but for flake-parts + STD layers of soil are unavailable?

Afaik, flake-parts allow you to define any sort of outputs much in the way the layers of soil in STD-only would. But without using flake-parts myself actively, I can't readily give an example, unfortunately.

i.e. in app.nix: src = ../../. will attempt to pull in the nix store, not other parts of the original source directory with the flake.nix that called STD.

Yes that is by design. inputs.self + /my/other/path would fetch sources from other parts of the directory. In my current understanding (which well might be wrong), either this is what you want or to alternatively use flake = false; to fetch your source from a different repo (if that where the alternative choice of code organization).

(at least with the flake-parts integration)

Ah! I can say this:

inputs here comes from flake-parts (as a flake-parts module)
inputs.self is defined by this assignation from paisano/core.

I don't really see where it would fail (because if inputs from flake-parts hadn't inputs.self [presumably representing the top level self] that assignation would fail completely) so on that basis I'm still a bit skeptical of the report, but I also don't have time to reproduce to disprove it.

I still hope this provides some clues that can get you lifted over the hurdle.

Edit: maybe, effectively, flake-parts doesn't provide inputs.self at all (for similar sanity reasons as STD only provides a reference to the source). If that's the case, maybe there's a way to graft self onto inputs with flake-parts... :shrug:

keidrych commented 4 months ago

grafting another path into inputs does expose the source to std, however, there is no ability to filter the source code :cry:

It is possible to get Nix Flakes to import a nix function file and then resolve that function via inputs later. This would allow filtering of the src but requires --impure to be used, and the function would need to return absolute paths.

I don't think this is ever going to work at this point, which is a shame as keeping configuration and src code separate does keep things simpler for those unfamiliar with Nix or STD.

blaggacao commented 4 months ago

which is a shame as keeping configuration and src code separate does keep things simpler for those unfamiliar with Nix or STD.

Just to rectify this statement if taken on its own (I know there's context above, but for people who might just pick this out), that is a core feature of Standard:


# within ./nix folder, away from the sources
{
  src = inputs.std.incl (inputs.self + /mysources) [
    "folder"
    "file.lock"
    ...
  ];
}

divnix / std