not possible to run tests from flake env

[brianmay]

Describe the bug

I don't think there is any provision from running the [tests](https://devenv.sh/tests/ from a flake environment.

To reproduce

$ devenv test
https://devenv.sh (version 1.0): Fast, Declarative, Reproducible, and Composable Developer Environments

This is a flake integration wrapper that comes with a subset of functionality from the flakeless devenv CLI.

Usage: devenv command

Commands:

up              Starts processes in foreground. See http://devenv.sh/processes
version         Display devenv version

Version

$ devenv version
devenv: 1.0

Git revision: a7ba6766c0cc351b8656c63560c6b19c3788455f

Same thing with latest: 5d99d9fb18882d2b2ceeb09688c158e39b927394

[domenkozar]

What's the reason you're using flakes instead of devenv?

[brianmay]

Not sure I understand the question. I am using devenv inside a flake. As documented here: https://devenv.sh/guides/using-with-flakes/

[domenkozar]

Why did you choose to go with the flakes guide?

[brianmay]

I already have a working flake, and I want my devenv to provide matching software versions.

[veselyn]

I'm having the same problem.

I tried adding ciDerivation as a flake check as suggested here but that doesn't quite solve my problem because I need internet access to run my tests and a flake check doesn't provide that.

In addition, the container commands seem to be missing as well.

[anoadragon453]

@domenkozar We're using flakes in order to have a centrally-managed repository of all devenv project configurations. Then, when developing in each project, one can just do nix develop --impure github:element-hq/nix-flakes#synapse and start working. And any updates to the nix developer environments can be made by people familiar with writing nix code - without needing to put the burden on the project maintainers by submitting a PR that they'll struggle to review/not really want to maintain.

I ran a thought experiment of using the standalone devenv tool with devenv.yaml/nix/lock files, and I don't think it would be a good experience. For instance, the version of devenv that we're using is currently set by our https://github.com/element-hq/nix-flakes repo, and is pulled in by nix develop .... If we asked developers to bring their own version of devenv, we'd inevitably find that some would become incompatible with our configurations.

It's also more work for developers to install devenv as well as nix. We could have minimal flake.nix in each repo that installs devenv... but it'd be an awful lot of PRs to update every one when it came time to update the devenv version across the board. Currently we can just do so in the nix-flakes repo - ensuring that it works with the current module configuration.

Our use case for devenv test is less for our developers to use, but more to do nightly runs in the CI of the https://github.com/element-hq/nix-flakes repo - ensuring that the development environments don't break over time and that our developers can keep working. Currently we can't make use of devenv test, and instead have our own hacky test script but it would be nice to use something more native to devenv.

[domenkozar]

Hey @anoadragon453, thanks for this throughout report!

This seems to be a features request for https://github.com/cachix/devenv/issues/67 that has come up a few times.

As for keeping track of devenv version, what if we add an option to devenv.nix that defines the minimal version that devenv cli is required and instructions how to upgrade?

[domenkozar]

@anoadragon453 I wonder having devenv.yaml like this would be good enough for your use case:

from: github:element-hq/nix-flakes#synapse

That way the commands are still devenv test without having to specify where it comes from and requires no Nix.

[anoadragon453]

@anoadragon453 I wonder having devenv.yaml like this would be good enough for your use case:

from: github:element-hq/nix-flakes#synapse

That way the commands are still devenv test without having to specify where it comes from and requires no Nix.

Does that mean that each project repo would only need to have .envrc, devenv.yaml and devenv.lock files? And github:element-hq/nix-flakes#synapse would point to the output of devenv.lib.mkShell? My only concern here is that a devenv.lock file in the project repo would still need updating - but we could achieve this with CI that auto-commits a devenv.lock update if devenv test passes.

One other nice aspect of our flake setup is that we have defined a composeShell function, which allows us to combine devenv shells together: https://github.com/element-hq/synapse/compare/develop...anoa/migrate_flake#diff-206b9ce276ab5971a2489d75eb1b12999d4bf3843b7988cbe8d687cfde61dea0R46-R54

"Complement" and "Sytest" are our two integration test suites - each written in separate programming languages and with their own set of dependencies. When one is developing on Complement, they only need the Complement development environment. However, our Synapse developers need the dependencies of all three projects to be able to run each test suite as well as Synapse itself. Hence the ability to combine development environments like this.

Perhaps this could be done by having the from property in devenv.yaml take a list?

from:
  - github:element-hq/nix-flakes#synapse
  - github:element-hq/nix-flakes#complement
  - github:element-hq/nix-flakes#sytest

As for keeping track of devenv version, what if we add an option to devenv.nix that defines the minimal version that devenv cli is required and instructions how to upgrade?

That sounds relatively developer friendly. I suppose there may be some danger of the developers having a devenv version that's too new (which drops backwards-compatibility for some feature). We'd then need to ask the developer to downgrade their devenv installation until we have had a chance to fix things.

[domenkozar]

Does that mean that each project repo would only need to have .envrc, devenv.yaml and devenv.lock files? And github:element-hq/nix-flakes#synapse would point to the output of devenv.lib.mkShell? My only concern here is that a devenv.lock file in the project repo would still need updating - but we could achieve this with CI that auto-commits a devenv.lock update if devenv test passes.

Only .envrc where devenv.nix and devenv.yaml would come from the referenced source.

One other nice aspect of our flake setup is that we have defined a composeShell function, which allows us to combine devenv shells together: element-hq/synapse@develop...anoa/migrate_flake#diff-206b9ce276ab5971a2489d75eb1b12999d4bf3843b7988cbe8d687cfde61dea0R46-R54

That's possible using imports bit in devenv.yaml, but we could also support the same using from.

"Complement" and "Sytest" are our two integration test suites - each written in separate programming languages and with their own set of dependencies. When one is developing on Complement, they only need the Complement development environment. However, our Synapse developers need the dependencies of all three projects to be able to run each test suite as well as Synapse itself. Hence the ability to combine development environments like this.

Perhaps this could be done by having the from property in devenv.yaml take a list?

from:
  - github:element-hq/nix-flakes#synapse
  - github:element-hq/nix-flakes#complement
  - github:element-hq/nix-flakes#sytest

As for keeping track of devenv version, what if we add an option to devenv.nix that defines the minimal version that devenv cli is required and instructions how to upgrade?

That sounds relatively developer friendly. I suppose there may be some danger of the developers having a devenv version that's too new (which drops backwards-compatibility for some feature). We'd then need to ask the developer to downgrade their devenv installation until we have had a chance to fix things.

We won't drop backwards compatibility just like that, planning to deprecate things and remove them later on.

[anoadragon453]

Only .envrc where devenv.nix and devenv.yaml would come from the referenced source.

But then, where would I specify from: in the project repo (https://github.com/element-hq/synapse)? If it doesn't have a devenv.yaml file?

Or did you mean devenv.lock instead of devenv.yaml? If you did, then, that sounds perfect.

That's possible using imports bit in devenv.yaml, but we could also support the same using from.

If I understand this correctly, one could have something like the following snippet in their devenv.yaml to achieve what we're after?

from: github:element-hq/nix-flakes
imports:
    - nix-flakes/synapse
    - nix-flakes/complement
    - nix-flakes/sytest

One final complication is that we actually have a private repo at https://github.com/element-hq/nix-flakes-internal which holds the development environments for our private repos (where even the name of the project would give away information). When working on private code, we may need to compose together multiple development environments from multiple git repos, both public and private. In this case, having from take a list may still be desirable.