Devenv slows down with big code repositories

[shyim]

Describe the bug

When you have a big project and maybe some cache like 10 GB it gets really slowed down by git commands. Maybe that command can be ignored?

 |   \-+= 25589 root /usr/bin/login -fpl shyim /Applications/iTerm.app/Contents/MacOS/ShellLauncher --launch_shell
 |     \-+= 25590 shyim -zsh
 |       \-+- 25994 shyim -zsh
 |         \-+- 25995 shyim /usr/local/bin/direnv export zsh
 |           \-+- 25996 shyim /bin/bash -c eval "$("/usr/local/bin/direnv" stdlib)" && __main__ source_env $'/Users/shyim/Code/shopware/cloud/.envrc'
 |             \-+- 26011 shyim /bin/bash -c eval "$("/usr/local/bin/direnv" stdlib)" && __main__ source_env $'/Users/shyim/Code/shopware/cloud/.envrc'
 |               \-+- 26012 shyim bash /Users/shyim/.nix-profile/bin/devenv print-dev-env
 |                 \-+- 26021 shyim /nix/store/ynwgbyq3p1db8wg5hw8b192l3336g65b-nix-2.12.0pre20221221_51b770e/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes print-dev-env --impure
 |                   \--- 26027 shyim git -C /Users/shyim/Code/shopware/cloud --git-dir .git ls-files --cached -z --others

[shyim]

--others seems like REALLY slow down because it looks then also in the cache folder / node_modules and so

[shyim]

Seems like introduced by custom nix https://github.com/domenkozar/nix/commit/bf9922e7b21b83450006640d2382e04e5a203692

[domenkozar]

We need this to be able to support .devenv.flake.nix, probably a better way is to just to add it to the output.

[shyim]

We could also generate a generic flake.nix and commit that? 🤔

[domenkozar]

@shyim do you have a workaround for this or does it need to be fixed soon?

[shyim]

I am thinking about adding a hybrid solution. .envrc uses nix flake directly and maybe the users still use devenv up. and the flake just imports existing devenv.nix

[domenkozar]

Flakes verbatim copy the whole repo into the /nix/store so that's also slower in general :)

[shyim]

okay then I don't have a solution 😂

[shyim]

But your --others is there to find like devenv.nix and so? Can we not explicit check that this files exists?

[domenkozar]

I'll look into this tomorrow

[domenkozar]

I know what to fix there, needs a few hours of work. Making it high priority :)

[GaetanLepage]

Flakes verbatim copy the whole repo into the /nix/store so that's also slower in general :)

Eelco and others are working on getting rid of this constraint: https://github.com/NixOS/nix/pull/6530

[domenkozar]

devenv already includes this PR, but it's still slow (not minutes as it was before, but now in seconds).

[GaetanLepage]

devenv already includes this PR, but it's still slow (not minutes as it was before, but now in seconds).

Oh cool ! Do you think it will get merged upstream anytime soon ? Has it been working fine in devenv until now ?

[domenkozar]

devenv already includes this PR, but it's still slow (not minutes as it was before, but now in seconds).

Oh cool ! Do you think it will get merged upstream anytime soon ? Has it been working fine in devenv until now ?

I hope it will be merged soon, it has been working more or less fine :)

[domenkozar]

I believe this is fixed in #745, could you give it a try and report back?

[shyim]

I can't test it at Home, somehow the python cannot be built on my machine. It just hangs forever configure. I will try it at work tomorrow, maybe that helps.

[domenkozar]

I can't test it at Home, somehow the python cannot be built on my machine. It just hangs forever configure. I will try it at work tomorrow, maybe that helps.

Having the full log would be helpful and what OS exactly it hangs on.

[domenkozar]

I believe this is fixed in 1.0.1, can somebody confirm?

[shyim]

for me it takes like 25seconds to devenv up starts the processes. feels long 🤔

[domenkozar]

Using 1.0.1? How can I reproduce this?

[shyim]

yup

devenv 1.0.1 (aarch64-darwin)

git clone https://github.com/shopware/shopware.git
cd shopware
devenv up

[shyim]

Is there something nice to debug what is slow at devenv up. We have here an M1 Pro with 32GB of memory it takes 3 minutes to start it. It takes like a minute until it shows something and then two times resolving something

Maybe an idea to have an --debug flag, which shows any command executed and the duration in the shell?