Builds broken in nix-build (Nix) 2.22.1

mbwmbw1337 commented 3 months ago

Upgraded from nix 2.16 and 2.18to nix-build (Nix) 2.22.1. Builds are now broken in block-explorer.

root@mightybyte-sexy:/home/chainweb-explorer/block-explorer# nix-build -A exe
error: stack overflow (possible infinite recursion)

enobayram commented 3 months ago

That's odd, I've just tried doing the same locally and it worked for me. To reduce the number of moving parts, would you mind trying to build it like this:

nix shell nix/2.22.1 -c nix-build -A exe

mbwmbw1337 commented 3 months ago

Same issue. I am running this on Ubuntu 22.04.4 LTS with a Multi-user installation Install Nix via the recommended multi-user installation:

$ sh <(curl -L https://nixos.org/nix/install)

Error

root@fmightybyte-sexy-4 /home/chainweb-explorer/block-explorer # nix --extra-experimental-features nix-command --extra-experimental-features flakes  shell nix/2.22.1 -c nix-build -A exe
error: stack overflow (possible infinite recursion)
root@mightybyte-sexy-4 /home/chainweb-explorer/block-explorer #

mbwmbw1337 commented 3 months ago

Verbose Log

instantiated 'cabal2nix-2.14.4.tar.gz' -> '/nix/store/is4j4v3wxrg3ds9zqkf898kfwgq2475j-cabal2nix-2.14.4.tar.gz.drv'
instantiated 'cabal2nix-2.14.4' -> '/nix/store/hkyvziz22aw8k6rq32nkfhd587dpjfm7-cabal2nix-2.14.4.drv'
instantiated 'cabal2nix-backend' -> '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv'
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: created
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: woken up
querying info about missing paths...
starting pool of 24 threads
entered goal loop
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: init
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: loading derivation
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: have derivation
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: done
building of '/nix/store/s2scj5hl53sac5ghfk2z3g8kx5x8gwyi-cabal2nix-backend.drv^out' from .drv file: goal destroyed
evaluating file '/nix/store/zm9fvjmmhsn444q0pgyyl8z7fj68rvw8-cabal2nix-backend/default.nix'
evaluating file '/nix/store/fk1pqfl3qv5r5jaf0lq1xzxnlzj13b2q-source/dep/gitignore.nix/default.nix'
evaluating file '/nix/store/b5w4rjxslafcb5clk0vbvmkyd1ra9w67-source/default.nix'
source path '/nix/store/fk1pqfl3qv5r5jaf0lq1xzxnlzj13b2q-source/lib/run' is uncacheable
copying '/nix/store/fk1pqfl3qv5r5jaf0lq1xzxnlzj13b2q-source/lib/run' to the store...
evaluating file '/nix/store/b5w4rjxslafcb5clk0vbvmkyd1ra9w67-source/find-files.nix'
evaluating file '/nix/store/b5w4rjxslafcb5clk0vbvmkyd1ra9w67-source/rules.nix'
evaluating file '/nix/store/b5w4rjxslafcb5clk0vbvmkyd1ra9w67-source/parse-git-config.nix'
error: stack overflow (possible infinite recursion)

Looks like the issue is with hercules-ci/gitignore

enobayram commented 3 months ago

Thanks for the extra information. Here's what I did:

Start an AWS EC2 instance of size t2.xlarge from AMI: ami-0e001c9271cf7f3b9, which is AWS's official AMI for Ubuntu 22.04
In the image run the following:
```
sh <(curl -L https://nixos.org/nix/install)
. /home/ubuntu/.nix-profile/etc/profile.d/nix.sh
git clone https://github.com/kadena-io/block-explorer.git
cd block-explorer/
nix --extra-experimental-features nix-command --extra-experimental-features flakes  shell nix/2.22.1 -c nix-build -A exe
```
This build takes a very long time since we're not using any binary caches like this and it eventually ran out of disk space for me since I unfortunately forgot to give it more space, but it made enough progress to make me believe that this nix-eval-time stack overflow wouldn't show up there.

I see the following in your last message:

warning: Git tree '/home/chainweb-explorer/block-explorer' is dirty

Given that it's hercules-ci/gitignore that's causing the issue, I'm wondering if you have a lot of ignored filed lying around in your local checkout. Would you mind trying from a fresh checkout of the repository?

BTW, I see that you've tried nix runing the repo, which would start a block-explorer server pointing at localhost, mostly for devnet purposes, but you could try the following:

nix --extra-experimental-features "nix-command flakes recursive-nix" build .#exe

This essentially builds the same thing as nix-build -A exe, but it runs the nix-build nested inside a nix build "recursively" in a reproducible environment, which makes the nix eval time reproducible as well. We had to make that special setup for the flake.nix in this repo, it might help with your local checkout.

mbwmbw1337 commented 3 months ago

Thank you for the very detailed and excellent response, plus troubleshooting. I am trying your suggestions and trying with a fresh pull as well. I've made some modifications in local source which is why it's showing dirty (including backing up files).

Give me a few hours and I will return with my findings. I figured out to add recurisve-nix in the build so we'll see what happens!

enobayram commented 3 months ago

And thank you for the report and all the information! BTW, in a multi-user Nix installation, just passing --extra-experimental-features recursive-nix won't be enough, the build daemon needs to be configured with recursive-nix as well, so you'll need to add

experimental-features = recursive-nix

to your /etc/nix/nix.conf and restart the build daemon with systemctl restart nix-daemon.

mbwmbw1337 commented 3 months ago

Thank you.

Ironically, I also tried "update-nix", which downgraded Nix, and then the build worked...

root@mightbyte-is-sexy ~ #  nix upgrade-nix
replacing old 'nix-2.22.1'
installing 'nix-2.18.1'
building '/nix/store/hsygjmvkic6bs78y0vp2pahn7xqjrl9c-user-environment.drv'...

Then I ran nix-build -A exe, and there were no issues at all; it builds fine and runs.

I wonder if something in my nix-store is problematic. I tried from a fresh AWS image like yours, without issuessues at all. All my 20+ servers are built from source and have the same nix-store.

Next, I will try taking a machine, destroying the entire nix-store and building with nix-2.22.1. If this works then something must be problematic in the store.

mbwmbw1337 commented 3 months ago

Are you on Discord? What's your username? I'd like to give you access to an affected server...

enobayram commented 3 months ago

It's not hard to believe that there might be a subtle regression with the latest version of Nix, particularly since block-explorer's Nix setup is quite complex and fairly old at this point. My discord handle is the same as my GitHub handle, but I'm not active there.

kadena-io / block-explorer

Builds broken in nix-build (Nix) 2.22.1 #93