BufferFull error on specific store path

[mikepurvis]

An empty response (no headers, nothing) comes back, and the log shows:

10.26.14.222 - - [29/Nov/2022:17:34:55 +0000] "GET /y94ivgf0ygn86fwsqqhpb83lk7bhprlb.narinfo HTTP/1.1" 200 22967 "" "curl/7.81.0"
toBufIOWith: BufferFull: minSize
CallStack (from HasCallStack):
  error, called at ./Network/Wai/Handler/Warp/IO.hs:23:36 in warp-3.3.21-Fi06NMufirnA6UXRcnfV9n:Network.Wai.Handler.Warp.IO

Regular nix-serve and harmonia have no problem serving this narinfo, though the References list is quite extensive— the overall size of the narinfo response is almost 23k bytes.

[Gabriella439]

If it's not sensitive, could you upload a NAR for the path in question by running:

$ nix-store --dump /nix/store/y94ivgf0ygn86fwsqqhpb83lk7bhprlb* > example.nar

That would help us reproduce the issue locally

[mikepurvis]

Unfortunately I'm not able to upload it, but the length of the References field was definitely the most significant and interesting aspect:

$ curl http://hydra.clearpath.ai/cache/y94ivgf0ygn86fwsqqhpb83lk7bhprlb.narinfo | grep References | wc
      1     455   22507

Maybe a trivial derivation that references a few hundred things in nixpkgs would repro the issue?

[angerman]

Maybe it's related to this: https://github.com/yesodweb/wai/issues/894, and a simple byte string update fixes it. This to me happens with a [("Content-Type","text/x-nix-narinfo"),("Content-Length","27466")]. The references line alone is ~17kb.

[angerman]

And this is the relevant https://github.com/haskell/bytestring/pull/538 issue. Until byte string 0.11.4 is released, I guess the best is to either s-r-p a byte string, or downgrade to say ghc 8.10.7.

[mikatammi]

I'm also seeing this:

warning: error: unable to download 'http://172.18.8.142:5000/bqnisnimwjiaki9376232b1jlydz3qn2.narinfo': Server returned nothing (no headers, no data) (52); retrying in 334 ms
warning: error: unable to download 'http://172.18.8.142:5000/bqnisnimwjiaki9376232b1jlydz3qn2.narinfo': Server returned nothing (no headers, no data) (52); retrying in 543 ms
warning: error: unable to download 'http://172.18.8.142:5000/bqnisnimwjiaki9376232b1jlydz3qn2.narinfo': Server returned nothing (no headers, no data) (52); retrying in 1024 ms
warning: error: unable to download 'http://172.18.8.142:5000/bqnisnimwjiaki9376232b1jlydz3qn2.narinfo': Server returned nothing (no headers, no data) (52); retrying in 2440 ms
error: unable to download 'http://172.18.8.142:5000/bqnisnimwjiaki9376232b1jlydz3qn2.narinfo': Server returned nothing (no headers, no data) (52)

The package in question is "runtime-deps", I get this while running nixos-rebuild

In the server's logs:

May 30 15:05:10 nixos-ryzen nix-daemon[2231718]: accepted connection from pid 63210, user nix-serve
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]: 172.18.8.152 - - [30/May/2023:15:05:10 +0300] "GET /bqnisnimwjiaki9376232b1jlydz3qn2.narinfo HTTP/1.1" 200 28219 "" "curl/7.86.0 Nix/2.11.1"
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]: toBufIOWith: BufferFull: minSize
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]: CallStack (from HasCallStack):
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]:   error, called at ./Network/Wai/Handler/Warp/IO.hs:23:36 in warp-3.3.21-Fi06NMufirnA6UXRcnfV9n:Network.Wai.Handler.Warp.IO
May 30 15:05:10 nixos-ryzen hydra-queue-runner[2231957]: checking the queue for builds > 822...
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]: 172.18.8.152 - - [30/May/2023:15:05:10 +0300] "GET /bqnisnimwjiaki9376232b1jlydz3qn2.narinfo HTTP/1.1" 200 28219 "" "curl/7.86.0 Nix/2.11.1"
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]: toBufIOWith: BufferFull: minSize
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]: CallStack (from HasCallStack):
May 30 15:05:10 nixos-ryzen nix-serve-start[63210]:   error, called at ./Network/Wai/Handler/Warp/IO.hs:23:36 in warp-3.3.21-Fi06NMufirnA6UXRcnfV9n:Network.Wai.Handler.Warp.IO
May 30 15:05:11 nixos-ryzen nix-serve-start[63210]: 172.18.8.152 - - [30/May/2023:15:05:10 +0300] "GET /bqnisnimwjiaki9376232b1jlydz3qn2.narinfo HTTP/1.1" 200 28219 "" "curl/7.86.0 Nix/2.11.1"
May 30 15:05:11 nixos-ryzen nix-serve-start[63210]: toBufIOWith: BufferFull: minSize
May 30 15:05:11 nixos-ryzen nix-serve-start[63210]: CallStack (from HasCallStack):
May 30 15:05:11 nixos-ryzen nix-serve-start[63210]:   error, called at ./Network/Wai/Handler/Warp/IO.hs:23:36 in warp-3.3.21-Fi06NMufirnA6UXRcnfV9n:Network.Wai.Handler.Warp.IO
May 30 15:05:12 nixos-ryzen nix-serve-start[63210]: 172.18.8.152 - - [30/May/2023:15:05:12 +0300] "GET /bqnisnimwjiaki9376232b1jlydz3qn2.narinfo HTTP/1.1" 200 28219 "" "curl/7.86.0 Nix/2.11.1"
May 30 15:05:12 nixos-ryzen nix-serve-start[63210]: toBufIOWith: BufferFull: minSize
May 30 15:05:12 nixos-ryzen nix-serve-start[63210]: CallStack (from HasCallStack):
May 30 15:05:12 nixos-ryzen nix-serve-start[63210]:   error, called at ./Network/Wai/Handler/Warp/IO.hs:23:36 in warp-3.3.21-Fi06NMufirnA6UXRcnfV9n:Network.Wai.Handler.Warp.IO
May 30 15:05:15 nixos-ryzen nix-serve-start[63210]: 172.18.8.152 - - [30/May/2023:15:05:15 +0300] "GET /bqnisnimwjiaki9376232b1jlydz3qn2.narinfo HTTP/1.1" 200 28219 "" "curl/7.86.0 Nix/2.11.1"
May 30 15:05:15 nixos-ryzen nix-serve-start[63210]: toBufIOWith: BufferFull: minSize
May 30 15:05:15 nixos-ryzen nix-serve-start[63210]: CallStack (from HasCallStack):
May 30 15:05:15 nixos-ryzen nix-serve-start[63210]:   error, called at ./Network/Wai/Handler/Warp/IO.hs:23:36 in warp-3.3.21-Fi06NMufirnA6UXRcnfV9n:Network.Wai.Handler.Warp.IO

[mikatammi]

Moved to @Mic92 's fork for now https://github.com/Mic92/nix-serve-ng/commits/fix-build Until I change this system to a flake-based configuration where I can override nixpkgs in the inputs

[Gabriella439]

Unfortunately I no longer have the commit bit on this repository so I can't merge any fixes at the moment (cc: @jsoo1)

[jsoo1]

I am so sorry! This has totally slipped my mind. I can prioritize review now.