search

haskell-nix / hnix-store

Haskell implementation of the Nix store
Apache License 2.0
87 stars 23 forks source link

Nar uses lazy IO instead of streaming #51

Closed domenkozar closed 3 years ago

domenkozar commented 5 years ago

Feed it a store path with lots of files and you end up with:

/nix/store/3m1hc2mrz6zhqng088s4fw0rzfb1g6qy-perl-5.28.1: getDirectoryContents:openDirStream: resource exhausted (Too many open files)
roberth commented 5 years ago

The culprit.

One may also consider unix's bytestring directory functions, to match Nix more closely. For performance, it's best to getFileStatus once per path.

shlevy commented 5 years ago

@imalsogreg Are you able to take a look?

imalsogreg commented 5 years ago

Yeah - thanks for the analysis on this. I'll try to reproduce with a test and then fix with some combination of streaming and unix. Will update the ticket when I have a WIP branch.

imalsogreg commented 5 years ago

The vigorous-io branch includes a test that can hit the resource exhausted (Too many open files) error.

Right now I have the test set to create just 100 files and assert that 50 file descriptors are open after nar serialization. If I raise the number to 10000, we see an exception. Maybe the test should use 10000 and watch for the exception.

Anyway, now to code up the solution :)

domenkozar commented 5 years ago

Relevant: https://github.com/haskell-nix/hnix-store/pull/52

domenkozar commented 5 years ago

Hey @imalsogreg did you make some progress on this one?

imalsogreg commented 4 years ago

@domenkozar Nope, attention's been elsewhere, but thanks for the bump.

domenkozar commented 4 years ago

I'm happy to sponsor $150 for someone fixing this issue.

imalsogreg commented 4 years ago

@domenkozar I've been focusing on this the past couple days, on the vigorous-io branch.

sorki commented 3 years ago

Closing since this landed some time ago, it could use some real world testing! :smile: