In large projects ulimit is hit with universal target

sbt / sbt-native-packager

sbt Native Packager

https://sbt-native-packager.readthedocs.io/en/stable/

BSD 2-Clause "Simplified" License

1.6k stars 441 forks source link

In large projects ulimit is hit with universal target #355

Open magnusart opened 10 years ago

magnusart commented 10 years ago

I have a Play Framework 2.3.x application. When running dist in a large-ish project I hit the limit for open files. This is not a problem on my local machine but using services like Wercker where we can't change this setting makes it problematic.

I have tested with building both deb and docker and those target can build the same project without hitting altering ulimit.

I am using the 8-M1 version.

muuki88 commented 10 years ago

Hi Magnus,

Thanks for the report. Can you give a bit more information on

OS it works and not. This probably depends strongly on the used OS
the size we are talking about. 10 000, 100 000, 1 000 000 files
if you're able to, the location where the handles are created (wild guess, somewhere in the zipping utils)

magnusart commented 10 years ago

Sorry for the late reply. Here is what I know right now, of course I can do some tests as well:

OS: I have tested with both Mac OSX and Ubuntu 14.04
Size: the limit was 1024 files in both cases
Location: This I'm not sure of. But I'd say your guess is the same as mine. As I wrote I have tested with both the deb and docker targets without problems under the same OS:es and file limits. Those targets completes so the best place to start looking would be in the zipping.

The reason I found out about this was because we made builds with Wercker.com that imposes this limit. I have then reproduced it on my local OS X machine.

Since our code is proprietary I can't give you something to test, but a fairly large OSS code base should do the trick. Running with universal:binary after reducing the file handler limit to 1024 would hopefully produce the same issue.

magnusart commented 10 years ago

Here are a bit more information, I dug up the old stack trace:

https://gist.github.com/magnusart/74bed5d66c0e96bd58be

Also when looking around for java zipping and file handler problems, while not very common, is not entirely unheard of similar issues: http://www-01.ibm.com/support/docview.wss?uid=swg21403391

muuki88 commented 10 years ago

Thanks a lot for the research. Any other help on solving this is very appreciated :)

magnusart commented 8 years ago

@francisdb I think you misunderstood this bug. The issue occurs whilst packaging (at build time, running dist). Any problems with file descriptor when executing the packages application is a separate issue.

francisdb commented 8 years ago

I indeed did, well let me move that stuff to the other ticket then.