search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.58k stars 5.47k forks source link

Too many open files (EMFILE) when doing `Pkg.update()` on v0.3.- #8601

Closed jakebolewski closed 9 years ago

jakebolewski commented 10 years ago

I installed every registered Julia pakage to test against. On OS X I'm seeing this error (it makes it down to the F's before dying). Trying to call Pkg.update() again segfaults.

   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.1 (2014-09-21 21:30 UTC)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |  x86_64-apple-darwin13.4.0

julia> Pkg.update()
INFO: Updating METADATA...
ERROR: could not spawn setenv(`git rev-parse --git-dir`; dir="/Users/jacobbolewski/.julia/v0.3/.cache/FunctionalCollections"): too many open files (EMFILE)
 in wait at task.jl:51
 in sync_end at /usr/local/Cellar/julia/0.3.1/lib/julia/sys.dylib
 in update at /usr/local/Cellar/julia/0.3.1/lib/julia/sys.dylib
 in anonymous at pkg/dir.jl:28
 in cd at /usr/local/Cellar/julia/0.3.1/lib/julia/sys.dylib
 in __cd#227__ at /usr/local/Cellar/julia/0.3.1/lib/julia/sys.dylib
 in update at /usr/local/Cellar/julia/0.3.1/lib/julia/sys.dylib (repeats 2 times)

julia> Pkg.update()
INFO: Updating METADATA...
Assertion failed: (loop->watchers[w->fd] == w), function uv__io_stop, file src/unix/core.c, line 764.

signal (6): Abort trap: 6
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
[1]    93827 abort      julia
tkelman commented 10 years ago

All the more reason to work on #7339 ? Not positive that it would fix this immediately, but it might.

SimonDanisch commented 10 years ago

I this related?

julia> versioninfo()
Julia Version 0.4.0-dev+1060
Commit 159274f* (2014-10-12 17:15 UTC)
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
julia> Pkg.add("Jewel")
fatal: Not a git repository (or any of the parent directories): .git
ERROR: failed process: Process(setenv(`git rev-parse --git-dir`; dir="C:\\Users\
\Sim\\.julia\\v0.4\\.cache\\Compose"), ProcessExited(128)) [128]
 in stream_wait at stream.jl:265
tkelman commented 10 years ago

@SimonDanisch is that a from-source build? Did you do make win-extras && make dist and run the installer? Unless you do that, the package manager doesn't really work on a Windows from-source build.

SimonDanisch commented 10 years ago

Ah, I totally forgot to tell the most important bits: I did install it with win-extras, and other packages do install, just not Jewel (and probably others?!). Also run('echo hello') works.

tkelman commented 10 years ago

Sorry for my bad guess then. What does ls -al /c/Users/Sim/.julia/v0.4/.cache/Compose looks like? You might have a corrupted cache somehow.

SimonDanisch commented 10 years ago

Oh I see. That was indeed the error... Thanks a lot! Maybe another error message would be appropriate?!

jiahao commented 9 years ago

bump

tkelman commented 9 years ago

Instead of just using stock @sync and @async here https://github.com/JuliaLang/julia/blob/ef062117bab19959a6905b1bb6d7963b42dddea4/base/pkg/entry.jl#L255-L278 I think it might make sense to limit the number of simultaneous instances of git we spawn to CPU_CORES ?

jiahao commented 9 years ago

One hack is to increase the OSX kernel limits from 10,240 (!), e.g. with

sysctl -w kern.maxfiles=20480 (or whatever number you choose)
sysctl -w kern.maxfilesperproc=18000 (or whatever number you choose)

Ref: http://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1

jakebolewski commented 9 years ago

Closed by #11152