search

grosser / parallel_tests

Ruby: 2 CPUs = 2x Testing Speed for RSpec, Test::Unit and Cucumber
3.38k stars 495 forks source link

Trouble with running parallel tests on Mac. Too many open files in system #507

Open truekabal opened 8 years ago

truekabal commented 8 years ago

Hi. I'm using parallel_tests on Mac and I have such trouble. While I run tests up to 8 threads one of my tests fails due to exception

/Users/mihail/.rvm/gems/ruby-1.9.3-p551@dev2/gems/parallel_tests-2.5.0/lib/parallel_tests/rspec/runtime_logger.rb:53:in pwd': Too many open files in system - getcwd (Errno::ENFILE) from /Users/mihail/.rvm/gems/ruby-1.9.3-p551@dev2/gems/parallel_tests-2.5.0/lib/parallel_tests/rspec/runtime_logger.rb:53:inblock (2 levels) in start_dump' from /Users/mihail/.rvm/gems/ruby-1.9.3-p551@dev2/gems/parallel_tests-2.5.0/lib/parallel_tests/rspec/runtime_logger.rb:52:in each' from /Users/mihail/.rvm/gems/ruby-1.9.3-p551@dev2/gems/parallel_tests-2.5.0/lib/parallel_tests/rspec/runtime_logger.rb:52:inblock in start_dump' from /Users/mihail/.rvm/gems/ruby-1.9.3-p551@dev2/gems/parallel_tests-2.5.0/lib/parallel_tests/rspec/logger_base.rb:48:in `lock_output'

Is there any solution for this problem except increasing max open files value? Can i somehow limit open files count by thread?

Here is my gem list

actionmailer (2.3.5) actionpack (2.3.5) activerecord (2.3.12, 2.3.5) activeresource (2.3.5) activesupport (3.2.0, 3.0.6, 2.3.12, 2.3.5) addressable (2.3.8) archive-tar-minitar (0.5.2) autoparse (0.3.3) bigdecimal (1.1.0) bundler (1.10.5, 1.2.3) bundler-unload (1.0.2) colorize (0.7.7) columnize (0.3.6) daemons (1.1.4) debugger-ruby_core_source (1.2.3) delayed_job (2.0.6) diff-lcs (1.2.5, 1.1.3, 1.1.2) docile (1.1.5) eventmachine (0.12.10) executable-hooks (1.3.2) extlib (0.9.16) factory_girl (1.3.3) faraday (0.9.1) ffi (1.9.10) gem-wrappers (1.2.7) git (1.2.5) gnuplot (2.3.6) god (0.11.0) google-api-client (0.8.6) google_drive (1.0.1) googleauth (0.4.1) haml (3.1.2) i18n (0.7.0) io-console (0.3) jeweler (1.6.4) json (1.8.2, 1.8.1, 1.5.5, 1.5.3) json_pure (1.8.2, 1.8.1) jwt (1.5.1) launchy (2.4.3) linecache19 (0.5.12) little-plugger (1.1.3) logging (2.0.0) memcached (1.2.6) memoist (0.12.0) method_source (0.8.2) mime-types (1.23) mini_portile (0.6.2) minitest (5.7.0, 2.5.1) msgpack (0.4.5) multi_json (1.11.2, 1.11.0, 1.10.1) multi_xml (0.5.5) multipart-post (2.0.0) net-sftp (2.0.5) net-ssh (2.10.0.beta2, 2.9.1, 2.6.1, 2.2.1) nokogiri (1.6.6.2) oauth (0.4.7) oauth2 (1.0.0) open4 (1.0.1) parallel (1.8.0) parallel_tests (2.5.0) pg (0.18.3) rack (1.6.4, 1.1.2, 1.0.1) rack-test (0.6.0) rails (2.3.5) rake (0.9.2.2) rb-inotify (0.9.5) rb-readline (0.4.0) rdoc (4.1.1, 3.9.5) require (0.2.7) rest-client (1.6.7) retriable (1.4.1) roo (2.1.0) rspec (2.12.0) rspec-core (2.12.2) rspec-expectations (2.12.1) rspec-mocks (2.12.2, 2.12.1) rspec-retry (0.4.4) rubigen (1.5.4) ruby-debug-base19x (0.11.31, 0.11.30.pre15) ruby-debug-ide (0.6.0, 0.4.24) ruby_core_source (0.1.5) rubyforge (2.0.4) rubygems-bundler (1.4.4) rubygems-update (2.6.4, 2.4.6, 1.8.29, 1.8.25) rubyzip (1.1.7, 0.9.4) rvm (1.11.3.9, 1.11.3.8) scientist (1.0.0) signet (0.6.1) simplecov (0.10.0, 0.8.2) simplecov-html (0.10.0, 0.8.0) sinatra (1.2.7) thin (1.2.11) thread_safe (0.3.5) tilt (2.0.0.beta1, 1.4.1, 1.3.3) tzinfo (1.2.2) unicode (0.4.0) xml-simple (1.1.2) xmpp4r (0.5)

grosser commented 8 years ago

does it also fail when run with in_threads: 0 ? -> find out what opens all these file pointers and never closes them ... if it only happens when running in parallel then there must be some weird race condition or something about multithreading ... but lets rule out the obvious above problem first :)