search

grosser / parallel

Ruby: parallel processing made simple and fast
MIT License
4.16k stars 254 forks source link

rubocop --parallel failing with "negative array size" #349

Closed mark-young-atg closed 2 months ago

mark-young-atg commented 2 months ago

Our github action to run rubocop started failing when parallel was upgraded from 1.25.1 to 1.26.0

This is the error seen

Run bundle exec rubocop --parallel
negative array size
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `initialize'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `new'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `create_workers'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:520:in `work_in_processes'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:291:in `map'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:235:in `each'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:103:in `warm_cache'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:76:in `run'
/home/runner/work/sm-web/sm-web/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
...snip...

I presume this is because of the following change that uses Concurrent.available_processor_count.floor to determine the number of available processors if the gem concurrent-ruby is installed, which in my case it is.

I can see that I can use the environment variable PARALLEL_PROCESSOR_COUNT to override this behaviour for my configuration but I think there is an issue here to be understood and mitigated around the behaviour in github actions. Whilst I appreciate the incorrect value may come from concurrent-ruby, I think perhaps that parallel should handle a negative value in a different way. Perhaps by failing back to using Etc.nprocessors as the previous version did.

I have not created a PR for this because I felt it needed discussion and knowledge from those that understand parallel vastly more than I do.

mark-young-atg commented 2 months ago

As a further comment on this, I checked our Amazon AWS instance and the use of Concurrent.available_processor_count.floor looks as though it will cause problems by returning a negative number. This is what I see on our ECS instance:

irb(main):001> Concurrent.available_processor_count.floor
=> -1
irb(main):002> Etc.nprocessors
=> 2
grosser commented 2 months ago

yanked 1.26.0

On Thu, Aug 8, 2024 at 4:20 PM mark-young-atg @.***> wrote:

As a further comment on this, I checked our Amazon AWS instance and the use of Concurrent.available_processor_count.floor looks as though it will cause problems by returning a negative number. This is what I see on our ECS instance:

irb(main):001> Concurrent.available_processor_count.floor => -1 irb(main):002> Etc.nprocessors => 2

— Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/349#issuecomment-2275953547, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACYZYBFXYCNBCDKSZIWBLZQN5CXAVCNFSM6AAAAABMGSC452VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZVHE2TGNJUG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

ovsant commented 2 months ago

Same issue

bundle exec rubocop --color
negative array size
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `initialize'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `new'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `create_workers'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:520:in `work_in_processes'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:291:in `map'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:235:in `each'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:103:in `warm_cache'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:76:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:52:in `with_redirect'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:25:in `execute_runner'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:17:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command.rb:11:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/environment.rb:18:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:122:in `run_command'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:129:in `execute_runners'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:51:in `block in run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:81:in `profile_if_needed'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:43:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/exe/rubocop:19:in `block in <top (required)>'
/usr/local/lib/ruby/3.3.0/benchmark.rb:313:in `realtime'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/exe/rubocop:19:in `<top (required)>'
/app/vendor/bundle/ruby/3.3.0/bin/rubocop:25:in `load'
/app/vendor/bundle/ruby/3.3.0/bin/rubocop:25:in `<main>'
/usr/local/bundle/bin/ruby_executable_hooks:22:in `eval'
/usr/local/bundle/bin/ruby_executable_hooks:22:in `<main>'
mark-young-atg commented 2 months ago

Thank you very much @grosser. I think that was a sensible move.

mark-young-atg commented 2 months ago

@grosser although I think it might be sensible to fix forward by releasing v1.26.1 which reverts that commit. The reason being I'm still finding that bundle update and bundle install try to give me v1.26.0, possibly due to the gem still being installed locally on my laptop. Obviously I know to remove it with gem uninstall parallel:1.26.0 but this might catch out others and CI/CD systems.

grosser commented 2 months ago

Kk, will do

On Thu, Aug 8, 2024, 4:57 PM mark-young-atg @.***> wrote:

@grosser https://github.com/grosser although I think it might be sensible to fix forward by releasing v1.26.1 which reverts that commit. The reason being I'm still finding that bundle update and bundle install try to give me v1.26.0, possibly due to the gem still being installed locally on my laptop. Obviously I know to remove it with gem uninstall parallel:1.26.0 but this might catch out others and CI/CD systems.

— Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/349#issuecomment-2276039032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACYZ2VS2P65OEIKZCHBH3ZQOBOPAVCNFSM6AAAAABMGSC452VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZWGAZTSMBTGI . You are receiving this because you were mentioned.Message ID: @.***>

grosser commented 2 months ago

1.26.1 is out

eregon commented 2 months ago

Concurrent.available_processor_count is now fixed upstream and released: https://github.com/ruby-concurrency/concurrent-ruby/pull/1060 Sorry for the bug!

grosser commented 2 months ago

re-released as v1.26.2 🤞

honeyankit commented 2 months ago

@grosser : I am still getting negative array size error. 

negative array size
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `initialize'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `new'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `create_workers'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:520:in `work_in_processes'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:291:in `map'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:235:in `each'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:103:in `warm_cache'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:76:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:52:in `with_redirect'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:25:in `execute_runner'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:17:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command.rb:11:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/environment.rb:18:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:122:in `run_command'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:129:in `execute_runners'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:51:in `block in run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:81:in `profile_if_needed'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:43:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in `block in <top (required)>'
/usr/share/rbenv/versions/3.3.4/lib/ruby/3.3.0/benchmark.rb:313:in `realtime'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in `<top (required)>'
/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in `load'
/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in `<top (required)>'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `load'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:23:in `run'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:492:in `exec'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:34:in `dispatch'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:28:in `start'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:37:in `block in <top (required)>'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `load'
/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `<main>'

Edit: I had to manually update concurrent-ruby to 1.3.4 to fix the issue based on the comment here

grosser commented 2 months ago

What is your concurrent-ruby version ?

On Thu, Aug 15, 2024, 4:27 AM Ankit Honey @.***> wrote:

@grosser https://github.com/grosser : I am still getting negative array size error.

negative array size /home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in initialize'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:innew'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in create_workers'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:520:inwork_in_processes' /home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:291:in map'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:235:ineach'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:103:in warm_cache'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:76:inrun' /home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:26:in block in execute_runner'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:52:inwith_redirect'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:25:in execute_runner'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:17:inrun' /home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command.rb:11:in run'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/environment.rb:18:inrun'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:122:in run_command'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:129:inexecute_runners' /home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:51:in block in run'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:81:inprofile_if_needed'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:43:in run'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:inblock in <top (required)>' /usr/share/rbenv/versions/3.3.4/lib/ruby/3.3.0/benchmark.rb:313:in realtime'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in<top (required)>'/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in load'/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in<top (required)>' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in load'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:inkernel_load'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:23:in run'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:492:inexec' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:ininvoke_command'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:34:indispatch' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:28:instart'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:37:in block in <top (required)>'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117:inwith_friendly_errors' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:29:in <top (required)>'/usr/share/rbenv/versions/3.3.4/bin/bundle:25:inload'/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `

'

— Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/349#issuecomment-2290381980, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACYZ7N3UPPO2ZEYKCMAADZRQGX5AVCNFSM6AAAAABMGSC452VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGM4DCOJYGA . You are receiving this because you were mentioned.Message ID: @.***>

grosser commented 2 months ago

... and try updating to latest bundle update concurrent-ruby and see if that fixes it 🤞

VitaliySerov commented 2 months ago

Shouldn't concurrent-ruby v1.3.4 set as minumal version of dependcy of parallel in that case? or rubocop (not sure where the problem comes from)

eregon commented 2 months ago

A simple solution could be to check if Concurrent.available_processor_count returns <= 0 and if so use Etc.nprocessors. Or to add concurrent-ruby as a gem dependency to ensure it's always >= 1.3.4.

grosser commented 2 months ago

v1.26.3.