Closed mark-young-atg closed 2 months ago
As a further comment on this, I checked our Amazon AWS instance and the use of Concurrent.available_processor_count.floor
looks as though it will cause problems by returning a negative number. This is what I see on our ECS instance:
irb(main):001> Concurrent.available_processor_count.floor
=> -1
irb(main):002> Etc.nprocessors
=> 2
yanked 1.26.0
On Thu, Aug 8, 2024 at 4:20 PM mark-young-atg @.***> wrote:
As a further comment on this, I checked our Amazon AWS instance and the use of Concurrent.available_processor_count.floor looks as though it will cause problems by returning a negative number. This is what I see on our ECS instance:
irb(main):001> Concurrent.available_processor_count.floor => -1 irb(main):002> Etc.nprocessors => 2
— Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/349#issuecomment-2275953547, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACYZYBFXYCNBCDKSZIWBLZQN5CXAVCNFSM6AAAAABMGSC452VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZVHE2TGNJUG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Same issue
bundle exec rubocop --color
negative array size
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `initialize'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `new'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:581:in `create_workers'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:520:in `work_in_processes'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:291:in `map'
/app/vendor/bundle/ruby/3.3.0/gems/parallel-1.26.0/lib/parallel.rb:235:in `each'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:103:in `warm_cache'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/runner.rb:76:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:52:in `with_redirect'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:25:in `execute_runner'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command/execute_runner.rb:17:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/command.rb:11:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli/environment.rb:18:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:122:in `run_command'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:129:in `execute_runners'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:51:in `block in run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:81:in `profile_if_needed'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/lib/rubocop/cli.rb:43:in `run'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/exe/rubocop:19:in `block in <top (required)>'
/usr/local/lib/ruby/3.3.0/benchmark.rb:313:in `realtime'
/app/vendor/bundle/ruby/3.3.0/gems/rubocop-1.65.1/exe/rubocop:19:in `<top (required)>'
/app/vendor/bundle/ruby/3.3.0/bin/rubocop:25:in `load'
/app/vendor/bundle/ruby/3.3.0/bin/rubocop:25:in `<main>'
/usr/local/bundle/bin/ruby_executable_hooks:22:in `eval'
/usr/local/bundle/bin/ruby_executable_hooks:22:in `<main>'
Thank you very much @grosser. I think that was a sensible move.
@grosser although I think it might be sensible to fix forward by releasing v1.26.1 which reverts that commit. The reason being I'm still finding that bundle update
and bundle install
try to give me v1.26.0, possibly due to the gem still being installed locally on my laptop. Obviously I know to remove it with gem uninstall parallel:1.26.0
but this might catch out others and CI/CD systems.
Kk, will do
On Thu, Aug 8, 2024, 4:57 PM mark-young-atg @.***> wrote:
@grosser https://github.com/grosser although I think it might be sensible to fix forward by releasing v1.26.1 which reverts that commit. The reason being I'm still finding that bundle update and bundle install try to give me v1.26.0, possibly due to the gem still being installed locally on my laptop. Obviously I know to remove it with gem uninstall parallel:1.26.0 but this might catch out others and CI/CD systems.
— Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/349#issuecomment-2276039032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACYZ2VS2P65OEIKZCHBH3ZQOBOPAVCNFSM6AAAAABMGSC452VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZWGAZTSMBTGI . You are receiving this because you were mentioned.Message ID: @.***>
1.26.1 is out
Concurrent.available_processor_count
is now fixed upstream and released: https://github.com/ruby-concurrency/concurrent-ruby/pull/1060
Sorry for the bug!
re-released as v1.26.2 🤞
@grosser : I am still getting negative array size
error.
negative array size
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `initialize'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `new'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in `create_workers'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:520:in `work_in_processes'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:291:in `map'
/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:235:in `each'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:103:in `warm_cache'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:76:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:26:in `block in execute_runner'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:52:in `with_redirect'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:25:in `execute_runner'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:17:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command.rb:11:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/environment.rb:18:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:122:in `run_command'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:129:in `execute_runners'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:51:in `block in run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:81:in `profile_if_needed'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:43:in `run'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in `block in <top (required)>'
/usr/share/rbenv/versions/3.3.4/lib/ruby/3.3.0/benchmark.rb:313:in `realtime'
/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in `<top (required)>'
/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in `load'
/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in `<top (required)>'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `load'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in `kernel_load'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:23:in `run'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:492:in `exec'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:34:in `dispatch'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:28:in `start'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:37:in `block in <top (required)>'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:29:in `<top (required)>'
/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `load'
/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `<main>'
Edit: I had to manually update concurrent-ruby
to 1.3.4
to fix the issue based on the comment here
What is your concurrent-ruby version ?
On Thu, Aug 15, 2024, 4:27 AM Ankit Honey @.***> wrote:
@grosser https://github.com/grosser : I am still getting negative array size error.
negative array size /home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in
initialize'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:in
new'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:581:increate_workers'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:520:in
work_in_processes' /home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:291:inmap'/home/dependabot/.local/share/gem/ruby/gems/parallel-1.26.2/lib/parallel.rb:235:in
each'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:103:inwarm_cache'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/runner.rb:76:in
run' /home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:26:inblock in execute_runner'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:52:in
with_redirect'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:25:inexecute_runner'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command/execute_runner.rb:17:in
run' /home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/command.rb:11:inrun'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli/environment.rb:18:in
run'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:122:inrun_command'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:129:in
execute_runners' /home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:51:inblock in run'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:81:in
profile_if_needed'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/lib/rubocop/cli.rb:43:inrun'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in
block in <top (required)>' /usr/share/rbenv/versions/3.3.4/lib/ruby/3.3.0/benchmark.rb:313:inrealtime'/home/dependabot/.local/share/gem/ruby/gems/rubocop-1.65.0/exe/rubocop:19:in
<top (required)>'/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:inload'/home/dependabot/.local/share/gem/ruby/bin/rubocop:25:in
<top (required)>' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:inload'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:58:in
kernel_load'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli/exec.rb:23:inrun'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:492:in
exec' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/command.rb:27:inrun'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in
invoke_command'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor.rb:392:indispatch'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:34:in
dispatch' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/vendor/thor/lib/thor/base.rb:485:instart'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/cli.rb:28:in
start'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:37:inblock in <top (required)>'/home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/lib/bundler/friendly_errors.rb:117:in
with_friendly_errors' /home/dependabot/.local/share/gem/ruby/gems/bundler-2.4.18/exe/bundle:29:in<top (required)>'/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in
load'/usr/share/rbenv/versions/3.3.4/bin/bundle:25:in `' — Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/349#issuecomment-2290381980, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAACYZ7N3UPPO2ZEYKCMAADZRQGX5AVCNFSM6AAAAABMGSC452VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGM4DCOJYGA . You are receiving this because you were mentioned.Message ID: @.***>
... and try updating to latest bundle update concurrent-ruby
and see if that fixes it 🤞
Shouldn't concurrent-ruby
v1.3.4 set as minumal version of dependcy of parallel in that case?
or rubocop (not sure where the problem comes from)
A simple solution could be to check if Concurrent.available_processor_count
returns <= 0 and if so use Etc.nprocessors
.
Or to add concurrent-ruby as a gem dependency to ensure it's always >= 1.3.4.
v1.26.3.
Our github action to run rubocop started failing when parallel was upgraded from 1.25.1 to 1.26.0
This is the error seen
I presume this is because of the following change that uses
Concurrent.available_processor_count.floor
to determine the number of available processors if the gemconcurrent-ruby
is installed, which in my case it is.I can see that I can use the environment variable
PARALLEL_PROCESSOR_COUNT
to override this behaviour for my configuration but I think there is an issue here to be understood and mitigated around the behaviour in github actions. Whilst I appreciate the incorrect value may come from concurrent-ruby, I think perhaps that parallel should handle a negative value in a different way. Perhaps by failing back to usingEtc.nprocessors
as the previous version did.I have not created a PR for this because I felt it needed discussion and knowledge from those that understand
parallel
vastly more than I do.