search

oracle / truffleruby

A high performance implementation of the Ruby programming language, built on GraalVM.
https://www.graalvm.org/ruby/
Other
3.02k stars 185 forks source link

bundle install with many dependencies causes OutOfMemoryError #1824

Closed deepj closed 4 years ago

deepj commented 4 years ago 
truffleruby 19.3.0, like ruby 2.6.2, GraalVM CE Native [x86_64-darwin]

I wanted to try one old project to run on TruffleRuby but I was stopped while bundle install when I faced to multiple fails. At least on my machine this happens always when one of the following errors is occured.

Unfortunately, I can't share the Gemfile and Gemfile.lock publicly, but I can share it privately.

Errors:

$ bundle
The git source `git://github.com/activeadmin/activeadmin.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/activeadmin/inherited_resources.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
Fetching gem metadata from https://rubygems.org/............
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies.....................................................................................................................................................................
Using rake 12.0.0
Using concurrent-ruby 1.0.4
Using i18n 0.7.0
Using minitest 5.10.1
Using thread_safe 0.3.5
Using tzinfo 1.2.2
Using activesupport 5.0.1
Using builder 3.2.2
Using erubis 2.7.0
Using mini_portile2 2.1.0
Using nokogiri 1.7.0.1
Using rails-dom-testing 2.0.2
Using loofah 2.0.3
Using rails-html-sanitizer 1.0.3
Using actionview 5.0.1
Using rack 2.0.1
Using rack-test 0.6.3
Using actionpack 5.0.1
Using nio4r 1.2.1
Using websocket-extensions 0.1.2
Using websocket-driver 0.6.4
Using actioncable 5.0.1
Using globalid 0.3.7
Using activejob 5.0.1
Using mime-types-data 3.2016.0521
Using mime-types 3.1
Using mail 2.6.4
Using actionmailer 5.0.1
Using activemodel 5.0.1
Using case_transform 0.2
Using jsonapi-renderer 0.1.2
Using active_model_serializers 0.10.5
Using arbre 1.1.1
Using sass 3.4.23
Using thor 0.19.4
Using bourbon 4.2.7
Using coffee-script-source 1.12.2
Using execjs 2.7.0
Using coffee-script 2.4.1
Using method_source 0.8.2
Using railties 5.0.1
Using coffee-rails 4.2.1
Using formtastic 3.1.4
Using formtastic_i18n 0.6.0
Using has_scope 0.7.0
Using responders 2.3.0
Using inherited_resources 1.6.0 from git://github.com/activeadmin/inherited_resources.git (at master@6707bf7)
Using jquery-rails 4.2.2
Using jquery-ui-rails 5.0.5
Using kaminari 0.17.0
Using arel 7.1.4
Using activerecord 5.0.1
Using polyamorous 1.3.1
Using ransack 1.8.2
Using sprockets 3.7.1
Using sprockets-rails 3.2.0
Using tilt 2.0.5
Using sass-rails 5.0.6
Using activeadmin 1.0.0.pre4 from git://github.com/activeadmin/activeadmin.git (at master@cf291ad)
Using activemodel-serializers-xml 1.0.1
Using bundler 1.17.2
Using rails 5.0.1
Using request_store 1.3.2
Using acts_as_tenant 0.4.0
Using public_suffix 2.0.5
Using addressable 2.5.0
Using gyoku 1.3.1
Using akami 1.3.1
Using ast 2.3.0
Using encryptor 3.0.0
Using attr_encrypted 3.0.3
Using authority 3.2.2
Using aws-sigv4 1.0.0
Using jmespath 1.3.1
Using aws-sdk-core 2.6.44
Using aws-sdk-resources 2.6.44
Using aws-sdk 2.6.44
Using bcrypt 3.1.11
Using benchmark-ips 2.7.2
Using best_in_place 3.1.0
Fetching brakeman 3.4.1
Installing brakeman 3.4.1
Fetching uniform_notifier 1.10.0
Installing uniform_notifier 1.10.0
Fetching bullet 5.5.0
Installing bullet 5.5.0
Fetching bundler-audit 0.5.0
Installing bundler-audit 0.5.0
Fetching byebug 9.0.6
Installing byebug 9.0.6 with native extensions
Fetching xpath 2.0.0
Installing xpath 2.0.0
Fetching capybara 2.11.0
Installing capybara 2.11.0
Fetching capybara-email 2.5.0
Installing capybara-email 2.5.0
Using choice 0.2.0
Fetching chunky_png 1.3.8
Installing chunky_png 1.3.8
Fetching coderay 1.1.1
Installing coderay 1.1.1
Fetching connection_pool 2.2.1
Installing connection_pool 2.2.1
Fetching database_cleaner 1.5.3
Installing database_cleaner 1.5.3
Using debug_inspector 0.0.2
Fetching get_process_mem 0.2.1
Installing get_process_mem 0.2.1
Fetching heapy 0.1.2
Installing heapy 0.1.2
Fetching memory_profiler 0.9.7
Installing memory_profiler 0.9.7
Fetching derailed_benchmarks 1.3.2
Installing derailed_benchmarks 1.3.2
Using orm_adapter 0.5.0
Fetching warden 1.2.6
Installing warden 1.2.6
Fetching devise 4.2.0
Installing devise 4.2.0
Fetching devise-encryptable 0.2.0
Installing devise-encryptable 0.2.0
Fetching devise-i18n 0.12.1
Installing devise-i18n 0.12.1
Fetching rotp 2.1.2
Installing rotp 2.1.2
Fetching devise-two-factor 3.0.0
Installing devise-two-factor 3.0.0
Fetching diff-lcs 1.2.5
Installing diff-lcs 1.2.5
Fetching docsplit 0.7.6
Installing docsplit 0.7.6
Fetching doorkeeper 4.2.0
Installing doorkeeper 4.2.0
Fetching dotenv 2.1.1
Installing dotenv 2.1.1
Fetching dotenv-rails 2.1.1
Installing dotenv-rails 2.1.1
Fetching draper 3.0.0.pre1
Installing draper 3.0.0.pre1
Fetching htmlentities 4.3.4
Installing htmlentities 4.3.4
Using launchy 2.4.3
Fetching email_spec 2.1.0
Installing email_spec 2.1.0
Fetching enum_help 0.0.16
Installing enum_help 0.0.16
Fetching factory_girl 4.8.0
Installing factory_girl 4.8.0
Fetching fakes3 0.2.5
Installing fakes3 0.2.5
Fetching ffi 1.9.14
Installing ffi 1.9.14 with native extensions
Fetching flamegraph 0.9.5
Installing flamegraph 0.9.5
Fetching font-awesome-rails 4.7.0.1
Installing font-awesome-rails 4.7.0.1
Fetching foreman 0.82.0
Installing foreman 0.82.0
Fetching foundation-rails 5.5.3.2
Installing foundation-rails 5.5.3.2
Fetching foundation_rails_helper 2.0.0
Installing foundation_rails_helper 2.0.0
Fetching griddler 1.3.1
Installing griddler 1.3.1
Fetching socksify 1.7.0
Installing socksify 1.7.0
Fetching httpi 2.4.2
Installing httpi 2.4.2
Fetching intercom-rails 0.3.4
Installing intercom-rails 0.3.4
Fetching turbolinks 2.5.3
Installing turbolinks 2.5.3
Fetching jquery-turbolinks 2.1.0
Installing jquery-turbolinks 2.1.0
Fetching json-schema 2.7.0
Installing json-schema 2.7.0
Fetching letter_opener 1.4.1
Installing letter_opener 1.4.1
Fetching letter_opener_web 1.2.3
Installing letter_opener_web 1.2.3
Fetching libv8 5.3.332.38.3 (x86_64-darwin-16)
Installing libv8 5.3.332.38.3 (x86_64-darwin-16)
[truffle] opt fail         MatchData#[] (builtin)                                      |Reason java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.
java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.

^[fish: 'bundle' terminated by signal SIGKILL (Forced quit)
$ bundle
The git source `git://github.com/activeadmin/activeadmin.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/activeadmin/inherited_resources.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
Fetching gem metadata from https://rubygems.org/....[truffle] opt fail         Truffle::RegexpOperations.last_match resource:/truffleruby/core/truffle/regexp_operations.rb:24 <split-2c0d7f5>|Reason org.graalvm.compiler.graph.GraalGraphError: org.graalvm.compiler.debug.GraalError: should not reach here: node is not LIRLowerable: 32|NewFrame
    at node: 32|NewFrame
org.graalvm.compiler.graph.GraalGraphError: org.graalvm.compiler.debug.GraalError: should not reach here: node is not LIRLowerable: 32|NewFrame
    at node: 32|NewFrame
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.doBlock(NodeLIRBuilder.java:378)
    at org.graalvm.compiler.core.LIRGenerationPhase.emitBlock(LIRGenerationPhase.java:85)
    at org.graalvm.compiler.core.LIRGenerationPhase.run(LIRGenerationPhase.java:75)
    at org.graalvm.compiler.core.LIRGenerationPhase.run(LIRGenerationPhase.java:46)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:117)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:109)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLIR0(LIRCompilerBackend.java:155)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLIR(LIRCompilerBackend.java:113)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitBackEnd(LIRCompilerBackend.java:83)
    at org.graalvm.compiler.core.target.Backend.emitBackEnd(Backend.java:279)
    at org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:145)
    at org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:129)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.compilePEGraph(TruffleCompilerImpl.java:528)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.compileAST(TruffleCompilerImpl.java:470)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl$TruffleCompilationWrapper.performCompilation(TruffleCompilerImpl.java:644)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl$TruffleCompilationWrapper.performCompilation(TruffleCompilerImpl.java:586)
    at org.graalvm.compiler.core.CompilationWrapper.run(CompilationWrapper.java:179)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.actuallyCompile(TruffleCompilerImpl.java:299)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.doCompile(TruffleCompilerImpl.java:267)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime.compileImpl(GraalTruffleRuntime.java:710)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime.doCompile(GraalTruffleRuntime.java:691)
    at org.graalvm.compiler.truffle.runtime.BackgroundCompileQueue$Request.run(BackgroundCompileQueue.java:83)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.graalvm.compiler.truffle.runtime.BackgroundCompileQueue$TruffleCompilerThreadFactory$1.run(BackgroundCompileQueue.java:176)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:479)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: org.graalvm.compiler.debug.GraalError: should not reach here: node is not LIRLowerable: 32|NewFrame
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.emitNode(NodeLIRBuilder.java:484)
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.doRoot(NodeLIRBuilder.java:473)
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.doBlock(NodeLIRBuilder.java:376)
    ... 29 more                                                                         
$ bundle
The git source `git://github.com/activeadmin/activeadmin.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
The git source `git://github.com/activeadmin/inherited_resources.git` uses the `git` protocol, which transmits data without encryption. Disable this warning with `bundle config git.allow_insecure true`, or switch to the `https` protocol to keep your data secure.
Fetching gem metadata from https://rubygems.org/.............
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies.....................................................................................................................................................................
Using rake 12.0.0
Using concurrent-ruby 1.0.4
Using i18n 0.7.0
Using minitest 5.10.1
Using thread_safe 0.3.5
Using tzinfo 1.2.2
Using activesupport 5.0.1
Using builder 3.2.2
Using erubis 2.7.0
Using mini_portile2 2.1.0
Using nokogiri 1.7.0.1
Using rails-dom-testing 2.0.2
Using loofah 2.0.3
Using rails-html-sanitizer 1.0.3
Using actionview 5.0.1
Using rack 2.0.1
Using rack-test 0.6.3
Using actionpack 5.0.1
Using nio4r 1.2.1
Using websocket-extensions 0.1.2
Using websocket-driver 0.6.4
Using actioncable 5.0.1
Using globalid 0.3.7
Using activejob 5.0.1
Using mime-types-data 3.2016.0521
Using mime-types 3.1
Using mail 2.6.4
Using actionmailer 5.0.1
Using activemodel 5.0.1
Using case_transform 0.2
Using jsonapi-renderer 0.1.2
Using active_model_serializers 0.10.5
Using arbre 1.1.1
Using sass 3.4.23
Using thor 0.19.4
Using bourbon 4.2.7
Using coffee-script-source 1.12.2
Using execjs 2.7.0
Using coffee-script 2.4.1
Using method_source 0.8.2
Using railties 5.0.1
Using coffee-rails 4.2.1
Using formtastic 3.1.4
Using formtastic_i18n 0.6.0
Using has_scope 0.7.0
Using responders 2.3.0
Using inherited_resources 1.6.0 from git://github.com/activeadmin/inherited_resources.git (at master@6707bf7)
Using jquery-rails 4.2.2
Using jquery-ui-rails 5.0.5
Using kaminari 0.17.0
Using arel 7.1.4
Using activerecord 5.0.1
Using polyamorous 1.3.1
Using ransack 1.8.2
Using sprockets 3.7.1
Using sprockets-rails 3.2.0
Using tilt 2.0.5
Using sass-rails 5.0.6
Using activeadmin 1.0.0.pre4 from git://github.com/activeadmin/activeadmin.git (at master@cf291ad)
Using activemodel-serializers-xml 1.0.1
Using bundler 1.17.2
Using rails 5.0.1
Using request_store 1.3.2
Using acts_as_tenant 0.4.0
Using public_suffix 2.0.5
Using addressable 2.5.0
Using gyoku 1.3.1
Using akami 1.3.1
Using ast 2.3.0
Using encryptor 3.0.0
Using attr_encrypted 3.0.3
Using authority 3.2.2
Using aws-sigv4 1.0.0
Using jmespath 1.3.1
Using aws-sdk-core 2.6.44
Using aws-sdk-resources 2.6.44
Using aws-sdk 2.6.44
Using bcrypt 3.1.11
Using benchmark-ips 2.7.2
Using best_in_place 3.1.0
Using brakeman 3.4.1
Using uniform_notifier 1.10.0
Using bullet 5.5.0
Using bundler-audit 0.5.0
Using byebug 9.0.6
Using xpath 2.0.0
Using capybara 2.11.0
Using capybara-email 2.5.0
Using choice 0.2.0
Using chunky_png 1.3.8
Using coderay 1.1.1
Using connection_pool 2.2.1
Using database_cleaner 1.5.3
Using debug_inspector 0.0.2
Using get_process_mem 0.2.1
Using heapy 0.1.2
Using memory_profiler 0.9.7
Using derailed_benchmarks 1.3.2
Using orm_adapter 0.5.0
Using warden 1.2.6
Using devise 4.2.0
Using devise-encryptable 0.2.0
Using devise-i18n 0.12.1
Using rotp 2.1.2
Using devise-two-factor 3.0.0
Using diff-lcs 1.2.5
Using docsplit 0.7.6
Using doorkeeper 4.2.0
Using dotenv 2.1.1
Using dotenv-rails 2.1.1
Using draper 3.0.0.pre1
Using htmlentities 4.3.4
Using launchy 2.4.3
Using email_spec 2.1.0
Using enum_help 0.0.16
Using factory_girl 4.8.0
Using fakes3 0.2.5
Using ffi 1.9.14
Using flamegraph 0.9.5
Using font-awesome-rails 4.7.0.1
Using foreman 0.82.0
Using foundation-rails 5.5.3.2
Using foundation_rails_helper 2.0.0
Using griddler 1.3.1
Using socksify 1.7.0
Using httpi 2.4.2
Using intercom-rails 0.3.4
Using turbolinks 2.5.3
Using jquery-turbolinks 2.1.0
Using json-schema 2.7.0
Using letter_opener 1.4.1
Using letter_opener_web 1.2.3
Fetching libv8 5.3.332.38.3 (x86_64-darwin-16)
Installing libv8 5.3.332.38.3 (x86_64-darwin-16)
[truffle] opt fail         Array#map! (builtin) <split-2da95667>                       |Reason java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.
java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.

^Cfish: 'bundle' terminated by signal SIGKILL (Forced quit)
eregon commented 4 years ago

Thank you for the bug report. Both the 1st and 3rd logs seem to be OutOfMemoryError. I would think because of some leak or using excessive memory for representing some of the data Bundler uses.

The OutOfMemoryError seems to happen for libv8 which is a very large gem: 190MB on macOS for the version you're using: https://rubygems.org/gems/libv8/versions.

Could you try bundle install without the libv8 gem? We'll try to reproduce the OutOfMemoryError.

The second error is a compilation error:

[truffle] opt fail         Truffle::RegexpOperations.last_match resource:/truffleruby/core/truffle/regexp_operations.rb:24 <split-2c0d7f5>|Reason org.graalvm.compiler.graph.GraalGraphError: org.graalvm.compiler.debug.GraalError: should not reach here: node is not LIRLowerable: 32|NewFrame
    at node: 32|NewFrame
org.graalvm.compiler.graph.GraalGraphError: org.graalvm.compiler.debug.GraalError: should not reach here: node is not LIRLowerable: 32|NewFrame
    at node: 32|NewFrame
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.doBlock(NodeLIRBuilder.java:378)
    at org.graalvm.compiler.core.LIRGenerationPhase.emitBlock(LIRGenerationPhase.java:85)
    at org.graalvm.compiler.core.LIRGenerationPhase.run(LIRGenerationPhase.java:75)
    at org.graalvm.compiler.core.LIRGenerationPhase.run(LIRGenerationPhase.java:46)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:117)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:109)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLIR0(LIRCompilerBackend.java:155)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLIR(LIRCompilerBackend.java:113)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitBackEnd(LIRCompilerBackend.java:83)
    at org.graalvm.compiler.core.target.Backend.emitBackEnd(Backend.java:279)
    at org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:145)
    at org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:129)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.compilePEGraph(TruffleCompilerImpl.java:528)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.compileAST(TruffleCompilerImpl.java:470)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl$TruffleCompilationWrapper.performCompilation(TruffleCompilerImpl.java:644)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl$TruffleCompilationWrapper.performCompilation(TruffleCompilerImpl.java:586)
    at org.graalvm.compiler.core.CompilationWrapper.run(CompilationWrapper.java:179)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.actuallyCompile(TruffleCompilerImpl.java:299)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.doCompile(TruffleCompilerImpl.java:267)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime.compileImpl(GraalTruffleRuntime.java:710)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime.doCompile(GraalTruffleRuntime.java:691)
    at org.graalvm.compiler.truffle.runtime.BackgroundCompileQueue$Request.run(BackgroundCompileQueue.java:83)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.graalvm.compiler.truffle.runtime.BackgroundCompileQueue$TruffleCompilerThreadFactory$1.run(BackgroundCompileQueue.java:176)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:479)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: org.graalvm.compiler.debug.GraalError: should not reach here: node is not LIRLowerable: 32|NewFrame
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.emitNode(NodeLIRBuilder.java:484)
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.doRoot(NodeLIRBuilder.java:473)
    at org.graalvm.compiler.core.gen.NodeLIRBuilder.doBlock(NodeLIRBuilder.java:376)
    ... 29 more

That should let the process continue, although it might be quite a bit slower.

eregon commented 4 years ago

I split the compilation error in a separate issue: #1825

deepj commented 4 years ago

It seems libv8 caused the OutOfMemoryError error. It was a dependency of mini_racer.

But the whole bundle install is extremely slow and I can't use my machine during installation due to throttling.

eregon commented 4 years ago

@deepj To get an idea, how much RAM do you have available (free -m)?

deepj commented 4 years ago

@eregon unfortunately, there is no such command on macOS. Any way, bundle install is significant slower with TruffleRuby 19.3.0 on my machine

eregon commented 4 years ago

What's the available memory in Activity Monitor then?

and I can't use my machine during installation due to throttling.

Maybe it's swapping memory, or do you mean something else?

deepj commented 4 years ago

@eregon I tried truffleruby 20.1.0-dev-a575d7bc, like ruby 2.6.5, GraalVM CE Native [x86_64-darwin]

And I got it again:

Fetching rails-html-sanitizer 1.3.0
Installing rails-html-sanitizer 1.3.0
Fetching actionview 6.0.2.1
Installing actionview 6.0.2.1
Fetching rack 2.1.2
Installing rack 2.1.2
Fetching rack-test 1.1.0
[truffle] opt fail         Gem::Package::TarHeader.from ~/.rubies/truffleruby-20.1.0-dev/lib/mri/rubygems/package/tar_header.rb:102|ASTSize           252 |Reason java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.
java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.

Installing rack-test 1.1.0
Fetching actionpack 6.0.2.1
Installing actionpack 6.0.2.1
Fetching nio4r 2.5.2
Installing nio4r 2.5.2 with native extensions

This happens mostly when there is 150+ gems to be installed and all it's installed from the scratch

eregon commented 4 years ago

@deepj Could you run with --vm.XX:+PrintGCSummary, for instance:

ruby --vm.XX:+PrintGCSummary -S bundle install

That should show the heap sizes chosen by Native Image.

deepj commented 4 years ago

@eregon here is the result. I tried it during the last two days but the problem didn't occur again (I always remove all previous gems).

PrintGCSummary: YoungGenerationSize: 1073741824
PrintGCSummary: MinimumHeapSize: 2147483648
PrintGCSummary: MaximumHeapSize: 6871947600
PrintGCSummary: AlignedChunkSize: 1048576
PrintGCSummary: CollectedTotalChunkBytes: 105803262600
PrintGCSummary: CollectedTotalObjectBytes: 105079597760
PrintGCSummary: AllocatedNormalChunkBytes: 110604650616
PrintGCSummary: AllocatedNormalObjectBytes: 109841922176
PrintGCSummary: IncrementalGCCount: 88
PrintGCSummary: IncrementalGCNanos: 167289109886
PrintGCSummary: CompleteGCCount: 14
PrintGCSummary: CompleteGCNanos: 195090777187
PrintGCSummary: GCNanos: 362379887073
PrintGCSummary: TotalNanos: 1611073956986
PrintGCSummary: GCLoadPercent: 22
deepj commented 4 years ago

My laptop has only 8 GB RAM (+ something is reserved for iGPU)

deepj commented 4 years ago

This happened when I installed dependencies of https://github.com/chatwoot/chatwoot

Installing rack 2.1.2
Fetching rack-test 1.1.0
Installing rack-test 1.1.0
Fetching actionpack 6.0.2.1
[truffle] opt fail         Gem::Package::TarHeader.from ~/.rubies/truffleruby-20.1.0-dev/lib/mri/rubygems/package/tar_header.rb:102|ASTSize           252 |Reason org.graalvm.compiler.core.common.PermanentBailoutException: Out-of-memory during live set allocation of size 88735
org.graalvm.compiler.core.common.PermanentBailoutException: Out-of-memory during live set allocation of size 88735
    at org.graalvm.compiler.lir.alloc.lsra.LinearScanLifetimeAnalysisPhase.computeLocalLiveSets(LinearScanLifetimeAnalysisPhase.java:262)
    at org.graalvm.compiler.lir.alloc.lsra.LinearScanLifetimeAnalysisPhase.run(LinearScanLifetimeAnalysisPhase.java:93)
    at org.graalvm.compiler.lir.alloc.lsra.LinearScanAllocationPhase.apply(LinearScanAllocationPhase.java:53)
    at org.graalvm.compiler.lir.alloc.lsra.LinearScanAllocationPhase.apply(LinearScanAllocationPhase.java:46)
    at org.graalvm.compiler.lir.alloc.lsra.LinearScan.allocate(LinearScan.java:691)
    at org.graalvm.compiler.lir.alloc.lsra.LinearScanPhase.run(LinearScanPhase.java:42)
Installing actionpack 6.0.2.1
    at org.graalvm.compiler.lir.alloc.lsra.LinearScanPhase.run(LinearScanPhase.java:35)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:117)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:109)
    at org.graalvm.compiler.lir.phases.LIRPhaseSuite.run(LIRPhaseSuite.java:98)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:117)
    at org.graalvm.compiler.lir.phases.LIRPhase.apply(LIRPhase.java:109)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLowLevel(LIRCompilerBackend.java:180)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLIR0(LIRCompilerBackend.java:160)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitLIR(LIRCompilerBackend.java:113)
    at org.graalvm.compiler.core.gen.LIRCompilerBackend.emitBackEnd(LIRCompilerBackend.java:83)
    at org.graalvm.compiler.core.target.Backend.emitBackEnd(Backend.java:279)
    at org.graalvm.compiler.core.GraalCompiler.compile(GraalCompiler.java:145)
Fetching nio4r 2.5.2
    at org.graalvm.compiler.core.GraalCompiler.compileGraph(GraalCompiler.java:129)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.compilePEGraph(TruffleCompilerImpl.java:539)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.compileAST(TruffleCompilerImpl.java:481)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl$TruffleCompilationWrapper.performCompilation(TruffleCompilerImpl.java:660)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl$TruffleCompilationWrapper.performCompilation(TruffleCompilerImpl.java:598)
    at org.graalvm.compiler.core.CompilationWrapper.run(CompilationWrapper.java:179)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.actuallyCompile(TruffleCompilerImpl.java:308)
    at org.graalvm.compiler.truffle.compiler.TruffleCompilerImpl.doCompile(TruffleCompilerImpl.java:259)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime.compileImpl(GraalTruffleRuntime.java:721)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime.doCompile(GraalTruffleRuntime.java:702)
    at org.graalvm.compiler.truffle.runtime.GraalTruffleRuntime$1.execute(GraalTruffleRuntime.java:777)
    at org.graalvm.compiler.truffle.runtime.BackgroundCompileQueue$RequestImpl.call(BackgroundCompileQueue.java:195)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.graalvm.compiler.truffle.runtime.BackgroundCompileQueue$TruffleCompilerThreadFactory$1.run(BackgroundCompileQueue.java:238)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:527)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
Caused by: java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.

Installing nio4r 2.5.2 with native extensions
Fetching websocket-extensions 0.1.4
PrintGCSummary: YoungGenerationSize: 1073741824
PrintGCSummary: MinimumHeapSize: 2147483648
PrintGCSummary: MaximumHeapSize: 6871947600
PrintGCSummary: AlignedChunkSize: 1048576
PrintGCSummary: CollectedTotalChunkBytes: 99840136016
PrintGCSummary: CollectedTotalObjectBytes: 99173061584
PrintGCSummary: AllocatedNormalChunkBytes: 101859271536
PrintGCSummary: AllocatedNormalObjectBytes: 101175348136
PrintGCSummary: IncrementalGCCount: 81
PrintGCSummary: IncrementalGCNanos: 48363886437
PrintGCSummary: CompleteGCCount: 13
PrintGCSummary: CompleteGCNanos: 101418291795
PrintGCSummary: GCNanos: 149782178232
PrintGCSummary: TotalNanos: 841578348840
PrintGCSummary: GCLoadPercent: 18
eregon commented 4 years ago

Are you using TruffleRuby 19.3 or the nightly builds (which one?) for the above output? I'd guess the nightly build since the young generation is 1GB. That last output is the same issue as https://github.com/oracle/truffleruby/issues/1893#issuecomment-582490912.

eregon commented 4 years ago

Ah, it's indeed the nightly, https://github.com/oracle/truffleruby/issues/1824#issuecomment-582004165 sorry, I missed that.

deepj commented 4 years ago

@eregon truffleruby 20.1.0-dev-a575d7bc, like ruby 2.6.5, GraalVM CE Native [x86_64-darwin]

deepj commented 4 years ago

Another fault

Fetching yard 0.9.24
Installing yard 0.9.24
[truffle] opt fail         IO#fsync resource:/truffleruby/core/io.rb:1660              |ASTSize            48 |Reason java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.
java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.

Fetching yard-junk 0.0.7
Installing yard-junk 0.0.7
deepj commented 4 years ago

@eregon maybe close this? Since using nighly builds I've never met with this. At least with last changes around Bundler

eregon commented 4 years ago

Right, the root cause seems to be the same as https://github.com/oracle/truffleruby/issues/1893