Closed D-system closed 2 months ago
Hey @D-system. A couple of question hopefully helping to narrow the issue down
Can you reproduce it with a freshly created rails application? On aarch64-linux
my basic checks works, as well as rails new foo && cd foo && bundle install && RAILS_ENV=production SECRET_KEY_BASE=1 bin/rails assets:clobber assets:precompile
using cimg/ruby:3.2.3-node
.
Can you reproduce the issue with Ruby 3.3.0?
Can you confirm if you're using jemalloc or not? Looks like the image is at least installing libjemalloc, I haven't checked if they build Ruby against it, or using LD_PRELOAD
, or using this up to the user. There have been various issues with jemalloc and mini_racer/v8. In case jemalloc is used, can you try without?
I don't think jemalloc is enabled:
$ ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lrt -lrt -ldl -lcrypt -lm -lpthread
Even by using the LD_PRELOAD
, it does not seems to work:
$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lrt -lrt -ldl -lcrypt -lm -lpthread
# Checking the directory
$ ls -l /usr/lib/x86_64-linux-gnu/libjemalloc*
-rw-r--r-- 1 root root 1119814 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.a
-rw-r--r-- 1 root root 1119970 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a
lrwxrwxrwx 1 root root 16 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.so -> libjemalloc.so.2
-rw-r--r-- 1 root root 744440 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
Let me see if I can make a new app and get the same error.
I didn't realized there was a more debug output:
I made a brand new Rails 6.1.7.7 app I got the same error with those changes:
Note: it does precompile the files then seg fault.
I also tried the same project with mini_racer 0.9. It works.
$ bundle ; ./bin/rails assets:clobber assets:precompile
Fetching gem metadata from https://rubygems.org/..........
Resolving dependencies...
Fetching libv8-node 18.19.0.0 (x86_64-linux) (was 21.7.2.0)
Installing libv8-node 18.19.0.0 (x86_64-linux) (was 21.7.2.0)
Fetching mini_racer 0.9.0 (was 0.12.0)
Installing mini_racer 0.9.0 (was 0.12.0) with native extensions
Bundle complete! 18 Gemfile dependencies, 84 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
I, [2024-04-23T19:04:10.030111 #10198] INFO -- : Removed /home/circleci/new_rails_app/public/assets
yarn install v1.22.19
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.37s.
I, [2024-04-23T19:04:11.476483 #10198] INFO -- : Writing /home/circleci/new_rails_app/public/assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js
I, [2024-04-23T19:04:11.477493 #10198] INFO -- : Writing /home/circleci/new_rails_app/public/assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js.gz
I, [2024-04-23T19:04:11.477738 #10198] INFO -- : Writing /home/circleci/new_rails_app/public/assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css
I, [2024-04-23T19:04:11.478602 #10198] INFO -- : Writing /home/circleci/new_rails_app/public/assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css.gz
$
It seems sprocket is used by default if no other assets pipeline exists.
The Gemfile without the comments
source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby '3.2.3'
gem 'rails', '~> 6.1.7', '>= 6.1.7.7'
gem 'sqlite3', '~> 1.4'
gem 'puma', '~> 5.0'
gem 'sass-rails', '>= 6'
gem 'turbolinks', '~> 5'
gem 'jbuilder', '~> 2.7'
gem "bootstrap", "~> 4.3" # <- the problem
gem "mini_racer", "~> 0.9.0" # <- added
gem 'bootsnap', '>= 1.4.4', require: false
group :development, :test do
gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
end
group :development do
gem 'web-console', '>= 4.1.0'
gem 'rack-mini-profiler', '~> 2.0'
gem 'listen', '~> 3.3'
gem 'spring'
end
group :test do
gem 'capybara', '>= 3.26'
gem 'selenium-webdriver', '>= 4.0.0.rc1'
gem 'webdrivers'
end
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]
@lloeki this looks urgent , should we roll out a release that puts us back at an earlier version while we work this out ?
@D-system does this repro in Linux as well
@SamSaffron it's all tested inside the CircleCI image linked above. So, it's an Ubuntu based linux on x86_64.
should we roll out a release that puts us back at an earlier version while we work this out ?
So the idea would be to have 0.12.1 be 0.9.0? That would be odd.
I'll try a repro.
The official ruby:3.2.3
Docker image does not have that problem.
Trying with cimg/ruby:3.2.3-node
on my local machine (instead of their data centers)
I'm not able to reproduce yet, can't spend much time on it right now though.
The product I'm working on in my day job is using docker images based on the ruby:3.3.0
and we have an extensive test suite with lots of JavaScript that is evaluated by mini_racer. It works with 0.12.0 on x86 and aarch64 (both linux) just fine, I also have no issues on macOS 14.4.1 (arm). Running directly on x86 without Docker in Github Actions is also fine.
🤔
I tried with cimg/ruby:3.2.3-node
as base image with --platform=x86_64
, I was not able to reproduce the problem.
eg: FROM --platform=x86_64 cimg/ruby:3.2.3-node
I'm not able to reproduce the issue in a local docker either.
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
LD_PRELOAD
wouldn't change RbConfig::CONFIG['MAINLIBS']
output, it'd just forcefully inject jemalloc symbols into it.
In any case it appears that RbConfig::CONFIG['MAINLIBS']
doesn't contain anything about jemalloc anyway, even when statically built against jemalloc.
I'm not able to reproduce the issue in a local docker either.
I couldn't reproduce it either inside that x86_64
image: test suite passes, even when artificially stressing libv8 by throwing copious amounts of quickly hacked JS code processing to it (granted it was a real quick test).
@D-system any chance you can create a container that repros this consistently. Then we can docker run <SOMETHING>
and have it crash?
I just wanted to add another case of this issue. I tried to update yesterday from 0.8 to 0.12 and started getting segfaults. I don't use docker or any version managers, just the system's 3.0.2p107 (the default ruby) on Ubuntu 22.04. This is on rails 7.0.8.1 with libv8 21.7.2.0. I've reverted to 0.8 and libv8 18.16.0.0 for now. I'm not using jemalloc (I have no idea what it is, and the mentioned libraries are not on my system). If you need any info or need anything tested let me know, but I don't want to figure out how to build or use docker images (have no interest in that).
I also saw it yesterday when upgrading from 0.9 to 0.12 both on Fedora 40 with ruby 3.3.0 and also on Ubuntu 22.04 with ruby 3.0.2p107.
Thanks for the additional reports. Is it Sprockets-related as well?
It was using rails with sprockets yes, specifically using https://rubygems.org/gems/rtlcss in the sprockets chain which uses mini_racer to run https://www.npmjs.com/package/rtlcss on CSS files.
I am using sprockets, but I had not checked whether it's involved or not. I first got these errors when rendering a js response to an ajax request, and also saw them when shutting down the server (running puma).
@SamSaffron I was not able to reproduce on my local host or local docker yesterday. I will give a new try tomorrow.
We were getting this on 0.9.0 -> 0.12.0 (production only) on Heroku on the '20' image with Ruby 3.3.0 and jemalloc and node v20.9.0
I did see this in a prod console, though may not be related
WARNING: V8 isolate was interrupted by Ruby, it can not be disposed and memory will not be reclaimed till the Ruby process exits.
and looking at prod crash logs
/app/vendor/bundle/ruby/3.3.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:183: [BUG] Segmentation fault at 0x0000000000000000
we're just using this to run embedded javascript
@fjl82
I am using sprockets, but I had not checked whether it's involved or not. I first got these errors when rendering a js response to an ajax request, and also saw them when shutting down the server (running puma).
Are you using puma in clustered mode? forking is known to cause some troubles with mini_racer/v8, see https://github.com/rubyjs/mini_racer#fork-safety.
I am indeed using puma in clustered mode, but it was working fine on 0.8.0, and starting puma in single mode gives me the same segfault in the same place. I have the suggested fork safety line in an initializer, and removing this makes no difference in any scenario.
I'm still not able to reproduce it in a local Docker even when I use the Docker image of the CI.
I continue to investigate on the CI directly. SSH and vi for the win!
Anyway, I track down the issue. On a brand new Rails 6.1 app:
autoprefixer-rails
has only execjs
as dependency.
In my very case, autoprefixer-rails
might have some code that seg fault in the CI only.
I don't know what else I could search.
@D-system I wish I had more to add here, but so much depends on a standalone repro. In theory most people use mini_racer in a very simple way, if you can get a crash then you can monkey patch eval to save what it is evaling into a file and then we can use that to cause the crash.
Once we have one we will very likely be able to sort this out quickly
There is a chance here as well that libraries were using mini racer in a fork unsafe way and somehow got away with it in earlier versions due to luck.
The same happened to me after upgrading to the latest libv8-node (21.7.2.0) and mini_racer (0.12.0) during running specs
ruby/3.3.0/lib/ruby/gems/3.3.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:228: [BUG] Segmentation fault at 0x0000734c73d8700
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
After some lengthy monkey patching session, I'm 99% sure it's due to this file: https://github.com/ai/autoprefixer-rails/blob/master/vendor/autoprefixer.js I removed the source map at the last line, it does not change anything.
The file is 6.29MB which isn't the simplest to debug.
Oh wow, does an eval of this file cause the crash?
On Wed, 1 May 2024 at 6:28 PM, Thomas @.***> wrote:
After some lengthy monkey patching session, I'm 99% sure it's due to this file: https://github.com/ai/autoprefixer-rails/blob/master/vendor/autoprefixer.js I removed the source map at the last line, it does not change anything.
The file is 6.29MB which isn't the simplest to debug.
— Reply to this email directly, view it on GitHub https://github.com/rubyjs/mini_racer/issues/300#issuecomment-2088148748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAABIXPXZDI2KX724Z3GTFTZACRTFAVCNFSM6AAAAABGUEE3QSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBYGE2DQNZUHA . You are receiving this because you were mentioned.Message ID: @.***>
I copy/paste that giant file into a browser console and got an error:
Uncaught ReferenceError: SharedArrayBuffer is not defined
at <anonymous>:9900:39
at createCommonjsModule (<anonymous>:258:11)
at <anonymous>:9553:15
at <anonymous>:77306:3
Which is used at line 9900 (other references are inside a strings)
From Chrome 92 (and Firefox too), SharedArrayBuffer
is disabled by default for security reason due to the meltdown and specter attack (https://meltdownattack.com/)
https://web.dev/articles/cross-origin-isolation-guide
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer
Thanks for diving this deep into it @D-system, IIUC this still not reproduced locally, only in CircleCI?
I copy/paste that giant file into a browser console
I think what Sam meant is mimicking https://github.com/ai/autoprefixer-rails/blob/cf0af2807108e43f4858574b5d22dfaec0630d80/lib/autoprefixer-rails/processor.rb#L133C9-L133C23
i.e does the following cause a crash?
js = File.read('path/to/autoprefixer.js')
# direct mini_racer
context = MiniRacer::Context.new
context.eval(js)
# via execjs, goes through:
# - https://github.com/rails/execjs/blob/ea4731de8b7b68de8404029c2f3d52eacefabec7/lib/execjs/runtime.rb#L68
# - https://github.com/rails/execjs/blob/ea4731de8b7b68de8404029c2f3d52eacefabec7/lib/execjs/mini_racer_runtime.rb#L11
ExecJS.compile(js)
If that's not enough, maybe it can be reproduced by mimicking https://github.com/ai/autoprefixer-rails/blob/cf0af2807108e43f4858574b5d22dfaec0630d80/lib/autoprefixer-rails/processor.rb#L35 with a specific input file?
css = File.read('path/to/style.css')
process_opts = { from: ..., to: ..., map: ... }
plugin_opts = { ... }
# direct mini_racer
context.eval("autoprefixer.process.apply(this, #{JSON.generate([css, process_opts, plugin_opts])}")
# via execs, goes through:
# - https://github.com/rails/execjs/blob/ea4731de8b7b68de8404029c2f3d52eacefabec7/lib/execjs/mini_racer_runtime.rb#L35
ExecJS.call('autoprefixer.process', css, process_opts, plugin_opts)
Notes:
eval "#{identifier}.apply(this, #{::JSON.generate(args)})"
is vulnerable to code injection! (depends on how people use it)apply
+ JSON.generate
is complete bonkers, imagine the processing + memory cost with sizeable CSSI don't know if it's the same bug, but recently I added a new plugin to the Gemfile of a template repository, which caused the Docker image to be rebuilt with the latest versions of the libraries. The build failed in GitHub actions, with the message:
compiling mini_racer_extension.cc
linking shared-object mini_racer_extension.so
/usr/bin/ld: cannot find
/var/lib/gems/3.2.0/gems/libv8-node-21.7.2.0-x86_64-linux/vendor/v8/x86_64-linux-gnu/libv8/obj/libv8_monolith.a:
No such file or directory
collect2: error: ld returned 1 exit status
make: *** [Makefile:265: mini_racer_extension.so] Error 1
@george-gca entirely unrelated. see https://github.com/rubyjs/libv8-node/pull/36#issuecomment-2045274661
Thank you for dealing. FYI, the following combination worked for me. It caused no segmentation fault.
RAILS_ENV=production RBENV_ROOT=$HOME/.rbenv RBENV_VERSION=3.3.0 $HOME/.rbenv/bin/rbenv exec bundle exec rake assets:precompile
I don't have time to investigate further at the moment. I'll be AFK from tomorrow until after the RubyKaigi (2024-05-15 to 17).
I'm back.
After adding autoprefixer-rails
to the Gemfile, I tried js = File.read('/home/circleci/.rubygems/gems/autoprefixer-rails-10.4.16.0/vendor/autoprefixer.js'); MiniRacer::Context.new.eval(js)
, the command doesn't error out.
(the ./bin/rails assets:clobber assets:precompile
does not work)
Then I copied the autoprefixer.js
into the app/javascript/packs/application.js
, I removed the gem autoprefixer-rails
and run ./bin/rails assets:clobber assets:precompile
and it just works fine.
The error certainly happens after the file is loaded and when the CSS is being process. Another time, I'll look at how https://github.com/ai/autoprefixer-rails works to debug.
Interesting, @D-system. Would be great to have a working reproduction. Maybe all that's left is some CSS, if I'm reading your findings correctly. Thank you for your efforts!
I'm using a brand new app. The CSS is non existent. Just the application.css
below that only have require_tree
and require_self
which doesn't do much, there's no extra file. It is frustrating.
/*
* This is a manifest file that'll be compiled into application.css, which will include all the files
* listed below.
*
* Any CSS and SCSS file within this directory, lib/assets/stylesheets, or any plugin's
* vendor/assets/stylesheets directory can be referenced here using a relative path.
*
* You're free to add application-wide styles to this file and they'll appear at the bottom of the
* compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
* files in this directory. Styles in this file should be added after the last require_* statement.
* It is generally better to create a new file per style scope.
*
*= require_tree .
*= require_self
*/
I also would love to have the bug reproduced on my local. I'm still SSH-ing to CircleCI.
Here's the command that I use to reproduce the bug on CircleCI:
gem install rails -v 6.1.7.7
rails new --skip-webpack-install app
cd app
echo "gem 'mini_racer', '~> 0.12.0'" >> Gemfile
echo 'gem "autoprefixer-rails", "10.4.16.0"' >> Gemfile
bundle install
./bin/rails assets:clobber assets:precompile
This does reproduce it in docker:
docker run -it --rm ruby:3.0
gem install rails -v 6.1.7.7
rails new --skip-javascript app
cd app
echo "gem 'mini_racer', '~> 0.12.0'" >> Gemfile
echo 'gem "autoprefixer-rails", "10.4.16.0"' >> Gemfile
bundle install
./bin/rails assets:clobber assets:precompile
I looked at the files and there is nothing special about them, the backtrace for the segfault is on cleanup.
specifically miniracer calling deallocate from the finalize_list in ruby and free_context_raw
leading to a segfault.
@lloeki is this enough for you to debug, or do you need more help here? One possibility here is that somehow autoprefixer is forking processes, I looked at the code and it does not look like it.
The crash looks like is is from
if (isolate_info) { isolate_info->release(); }
@cataphract any ideas about :arrow_double_up: ? We tried to fix a premature cleanup previously.
jFYI: The example does not crash under aarch64-linux
or x86_64-linux
(under Rosetta emulation) on Apple Silicon, M1 Pro.
In our app we encountered segfaults with mini_racer and turf.js when doing the bump from 0.9.0 to 0.12.0. It didn't occur in our CI but it did in our prod instance.
CI is running on buildjet ubuntu 22.04 with ruby 3.2.2 and prod is running on heroku on ubuntu 20.04 with ruby 3.2.2 and YJIT and jemalloc enabled
and jemalloc enabled
This is most probably jemalloc-related, which is known to trigger core dumps unrelated to this very one.
is this enough for you to debug
@SamSaffron if I manage to reliably reproduce it, possibly so.
Sadly, no repro (neither on M1 hardware nor on Intel hardware) :/
If this is any help, @SamSaffron's repro did work for me and I traced the segfault a bit further down to this line:
I forced libv8-node down to 20.12.1.0 and the issue went away, so perhaps it's possible that it's either a bug in 21.7.2.0 or a breaking change requiring an update?
Edit: for those who don't know (like me before yesterday!) that isolate
in the line I think is a v8::Isolate, i.e. part of V8 proper and not mini_racer
.
If this is any help, @SamSaffron's repro did work for me and I traced the segfault a bit further down to this line:
I forced libv8-node down to 20.12.1.0 and the issue went away, so perhaps it's possible that it's either a bug in 21.7.2.0 or a breaking change requiring an update?
Edit: for those who don't know (like me before yesterday!) that
isolate
in the line I think is a v8::Isolate, i.e. part of V8 proper and notmini_racer
.
Can confirm a downgrade of libv8-node fixed the issue.
I'm having the exact same issues and have three crash reports I can share if that will help. Ruby 3.3.2 on heroku with Rails 6.1.7.7. Segmentation fault is always here:
2024-06-03T03:53:04.924655+00:00 app[web.1]: /app/vendor/bundle/ruby/3.3.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:228: [BUG] Segmentation fault at 0x00007ff1507a0008
2024-06-03T03:53:04.924685+00:00 app[web.1]: ruby 3.3.2 (2024-05-30 revision e5a195edf6) [x86_64-linux]
2024-06-03T03:53:04.924686+00:00 app[web.1]:
2024-06-03T03:53:04.924687+00:00 app[web.1]: -- Control frame information -----------------------------------------------
2024-06-03T03:53:04.924687+00:00 app[web.1]: c:0133 p:---- s:0879 e:000878 CFUNC :eval_unsafe
2024-06-03T03:53:04.924689+00:00 app[web.1]: c:0132 p:0009 s:0873 e:000872 BLOCK /app/vendor/bundle/ruby/3.3.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:228
2024-06-03T03:53:04.924689+00:00 app[web.1]: c:0131 p:0010 s:0870 e:000869 METHOD /app/vendor/bundle/ruby/3.3.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:348
2024-06-03T03:53:04.924690+00:00 app[web.1]: c:0130 p:0008 s:0859 e:000858 BLOCK /app/vendor/bundle/ruby/3.3.0/gems/mini_racer-0.12.0/lib/mini_racer.rb:227 [FINISH]
2024-06-03T03:53:04.924690+00:00 app[web.1]: c:0129 p:---- s:0856 e:000855 CFUNC :synchronize
2024-
I can provide more if needed.
Hello the team
Thank you for the update to libv8-node 21.7.2.0 yesterday. I wanted to report an error. Setup:
RAILS_ENV=production SECRET_KEY_BASE=1 bin/rails assets:clobber assets:precompile
It does work on Darwin Kernel Version 23.2.0 arm64 (Mac on arm) that compile from source.
The error is from eval -> eval_unsafe, so I suspect it is an issue with the
libv8-node
https://github.com/rubyjs/mini_racer/blob/5b2a25cdb4c84c7500ed47639ef8c664f5bac26d/lib/mini_racer.rb#L228