segfault during rails assets:precompile:nondigest

[coldnebo]

Hi, we're seeing this segfault intermittently in CentOS during our deploy process.

There seems to be no difference between the logs except the segfault when it occurs.

Admittedly this is an old version of therubyracer (0.9.9), but one of our apps doesn't seem to have this issue when precompiling and we can't tell what's going on.

Any ideas what might cause this?

I know there are other segfault issues open, but wasn't sure what their context was, so I opened this separately.

** Execute assets:precompile:nondigest
/usr/local/bin/rake: [BUG] Segmentation fault at 0x00000000000020
ruby 2.1.7p400 (2015-08-18 revision 51632) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0001 p:0000 s:0002 E:000158 TOP    [FINISH]

-- C level backtrace information -------------------------------------------
/usr/local/bin/ruby [0x2ab452920c83]
/usr/local/bin/ruby [0x2ab4529786d4]
/usr/local/bin/ruby(rb_bug+0xb8) [0x2ab4529789b8]
/usr/local/bin/ruby [0x2ab45289abe0]
/lib64/libpthread.so.0 [0x2ab452e80ca0]
/usr/local/bin/ruby(rb_gc_writebarrier+0) [0x2ab4527eab00]
/usr/local/bin/ruby [0x2ab452938dd7]
/usr/local/bin/ruby [0x2ab45294514c]
/usr/local/bin/ruby(rb_ary_unshift+0x18) [0x2ab452945378]
/usr/local/lib/ruby/gems/2.1.0/gems/therubyracer-0.9.9/ext/v8/v8.so [0x2ab4625e1156]
/usr/local/bin/ruby(rb_gc_call_finalizer_at_exit+0x220) [0x2ab4527f35e0]
/usr/local/bin/ruby(ruby_cleanup+0x3bb) [0x2ab4527d880b]
/usr/local/bin/ruby [0x2ab4527d3aaf]
/lib64/libc.so.6(__libc_start_main+0xf4) [0x2ab4539759f4]
/usr/local/bin/ruby [0x2ab4527d3989]

[coldnebo]

Just had a thought... if there is binding between therubyracer and ruby objects (i.e. to allow calls to ruby objects from javascript) could this be a mismatch thing between Rubies and the version of therubyracer? 0.9.9 was probably originally before the Ruby 1.9.2 release way back when we started using asset pipeline.

Another possible hazard is leaving the gem compiled for one ruby and then installing another ruby but using the same precompiled gem, no? In that case the wraps might be binary incompatible... but I'd expect that kind of error to fail all the time, not part of the time... hmmm. confusing.

[ignisf]

@coldnebo Ruby has ABI versioning, generally you should consider recompiling your binary gems whenever you change Ruby ABI. Though this may not be the reason behind your segfault.

[coldnebo]

@ignisf good point.

[coldnebo]

Our gems and Ruby versions are ok, and in sync (i.e. reinstalled/compiled), but that didn't solve the issue.

I can now reproduce this error consistently using the following bash snippet which loosely models what is happening on our deployment server (apps may be pushed by devs at any time):

pushd app1
rake assets:precompile &
popd
pushd app2
rake assets:precompile &
popd

It looks like therubyracer v8.so has issues with multiple processes compiling assets at the same time from different processes.

[coldnebo]

Another important piece of information seems to be that this only happens happens more frequently when one or more asset files are empty. If you remove these empty compilation targets from the pipeline, then this error goes away.

I'm not sure what the asset pipeline is doing in this case.