ribose-jeffreylau
commented
4 years ago Update
Version 1.2.8.6 also segfaults.
Closed ribose-jeffreylau closed 4 years ago
ribose-jeffreylau
commented
4 years ago Version 1.2.8.6 also segfaults.
ronaldtse
commented
4 years ago @CAMOBAP795 can you please help fix this ASAP? Thanks.
CAMOBAP
commented
4 years ago @ronaldtse I will check
CAMOBAP
commented
4 years ago Low-level crash in native cpp library libsaas
I can reproduce it with docker
Crash happens in Sass::Expression::Expression(Sass::ParserState, bool, bool, bool, Sass::Expression::Type) function call
To be continued...
ronaldtse
commented
4 years ago Ping @opoudjis are we doing something new in SASS from v1.2.8.4 and v1.2.8.5? Thanks.
opoudjis
commented
4 years ago Yes, the Asciidoctor boilerplate is now being processed in isodoc, centrally. But the crash is happening in 1/usr/local/bundle/gems/isodoc-1.0.14/lib/isodoc/convert.rb:2:in require'---it happens as soon as sassc is required in standoc, and never gets to the actual code doing the template processing. So I don't think it's something I'm doing.
opoudjis
commented
4 years ago And I'm using MacOS 10.14.6 as well.
Recall that we had serious trouble installing docker in recent builds; might the docker build have gotten truncated?
ronaldtse
commented
4 years ago I think the SassC build in docker may be broken, or it is too old. Will have to investigate.
CAMOBAP
commented
4 years ago I think the SassC build in docker may be broken, or it is too old.
@ronaldtse sassc gem now contain prebuild libsass, so we don't use the system version anymore
Will have to investigate.
I will compare two versions of libsass (binaries just in case)
ronaldtse
commented
4 years ago sassc gem now contain prebuild libsass, so we don't use the system version anymore
Then it is possible that the underlying libsass installed via system is different from the one provided by sassc gem? Or the prebuilt ones are incompatible with the base container?
ronaldtse
commented
4 years ago I'd prefer us to use minimal Ubuntu as the base container instead of the Ruby Debian containers...
ronaldtse
commented
4 years ago So I have implemented the metanorma and mn containers on Ubuntu, called metanorma-ubuntu and mn-ubuntu. I was hoping that this may solve the problem.
However, it didn't.
The same segfault occurs in:
It doesn't occur in:
Strangely, the segfaults only happen in the containers built on GitHub Actions. When they are locally built, they run properly.
They don't occur on their sister containers, metanorma/mn and metanorma/mn-ubuntu (when built on GHA). The only difference is that the mn containers use git master of metanorma-cli.
In metanorma-docker, we have a routine to squash the container layers. In my experimentation, if I build the containers locally, it works regardless if they were squashed. I have not tried not-squashing them on GHA.
The line that dies is this: https://github.com/sass/sassc-ruby/blob/8378681f3c212bc580ee41e01bf55c6679577292/lib/sassc/native.rb#L11
(Line 11)
I suspect that the libsass.so is broken -- the GHA platform somehow seems different from my local one.
If you look at the top of the trace:
-- Control frame information -----------------------------------------------
c:0040 p:---- s:0205 e:000204 CFUNC :open
c:0039 p:0022 s:0199 e:000198 BLOCK /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:112 [FINISH]
c:0038 p:---- s:0190 e:000189 CFUNC :each
c:0037 p:0113 s:0186 e:000185 BLOCK /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:109 [FINISH]
c:0036 p:---- s:0179 e:000178 CFUNC :map
c:0035 p:0069 s:0175 e:000174 METHOD /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:99
c:0034 p:0079 s:0168 e:000167 CLASS /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:11Notice that it failed on /usr/local/bundle/gems/ffi-1.12.2/lib/ffi/library.rb:112.
The content of that line is:
lib = FFI::DynamicLibrary.open(libname, lib_flags)Putting into context:
https://github.com/ffi/ffi/blob/ad524cb016fc788861d76e43a5067b8d441a76bc/lib/ffi/library.rb#L95-L113
I thought maybe libsass.so is missing, but lo and behold, it exists in the container that crashes:
In fact, I can reproduce this segfault by just doing:
docker run -it metanorma/metanorma-ubuntu:latest bash
$ # In container
$ irb
$ require 'sassc'
[segfault trace...]So maybe it's not related to Metanorma code. It's related to the libsass/sassc installation...
Here's the full trace:
ronaldtse
commented
4 years ago Or as someone at https://github.com/rails/sprockets/issues/633 suggests, downgrading to sassc 2.1.0 might work.
I think it's very strange that git metanorma-cli work but released metanorma-cli doesn't.
ronaldtse
commented
4 years ago This is very strange. metanorma/metanorma still segfaults, but metanorma/mn doesn't. Both are based on ruby:2.6-slim-buster
ronaldtse
commented
4 years ago @ribose-jeffreylau can you confirm that the new docker containers work? Can you try docker pull metanorma/metanorma:latest? Thanks.
ribose-jeffreylau
commented
4 years ago Here's the latest trace:
ribose-jeffreylau
commented
4 years ago Here's another trace featuring sassc:
ronaldtse
commented
4 years ago @ribose-jeffreylau are you seeing this locally or on GitHub or GitLab? If locally, can you let us know the docker version? Thanks.
ribose-jeffreylau
commented
4 years ago @ronaldtse This is seen on local.
$ docker version
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.39 (downgraded from 1.40)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:22:34 2019
OS/Arch: darwin/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
Go version: go1.10.4
Git commit: 4d60db4
Built: Wed Nov 7 00:52:55 2018
OS/Arch: linux/amd64
Experimental: falseThis is also seen on GitLab: https://gitlab.com/calconnect/tc-localization/csd-transcription-systems/-/jobs/439176074
ronaldtse
commented
4 years ago @ribose-jeffreylau but I'm also seeing it runs successfully on GitLab? https://gitlab.com/calconnect/tc-localization/csd-transcription-systems/-/jobs/439177620
ronaldtse
commented
4 years ago @ribose-jeffreylau I see the issue now, you're reverting the changes.
ronaldtse
commented
4 years ago @CAMOBAP795 can you please help investigate and fix this issue? When I build these containers locally, I don't see this issue. Thanks!
CAMOBAP
commented
4 years ago libsaas.so looks different for metanorma/metanorma:latest & metanorma/mn:latest while saasc stay on the same version 2.2.1, I have no uderstanding how this may happens
But looks like the problem not only in libsass.so because I tried to replace it with the one from metanorma/mn and it doesn't help
To be continued...
ronaldtse
commented
4 years ago In the Gemfile diff, the only two gems that changed that are not ours (our stuff is all pure Ruby) are:
- concurrent-ruby (1.1.6)
+ concurrent-ruby (1.1.5)
...
- nokogiri (1.10.8)
+ nokogiri (1.10.7)Maybe it's due to one of these two gems? Otherwise, it may be due to the compilation of libsass? If so, we can try using the system version (Ubuntu: apt-get install -y libsass)?
CAMOBAP
commented
4 years ago it may be due to the compilation of libsass?
Highly likely, I will enable debug logging on GHA later to check how exactly we compile libsass
CAMOBAP
commented
4 years ago Looks like a known issue https://github.com/sass/sassc-ruby/issues/146
I will try to implement this workaround:
Also, some users confirmed that downgrade to 2.1.0 should works
To be continued...
ronaldtse
commented
4 years ago Great find @CAMOBAP795 , let’s try the bundle config prebuilt binary:
bundle config --local build.sassc --disable-march-tune-native
If this doesn’t work, let’s downgrade to 2.1.
CAMOBAP
commented
4 years ago Retagged.
Please remove local metanorma/metanorma:latest image then run
docker run -v $(pwd):/metanorma/ metanorma/metanorma:latest metanorma sources/iso-rice-en.adoc
I can confirm that it works for me
ribose-jeffreylau
commented
4 years ago I can confirm 1.2.9.1 works. Thanks! Closing.
ronaldtse
commented
4 years ago Thanks @CAMOBAP795 @ribose-jeffreylau !
Description
Under macOS 10.14.6, using
metanorma/metanorma:1.2.8.4would run successfully, without segfaulting, whereas usingmetanorma/metanorma:1.2.8.5would cause itself to segfault.Steps to reproduce
git clone https://github.com/metanorma/mn-samples-isocd mn-samples-isodocker run -v $(pwd):/metanorma/ metanorma/metanorma:1.2.8.5 metanorma sources/iso-rice-en.adocExpected result
Document builds. No segfaulting.
Actual result
click for trace
``` /usr/local/bundle/gems/ffi-1.12.1/lib/ffi/library.rb:112: [BUG] Illegal instruction at 0x00007f2c10d2fcc5 ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux] -- Control frame information ----------------------------------------------- c:0042 p:---- s:0213 e:000212 CFUNC :open c:0041 p:0022 s:0207 e:000206 BLOCK /usr/local/bundle/gems/ffi-1.12.1/lib/ffi/library.rb:112 [FINISH] c:0040 p:---- s:0198 e:000197 CFUNC :each c:0039 p:0113 s:0194 e:000193 BLOCK /usr/local/bundle/gems/ffi-1.12.1/lib/ffi/library.rb:109 [FINISH] c:0038 p:---- s:0187 e:000186 CFUNC :map c:0037 p:0069 s:0183 e:000182 METHOD /usr/local/bundle/gems/ffi-1.12.1/lib/ffi/library.rb:99 c:0036 p:0079 s:0176 e:000175 CLASS /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:11 c:0035 p:0007 s:0172 e:000171 CLASS /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:6 c:0034 p:0014 s:0169 e:000168 TOP /usr/local/bundle/gems/sassc-2.2.1/lib/sassc/native.rb:5 [FINISH] c:0033 p:---- s:0166 e:000165 CFUNC :require_relative c:0032 p:0021 s:0161 e:000160 TOP /usr/local/bundle/gems/sassc-2.2.1/lib/sassc.rb:31 [FINISH] c:0031 p:---- s:0158 e:000157 CFUNC :require c:0030 p:0013 s:0153 e:000152 TOP /usr/local/bundle/gems/isodoc-1.0.14/lib/isodoc/convert.rb:2 [FINISH] c:0029 p:---- s:0150 e:000149 CFUNC :require c:0028 p:0104 s:0145 e:000144 TOP /usr/local/bundle/gems/isodoc-1.0.14/lib/isodoc.rb:17 [FINISH] c:0027 p:---- s:0142 e:000141 CFUNC :require c:0026 p:0055 s:0137 e:000136 TOP /usr/local/bundle/gems/metanorma-standoc-1.3.14/lib/asciidoctor/standoc/base.rb:8 [FINISH] c:0025 p:---- s:0134 e:000133 CFUNC :require c:0024 p:0020 s:0129 e:000128 TOP /usr/local/bundle/gems/metanorma-standoc-1.3.14/lib/asciidoctor/standoc/converter.rb:3 [FINISH] c:0023 p:---- s:0126 e:000125 CFUNC :require_relative c:0022 p:0036 s:0121 e:000120 TOP /usr/local/bundle/gems/metanorma-standoc-1.3.14/lib/metanorma-standoc.rb:3 [FINISH] c:0021 p:---- s:0118 e:000117 CFUNC :require c:0020 p:0013 s:0113 e:000112 TOP /usr/local/bundle/gems/metanorma-iso-1.3.13/lib/asciidoctor/iso/converter.rb:2 [FINISH] c:0019 p:---- s:0110 e:000109 CFUNC :require_relative c:0018 p:0036 s:0105 e:000104 TOP /usr/local/bundle/gems/metanorma-iso-1.3.13/lib/metanorma-iso.rb:2 [FINISH] c:0017 p:---- s:0102 e:000101 CFUNC :require c:0016 p:0136 s:0097 e:000096 METHOD /usr/local/bundle/gems/metanorma-0.3.18/lib/metanorma/compile.rb:58 c:0015 p:0006 s:0091 e:000090 METHOD /usr/local/bundle/gems/metanorma-0.3.18/lib/metanorma/compile.rb:45 c:0014 p:0020 s:0086 e:000085 METHOD /usr/local/bundle/gems/metanorma-0.3.18/lib/metanorma/compile.rb:13 c:0013 p:0021 s:0077 e:000076 METHOD /usr/local/bundle/gems/metanorma-cli-1.2.8.5/lib/metanorma/cli/compiler.rb:41 c:0012 p:0004 s:0073 e:000072 METHOD /usr/local/bundle/gems/metanorma-cli-1.2.8.5/lib/metanorma/cli/compiler.rb:29 c:0011 p:0011 s:0069 e:000068 METHOD /usr/local/bundle/gems/metanorma-cli-1.2.8.5/lib/metanorma/cli/compiler.rb:33 c:0010 p:0044 s:0063 e:000062 METHOD /usr/local/bundle/gems/metanorma-cli-1.2.8.5/lib/metanorma/cli/command.rb:34 c:0009 p:0064 s:0058 e:000057 METHOD /usr/local/bundle/gems/thor-0.20.3/lib/thor/command.rb:27 c:0008 p:0047 s:0050 e:000049 METHOD /usr/local/bundle/gems/thor-0.20.3/lib/thor/invocation.rb:126 c:0007 p:0259 s:0043 e:000042 METHOD /usr/local/bundle/gems/thor-0.20.3/lib/thor.rb:387 c:0006 p:0062 s:0030 e:000029 METHOD /usr/local/bundle/gems/thor-0.20.3/lib/thor/base.rb:466 c:0005 p:0035 s:0023 e:000022 METHOD /usr/local/bundle/gems/metanorma-cli-1.2.8.5/lib/metanorma/cli.rb:60 c:0004 p:0090 s:0017 e:000016 TOP /usr/local/bundle/gems/metanorma-cli-1.2.8.5/exe/metanorma:20 [FINISH] c:0003 p:---- s:0013 e:000012 CFUNC :load c:0002 p:0109 s:0008 E:000370 EVAL /usr/local/bundle/bin/metanorma:23 [FINISH] c:0001 p:0000 s:0003 E:001a90 (none) [FINISH] -- Ruby level backtrace information ---------------------------------------- /usr/local/bundle/bin/metanorma:23:in `related issue: https://gitlab.com/calconnect/tc-localization/csd-transcription-systems/issues/46