Closed carpnick closed 9 years ago
Can provide specs and recipes if necessary.
backtrace has ffi-yajl 0.2.0, the current version is 1.1.0 if upgrading doesn't fix it, then there's two known issues which will dump core:
https://github.com/opscode/ffi-yajl/issues/15
and
https://github.com/opscode/ffi-yajl/issues/16
both of those cases are probably bugs in the code that is feeding input to the parser, the coredump is just a particularly ugly error message.
Tried upping to 1.1.0 - same error.
@carpnick When you upgraded, did the Control frame information
section show that you were using the upgraded ffi-yajl? Could be bundler or something locking to the old version. We should be releasing a new ChefDK soon with everything upgraded, so it'd be helpful if you can try that when it comes out.
Yes it showed the new version. I actually uninstalled the previous version .2.0 to make sure it wasn't being used. I will give the new chefdk a try when it comes out.
Using 0.3.0 I am getting a really weird behavior.
When I set the RSPEC config to debug level logging I see this basic output(Condensed to part where it fails)
[2014-10-06T12:07:18-04:00] DEBUG: Loading Recipe lib_win_powershell1::powershell3 via include_recipe
[2014-10-06T12:07:18-04:00] DEBUG: Found recipe powershell3 in cookbook lib_win_powershell1
[2014-10-06T12:07:18-04:00] DEBUG: Loading Recipe lib_win_powershell1::reboot via include_recipe
[2014-10-06T12:07:18-04:00] DEBUG: Found recipe reboot in cookbook lib_win_powershell1
[2014-10-06T12:07:18-04:00] DEBUG: Loading Recipe windows::reboot_handler via include_recipe
[2014-10-06T12:07:18-04:00] DEBUG: Found recipe reboot_handler in cookbook windows
[2014-10-06T12:07:19-04:00] DEBUG: Loading Recipe lib_win_dotnet::dotnet4 via include_recipe
[2014-10-06T12:07:19-04:00] DEBUG: Found recipe dotnet4 in cookbook lib_win_dotnet
[2014-10-06T12:07:19-04:00] DEBUG: Loading Recipe lib_win_dotnet::reboot via include_recipe
[2014-10-06T12:07:19-04:00] DEBUG: Found recipe reboot in cookbook lib_win_dotnet
[2014-10-06T12:07:19-04:00] DEBUG: Loading Recipe windows::reboot_handler via include_recipe
[2014-10-06T12:07:19-04:00] DEBUG: I am not loading windows::reboot_handler, because I have already seen it.
[2014-10-06T12:07:19-04:00] DEBUG: Loading from cookbook_path: C:/Users/NCARPE~1/AppData/Local/Temp/d20141006-12108-j66dqo/cookbooks
C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chef-11.16.2-x86-mingw32/lib/chef/client.rb:464: [BUG] Segmentation fault
ruby 2.0.0p451 (2014-02-24) [i386-mingw32]
-- Control frame information -----------------------------------------------
c:0037 p:---- s:0138 e:000137 IFUNC
c:0036 p:---- s:0136 e:000135 CFUNC :each
c:0035 p:---- s:0134 e:000133 CFUNC :to_a
c:0034 p:---- s:0132 e:000131 CFUNC :entries
c:0033 p:0034 s:0128 e:000127 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chef-11.16.2-x86-mingw32/lib/chef/client.rb:464
c:0032 p:0009 s:0124 e:000123 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chef-11.16.2-x86-mingw32/lib/chef/client.rb:478 [FINISH]
c:0031 p:---- s:0121 e:000120 IFUNC
c:0030 p:---- s:0119 e:000118 CFUNC :each
c:0029 p:---- s:0117 e:000116 CFUNC :all?
c:0028 p:0072 s:0114 e:000113 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chef-11.16.2-x86-mingw32/lib/chef/client.rb:478
c:0027 p:0021 s:0108 e:000107 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chef-11.16.2-x86-mingw32/lib/chef/client.rb:266
c:0026 p:0078 s:0104 e:000103 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chefspec-4.0.1/lib/chefspec/runner.rb:124
c:0025 p:0031 s:0100 e:000099 LAMBDA C:/code/Internal/cookbooks/eis/lib_win_powershell1/spec/unit/powershell3_spec.rb:20 [FINISH]
c:0024 p:0015 s:0098 e:000094 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/memoized_helpers.rb:242 [FINI
SH]
c:0023 p:---- s:0092 e:000091 CFUNC :fetch
c:0022 p:0011 s:0088 e:000087 LAMBDA C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/memoized_helpers.rb:242 [FINI
SH]
c:0021 p:0008 s:0086 e:000084 BLOCK C:/code/Internal/cookbooks/eis/lib_win_powershell1/spec/unit/powershell3_spec.rb:39 [FINISH]
c:0020 p:---- s:0083 e:000082 CFUNC :instance_exec
c:0019 p:0021 s:0079 e:000078 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example.rb:151
c:0018 p:0015 s:0076 e:000075 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example.rb:328
c:0017 p:0076 s:0071 E:000980 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example.rb:148
c:0016 p:0050 s:0065 e:000064 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example_group.rb:500 [FINISH]
c:0015 p:---- s:0060 e:000059 CFUNC :map
c:0014 p:0014 s:0057 e:000056 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example_group.rb:496
c:0013 p:0072 s:0053 e:000052 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example_group.rb:463
c:0012 p:0010 s:0046 e:000045 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:111 [FINISH]
c:0011 p:---- s:0043 e:000042 CFUNC :map
c:0010 p:0039 s:0040 e:000039 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:111
c:0009 p:0017 s:0036 e:000035 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/reporter.rb:53
c:0008 p:0018 s:0032 e:000031 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:107
c:0007 p:0022 s:0028 e:000027 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:85
c:0006 p:0088 s:0023 e:000022 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:69
c:0005 p:0024 s:0016 e:000015 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:37
c:0004 p:0023 s:0012 e:000011 TOP C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/exe/rspec:4 [FINISH]
c:0003 p:---- s:0010 e:000009 CFUNC :load
c:0002 p:0134 s:0006 E:000a24 EVAL C:/opscode/chefdk/embedded/bin/rspec:23 [FINISH]
c:0001 p:0000 s:0002 E:00180c TOP [FINISH]
The line 464 in Client.rb: !File.exists?(path) || (Dir.entries(path).size <= 2)
Without debug logging....
FC:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/json-1.8.1/lib/json/common.rb:155: [BUG] Segmentation fault
ruby 2.0.0p451 (2014-02-24) [i386-mingw32]
-- Control frame information -----------------------------------------------
c:0037 p:---- s:0151 e:000150 CFUNC :initialize
c:0036 p:---- s:0149 e:000148 CFUNC :new
c:0035 p:0021 s:0144 e:000143 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/json-1.8.1/lib/json/common.rb:155
c:0034 p:0139 s:0139 e:000138 LAMBDA C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/fauxhai-2.2.0/lib/fauxhai/mocker.rb:56 [FINISH]
c:0033 p:---- s:0133 e:000132 CFUNC :call
c:0032 p:0023 s:0130 E:001f24 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/fauxhai-2.2.0/lib/fauxhai/mocker.rb:77
c:0031 p:0018 s:0127 e:000126 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/fauxhai-2.2.0/lib/fauxhai/mocker.rb:26 [FINISH]
c:0030 p:---- s:0121 e:000120 CFUNC :new
c:0029 p:0019 s:0117 e:000116 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/fauxhai-2.2.0/lib/fauxhai.rb:12
c:0028 p:0059 s:0112 e:000109 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chefspec-4.0.1/lib/chefspec/runner.rb:303
c:0027 p:0022 s:0107 e:000106 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chefspec-4.0.1/lib/chefspec/runner.rb:142
c:0026 p:0007 s:0104 e:000103 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/chefspec-4.0.1/lib/chefspec/runner.rb:109
c:0025 p:0031 s:0100 e:000099 LAMBDA C:/code/Internal/cookbooks/eis/lib_win_powershell1/spec/unit/powershell3_spec.rb:20 [FINISH]
c:0024 p:0015 s:0098 e:000094 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/memoized_helpers.rb:242 [FINI
SH]
c:0023 p:---- s:0092 e:000091 CFUNC :fetch
c:0022 p:0011 s:0088 e:000087 LAMBDA C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/memoized_helpers.rb:242 [FINI
SH]
c:0021 p:0008 s:0086 e:000084 BLOCK C:/code/Internal/cookbooks/eis/lib_win_powershell1/spec/unit/powershell3_spec.rb:48 [FINISH]
c:0020 p:---- s:0083 e:000082 CFUNC :instance_exec
c:0019 p:0021 s:0079 e:000078 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example.rb:151
c:0018 p:0015 s:0076 e:000075 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example.rb:328
c:0017 p:0076 s:0071 E:001cc0 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example.rb:148
c:0016 p:0050 s:0065 e:000064 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example_group.rb:500 [FINISH]
c:0015 p:---- s:0060 e:000059 CFUNC :map
c:0014 p:0014 s:0057 e:000056 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example_group.rb:496
c:0013 p:0072 s:0053 e:000052 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/example_group.rb:463
c:0012 p:0010 s:0046 e:000045 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:111 [FINISH]
c:0011 p:---- s:0043 e:000042 CFUNC :map
c:0010 p:0039 s:0040 e:000039 BLOCK C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:111
c:0009 p:0017 s:0036 e:000035 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/reporter.rb:53
c:0008 p:0018 s:0032 e:000031 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:107
c:0007 p:0022 s:0028 e:000027 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:85
c:0006 p:0088 s:0023 e:000022 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:69
c:0005 p:0024 s:0016 e:000015 METHOD C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/lib/rspec/core/runner.rb:37
c:0004 p:0023 s:0012 e:000011 TOP C:/opscode/chefdk/embedded/lib/ruby/gems/2.0.0/gems/rspec-core-3.1.5/exe/rspec:4 [FINISH]
c:0003 p:---- s:0010 e:000009 CFUNC :load
c:0002 p:0134 s:0006 E:002334 EVAL C:/opscode/chefdk/embedded/bin/rspec:23 [FINISH]
c:0001 p:0000 s:0002 E:00251c TOP [FINISH]
Line 155 from common.rb: Parser.new(source, opts).parse
@lamont-granquist can you check this out related to ffi-yajl ?
There's a couple of known segfaulting issues with ffi-yajl that it might be, i can't tell from the backtraces what is causing it though, i need the inputs.
Did a bit more debugging. Not sure if this helps. I went into : fauxhai-2.2.0/lib/fauxhai/mocker.rb:56 and I noticed it was calling : JSON.parse( File.read(filepath) )
So I started separating the File and JSON calls and I noticed it randomly failing sometimes on the read and sometimes within JSON.parse. So I went a bit further and made a local modification in the same spot like this:
puts 'Running GC.start' GC.start sleep 5 puts 'After sleep' JSON.parse( File.read(filepath) )
Now it always segment faults out of GC.start. So then I tried testing the theory with a quick script I wrote (I started writing all the files json to my own temp files named 1 to 4): require 'json' JSON.parse(File.read("C:\Temp\1")) JSON.parse(File.read("C:\Temp\2")) hme2=JSON.parse(File.read("C:\Temp\3")) hme=JSON.parse(File.read("C:\Temp\4")) GC.start sleep 10
The above worked every time I ran it. BTW we are basically re-reading 2008R2.json from the fauxhai data: http://www.speedyshare.com/UsHVK/Temp.zip
So these are known segfaults in the parser:
FFI_Yajl::Parser.parse(nil)
FFI_Yajl::Parser.parse(1)
FFI_Yajl::Parser.parse(true)
If its something different, then I'm suspicious that the GC in the 2.0.0 ruby that we're using on windows is buggy because we're not seeing these kinds of reports on 1.9.3 in chef-client or on 2.1.2/2.1.3 in chef-dk on unix. I've also built Chef 12 on ruby 2.1.2 as an omnibus package and run it through our spec tests which pretty reliably abuse the parser incessantly and used to cause coredumps in the past and haven't been seeing coredumps. We can't upgrade windows to 2.1.3 right now because of a problem with the ffi gem on windows with 2.1.3 though.
I would agree with this assessment. The fact that a GC.start call always fails either means the GC is trying to free memory still in use by one of the c extensions or the GC itself is buggy on windows.
Next steps - wait for ruby upgrade at this point? I assume we arent seeing reports about this because few windows cookbooks have unit tests?
Yeah, debugging GC crashes is a large yak shave. If its just ruby 2.0.0 being problematic and 2.1.3 will fix it, then its a waste of time to try to track it down. If it replicates on ruby 1.9.3 on windows, or on any of the unix platforms, that'd be more disconcerting. Unfortunately, right now upgrading to 2.1.3 on windows is a different kind of yak shave where everyone with a precompiled windows gem needs to be bugged to upgrade (ffi, win32-api, etc) and that is all outside of our control.
chefspec works for me under the following:
rake
results in bundler running, then 3 chefspec examples passingIf you're still having this issue then perhaps I can try to replicate with your code.
unable to reproduce this on Win 7 with ChefDK 0.3.5.
I suspect that either a JSON gem bump, monkey patch fixes for ffi-yajl or something else might have fixed this. If this can still be reproduced on the latest ChefDK, please comment and I will reopen.
yeah most likely JSON fighting with ffi-yajl/json_gem, which is gone now.
Log: