search

oracle / truffleruby

A high performance implementation of the Ruby programming language, built on GraalVM.
https://www.graalvm.org/ruby/
Other
3k stars 183 forks source link

Add support for assigning to `dfree` in native extensions #2830

Closed nirvdrum closed 1 year ago

nirvdrum commented 1 year ago

I'm working on a project that uses Commonmarker 0.23.6 to process text in the CommonMark format. It errors out because it tries to use the dfree field in rb_data_type_struct, which it looks like we don't currently implement.

https://github.com/gjtorikian/commonmarker/blob/v0.23.6/ext/commonmarker/commonmarker.c#L115

> ruby -v
truffleruby 23.0.0-dev-f3db7ba6, like ruby 3.1.3, GraalVM CE Native [aarch64-darwin]
RuntimeError: Neutered Exception Polyglot::ForeignException: Member 'dfree' not found.
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/ext/commonmarker/commonmarker.c:490:in `rb_node_unlink'
    /Users/nirvdrum/dev/workspaces/truffleruby-ws/graal/sdk/mxbuild/darwin-aarch64/GRAALVM_5080722BFA_JAVA19/graalvm-5080722bfa-java19-23.0.0-dev/Contents/Home/languages/ruby/lib/truffle/truffle/cext_ruby.rb:41:in `<unknown>'
    /Users/nirvdrum/dev/workspaces/truffleruby-ws/graal/sdk/mxbuild/darwin-aarch64/GRAALVM_5080722BFA_JAVA19/graalvm-5080722bfa-java19-23.0.0-dev/Contents/Home/languages/ruby/lib/truffle/truffle/cext_ruby.rb:41:in `CommonMarker::Node#delete'
    <... replaced ...>
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:16:in `CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:18:in `block in CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:17:in `CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:18:in `block in CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:17:in `CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:18:in `block in CommonMarker::Node#walk'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:17:in `CommonMarker::Node#walk'
    <... replaced ...>
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:44:in `CommonMarker::Renderer#render'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:28:in `block in CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `Array#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:24:in `block (2 levels) in CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/node.rb:72:in `CommonMarker::Node#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:24:in `block in CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `Array#each'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:21:in `CommonMarker::Renderer#out'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:53:in `CommonMarker::Renderer#document'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer/html_renderer.rb:6:in `CommonMarker::HtmlRenderer#document'
    /Users/nirvdrum/.gem/truffleruby/3.1.3/gems/commonmarker-0.23.6/lib/commonmarker/renderer.rb:38:in `CommonMarker::Renderer#render'
eregon commented 1 year ago

Seems a duplicate of https://github.com/oracle/truffleruby/issues/2732. I'd think https://github.com/oracle/truffleruby/issues/2771 which was fixed would fix this too, but maybe it's slightly different. I'll take a look.

nirvdrum commented 1 year ago

Oops. This is a dupe of #2732. I'm sorry about that. Just note that the version number is significant. commonmarker main tracks a pre-release version. I think it might not use dfree any longer, but I had some difficulties getting it to load at all. I'll have to look into that more some other time.

nirvdrum commented 1 year ago

I didn't look at the fix for #2771, but one thing that stands out is #2771 is about reading dfree whereas this issue and #2732 are about writing to the field.

eregon commented 1 year ago

Running the test suite I get:

dead handle 0xbad00000004a8b0 (com.oracle.truffle.api.CompilerDirectives.ShouldNotReachHere)
    from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:574)
    from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:520)
    from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.raiseError(UnwrapNode.java:95)
    from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.unwrapTaggedObject(UnwrapNode.java:80)
    from org.truffleruby.cext.UnwrapNodeGen$UnwrapNativeNodeGen.execute(UnwrapNodeGen.java:482)
    from org.truffleruby.cext.UnwrapNode.longToWrapper(UnwrapNode.java:253)
    from org.truffleruby.cext.UnwrapNodeGen.execute(UnwrapNodeGen.java:100)
    from org.truffleruby.cext.CExtNodes$CallWithCExtLockAndFrameAndUnwrapNode.callWithCExtLockAndFrame(CExtNodes.java:249)
    from org.truffleruby.cext.CExtNodesFactory$CallWithCExtLockAndFrameAndUnwrapNodeFactory$CallWithCExtLockAndFrameAndUnwrapNodeGen.execute(CExtNodesFactory.java:598)
    from org.truffleruby.language.locals.WriteLocalVariableNode.execute(WriteLocalVariableNode.java:28)
    from org.truffleruby.language.RubyNode.doExecuteVoid(RubyNode.java:64)
    from org.truffleruby.language.control.SequenceNode.execute(SequenceNode.java:34)
    from org.truffleruby.core.module.ModuleNodes$DefineMethodNode$CallMethodWithLambdaBody.execute(ModuleNodes.java:1482)
    from org.truffleruby.language.RubyLambdaRootNode.execute(RubyLambdaRootNode.java:84)
/home/eregon/.rubies/truffleruby-dev/lib/truffle/truffle/cext_ruby.rb:22:in `next'
    from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:71:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:133:in `strong'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:23:in `block (2 levels) in paragraph'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:84:in `container'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:22:in `block in paragraph'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:78:in `block'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:21:in `paragraph'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:53:in `document'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:6:in `document'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:38:in `render'
    from /home/eregon/code/commonmarker/test/test_spec.rb:26:in `test_sourcepos_example_404'

So the dree doesn't seem the only issue.

BTW I noticed the gem switched from libcmark-gfm for 0.x to comrak for 1.0, which is Rust-based.

eregon commented 1 year ago

I have a PR to support writing to dfree: https://github.com/oracle/truffleruby/pull/2841 With that, the test suite fails like so (note: it's actually a lot faster to run the test suite on jvm-ce than jvm):

$ chruby truffleruby-jvm-ce
$ bundle exec rake test TESTOPTS="-s0 -v"
...
TestSpec#test_to_html_example_553 = 0.00 s = .
TestSpec#test_sourcepos_example_436 = 
truffleruby: Unexpected internal exception in at_exit,
please report it to https://github.com/oracle/truffleruby/issues.

dead handle 0xbad000000002280 (com.oracle.truffle.api.CompilerDirectives.ShouldNotReachHere)
    from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:574)
    from com.oracle.truffle.api.CompilerDirectives.shouldNotReachHere(CompilerDirectives.java:520)
    from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.raiseError(UnwrapNode.java:95)
    from org.truffleruby.cext.UnwrapNode$UnwrapNativeNode.unwrapTaggedObject(UnwrapNode.java:80)
    from org.truffleruby.cext.UnwrapNodeGen$UnwrapNativeNodeGen.execute(UnwrapNodeGen.java:482)
    from org.truffleruby.cext.UnwrapNode.longToWrapper(UnwrapNode.java:253)
    from org.truffleruby.cext.UnwrapNodeGen.execute(UnwrapNodeGen.java:100)
    from org.truffleruby.cext.CExtNodes$CallWithCExtLockAndFrameAndUnwrapNode.callWithCExtLockAndFrame(CExtNodes.java:249)
    from org.truffleruby.cext.CExtNodesFactory$CallWithCExtLockAndFrameAndUnwrapNodeFactory$CallWithCExtLockAndFrameAndUnwrapNodeGen.execute(CExtNodesFactory.java:599)
    from org.truffleruby.language.locals.WriteLocalVariableNode.execute(WriteLocalVariableNode.java:28)
    from org.truffleruby.language.RubyNode.doExecuteVoid(RubyNode.java:64)
    from org.truffleruby.language.control.SequenceNode.execute(SequenceNode.java:34)
    from org.truffleruby.core.module.ModuleNodes$DefineMethodNode$CallMethodWithLambdaBody.execute(ModuleNodes.java:1482)
    from org.truffleruby.language.RubyLambdaRootNode.execute(RubyLambdaRootNode.java:84)
/home/eregon/code/truffleruby-ws/graal/sdk/mxbuild/linux-amd64/GRAALVM_A33290E019_JAVA20/graalvm-a33290e019-java20-23.0.0-dev/languages/ruby/lib/truffle/truffle/cext_ruby.rb:22:in `next'
    from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:71:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:133:in `strong'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:23:in `block (2 levels) in paragraph'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:84:in `container'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:22:in `block in paragraph'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:78:in `block'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:21:in `paragraph'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:44:in `render'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:28:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block (2 levels) in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/node.rb:72:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:24:in `block in out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `each'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:21:in `out'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:53:in `document'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer/html_renderer.rb:6:in `document'
    from /home/eregon/code/commonmarker/lib/commonmarker/renderer.rb:38:in `render'
    from /home/eregon/code/commonmarker/test/test_spec.rb:26:in `test_sourcepos_example_436'

I'll merge that first and we can investigate this dead handle issue later, they don't seem related.

Command-line to repro similar error: ruby -rbundler/setup -Ilib:test test/test_spec.rb -s0 -v

The whole test suite passes if we just keep all handles (obviously just a workaround): TRUFFLERUBYOPT="--experimental-options --keep-handles-alive" bundle exec rake test TESTOPTS="-s0 -v"

eregon commented 1 year ago

Tracking the dead handle issue in https://github.com/oracle/truffleruby/issues/2842