Access Violations in Sketchup’s RubyVM during mouse events

[JohannStudanskiEnscape]

1. SketchUp/LayOut Version: Sketchup Pro 22.0.354
2. OS Platform: Windows 10

Describe briefly what you are doing and what is happening.

We’re having an issue where after some time, we randomly get Access Violations in Sketchup’s Ruby VM , or to be more specific, in both rb_class_of during return RBASIC(obj)->klass; (when resolving ->klass, usually for calling enableVCB?) and in method_entry_get when trying to access the macro GLOBAL_METHOD_CACHE with a 0x0000’d klass.

All those calls result from (Windows) mouse events that go through Sketchup (where I don’t have a pdb to check what Sketchup is doing exactly) after we finish and return from our tool operations. Our current theory is that it is likely connected to tool pushing and popping (we’re using our custom placement tool in those cases), but it’s happening after we’ve finished all our operations, and as far as I can tell, the involved pointers are none that we allocated beforehand (we’ve tracked our Ruby allocations multiple times, and in no case did the pointers to our allocated values match the VALUE obj or klass pointers, we’re not even in a close range).

Any help would be appreciated—while it’s not 100% reliably reproducible, it can happen within a couple placements, and since it crashes Sketchup as a whole, it’s quite high impact.

Describe what the expected result is.

No crash.

Reproducible code

Unfortunately not easily possible to include, especially since we're not exactly sure what exactly triggers this condition.

If you have access to Enscape in Sketchup, it can happen randomly after you placed a couple light and / or sound objects, especially in more complex scenes, but there's no reliable repro case—it's probably timing or usage dependent and can take a couple minutes and placed lights to happen. If you need an Enscape license to test it on your end, we can certainly arrange that.

Truncated callstack from one of the crashes:

	Callstack	-
	x64-msvcrt-ruby270.dll!rb_class_of(unsigned __int64 obj) Line 2105	C
	x64-msvcrt-ruby270.dll!rb_obj_respond_to(unsigned int64 obj, unsigned int64 id, int priv) Line 2184	C
	SketchUp.exe!00007ff61eed4376()	Unknown
	SketchUp.exe!00007ff61ec0061f()	Unknown
	mfc140u.dll!CWnd::OnWndMsg(unsigned int message, unsigned int64 wParam, int64 lParam, __int64 * pResult) Line 2699	C++

message is 0x00000201 (WM_LBUTTONDOWN)

I can of course also provide a minidump (with or without heap) if it helps.

---

This issue has also been posted in your forums: https://forums.sketchup.com/t/access-violations-in-sketchups-rubyvm-during-mouse-events/245699, and I submitted a bugsplat “Crash #652379”—the description contains a link to the forum thread, too.

[thomthom]

Are you also testing other SketchUp versions or platforms? Do you see this in the latest released version of SU? 23.1?

[JohannStudanskiEnscape]

Yeah, getting it in SU2023, too:

[sketchup[bot]]

Logged as: SKEXT-3885

[thomthom]

Windows only? Or mac?

[JohannStudanskiEnscape]

I haven't been able to reproduce it on the Mac so far. But that could also be due to it being generally difficult to reproduce reliably and the Mac version maybe having a different GC behavior—so I can't guarantee it's not happening on the Mac, too.

[JohannStudanskiEnscape]

Any progress or news on this?

[AlexandreGuillemeMS]

Hi, I think I've fall into the same issue, same stack trace and same context, a custom tool. Looking at the log, I saw that it was ending by :

Tool(SelectionTool) Tool(SelectionTool)

After reviewing the ruby code, I saw that I was removing the tool twice :

...
        Sketchup.active_model.select_tool(self)
...

    def onLButtonDown(flags, x, y, view)
        inputPt = view.inputpoint(x, y)
        ...
        Sketchup.active_model.select_tool(nil) #first remove
        Sketchup.active_model.tools.pop_tool #unnecessary call, same action than previous line
    end

I commented up the last line (pop_tool) and now all is good, no more crash. Hope this helps.

[ChaosPepp]

Hi, I think I've fall into the same issue, same stack trace and same context, a custom tool. Looking at the log, I saw that it was ending by :

Tool(SelectionTool) Tool(SelectionTool)

We did indeed have one such occasion and can confirm that popping tools twice (right after one another and when there is only one in the stack to begin with) leads to a crash. So thanks for the hint! However, this is not the cause of the main issue at hand as there are other crashes without double pops. It seems like SketchUp crashes whenever you manually "force" a change to the toolstack while - I assume - it's doing some work on or with the current stack. Interestingly, it appears as if these crashes can (in some circumstances at least) be prevented by enabling the feature "Automatically fix problems when they are found". This only works in SU24 though, not 23 or of course older version where this setting does not exist.