macOS memory corruption on both arm64 and amd64 causing intermittent test failures

[eh-steve]

I don’t think I’ve seen this on anything other than darwin, which is why I suspect OS signal pre-emptions clobbering registers…

amd64: https://github.com/eh-steve/goloader/actions/runs/5717229081/job/15490546426

arm64: https://github.com/eh-steve/goloader/actions/runs/5717229081/job/15490548475

Suspects?:

• Preemptions from profiling signals at unsafe points (bad pcvalue data or register clobbering)
• Bad/missing gcprogs/gcmasks
• Stack moves using bad pcvalue data for gentraceback

[pkujhd]

@eh-steve, maybe this problem like pkujhd#87, but on arm64. ARM64 call instrcution now we use LDR & BR to replace CALL.

[eh-steve]

I think this is unlikely, given that it seems to affect both amd64 and arm64 on darwin, and most of the relocations are guaranteed to be within 32-bits due to mmap manager and so don't hit the alternative path...

[eh-steve]

That said, I might try a branch with >32 bit CALL relocs written as JMP then CALL then JMP instead of CALL then JMP for both arm64 and amd64 and see if it makes a difference...

[eh-steve]

Hmm, not super encouraging - I think the culprit is still elsewhere given we still get (at least) 2 failures on darwin while everything else is fine:

amd64: https://github.com/eh-steve/goloader/actions/runs/6103632297/job/16564380843#step:6:36 arm64: https://github.com/eh-steve/goloader/actions/runs/6103632297/job/16570982742#step:6:408

[pkujhd]

Hmm, not super encouraging - I think the culprit is still elsewhere given we still get (at least) 2 failures on darwin while everything else is fine: I don't reappear pkujhd#87 issue on linux/arm64(on qemu). maybe arm64 is not same as amd64. Maybe this issue is not relevant

[glycerine]

I reproduced this once (out of like 20 runs).

I wonder if something like record-and-replay could help here: https://rr-project.org/

[eh-steve]

Yeah, annoyingly I don't know of any MacOS record/replay debuggers... I might need to instrument the runtime to catch it

[glycerine]

Arg. Sorry. I thought it had macOS support. The commercial version might though?

https://www.replay.io/

hmm.... but might be in-browser only? hard to tell without looking deeply-- they say:

"Replay.io’s recorder is designed to be runtime and platform agnostic. Firefox for Mac, Windows, Linux Chrome for Linux Chrome for Mac and Windows in progress Major CI environments Node for Mac, and Linux in beta."

[eh-steve]

That's a browser frontend debugger, nothing to do with rr?

[glycerine]

Awdrat. Egg-on-face. Sorry(!), just brainstorming here. I'd really love to have a plugin replacement that was cross platform.

[eh-steve]

At some point I hope to get around to investigating this, and the other remaining issues to provide that