JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.61k stars 5.48k forks source link

Instant crashes with multithreading - no way of determining error #45796

Open jonbarkerlondon opened 2 years ago

jonbarkerlondon commented 2 years ago

Hi All,

I originally posted this to the discourse, but didn't get an answer there.

I have a multi-process, multi-threaded application. The problem is that when running in threaded mode, the (non main) processes die off gradually, without any error.

It's very possible that there's a multithreading bug in my code, but my issue with Julia is that there's no obvious way of figuring out what the bug is. No callstack or anything - the julia process just quits.

Is this a julia bug? Does anyone know how I should go about determining the cause of my issue. I attempted to use GDB in order to debug, but unfortunately I get constant SIGINTS and if I ignore them then I don't get a breakpoint hitting when the process fails.

Thanks

vtjnash commented 2 years ago

When using gdb, put a breakpoint in sigdie_handler in addition to ignoring SIGSEGV. Also, use rr instead (if you can).

jonbarkerlondon commented 2 years ago

Thanks - i'll give sigdie_handler a go.

Regarding RR - i'm on windows, but i spawned up a ubuntu VM, however it seems to run 10x slower in the VM so I haven't had a real chance to hit the error yet.

jonbarkerlondon commented 2 years ago

When using gdb, put a breakpoint in sigdie_handler in addition to ignoring SIGSEGV. Also, use rr instead (if you can).

I've attempted to put a breakpoint on 'jl_critical_error' (sigdie is for linux) however the problem i'm having is that having GDB attached slows down the process enough that the error doesn't appear to occurr.

Before attachment, the process runs at 100% CPU, after, it runs from 0-100% cpu, mainly 0% with the GDB process taking a constant 6% CPU. I'm guessing RR under linux might be the better option?

vtjnash commented 2 years ago

Don't use julia-debug as it breaks gdb and makes it slow

jonbarkerlondon commented 2 years ago

Don't use julia-debug as it breaks gdb and makes it slow

I'm just using regular julia for this. Unfortunately no sign of the crash yet with GDB attached.