Crash on early return. - Githubissues

gogins commented 2 years ago

Bower                    i   10.0000 t  359.5850 d    0.3483 k   93.0000 v   50.0000 p    0.0823 #  5
Bower                    i   10.0000 t  359.7294 d    0.2469 k   95.0000 v   50.0000 p    0.3221 #  5
Bower                    i   10.0000 t  359.7531 d    0.2469 k   96.0000 v   50.0000 p    0.2759 #  6
exitnow i    0.0000 t   15.0000 d  365.0000 k    0.0000 v    0.0000 p    0.0000 #  0
Early return from csoundPerform().
deativate: ip, nxt = 0x4d2ee950 , 0x4fda6860
deativate: ip, nxt = 0x4fda6860 , 0x3ee02060
deativate: ip, nxt = 0x3ee02060 , 0x4e8032d0
deativate: ip, nxt = 0x4e8032d0 , 0x37ec87d0
deativate: ip, nxt = 0x37ec87d0 , (nil)
==568699== Invalid write of size 8
==568699==    at 0x843E9FA: VSTPlugin::Free() (vsthost.cpp:549)
==568699==    by 0x843EA7D: VSTPlugin::~VSTPlugin() (vsthost.cpp:356)
==568699==    by 0x843EB6C: VSTPlugin::~VSTPlugin() (vsthost.cpp:357)
==568699==    by 0x843CB7F: csoundModuleDestroy (vst4cs.cpp:606)
==568699==    by 0x4A605C4: csoundDestroyModules (csmodule.c:918)
==568699==    by 0x48CD3D0: reset (csound.c:3346)
==568699==    by 0x48CE32D: csoundDestroy (csound.c:1424)
==568699==    by 0x10A840: main (csound_main.c:332)
==568699==  Address 0x2d1b9cf8 is 200 bytes inside a block of size 1,160 free'd
==568699==    at 0x483D1CF: operator delete(void*, unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==568699==    by 0x269E8E2D: ??? (in /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.lv2/Pianoteq_7.so)
==568699==    by 0x843E9F5: Dispatch (vsthost.h:215)
==568699==    by 0x843E9F5: VSTPlugin::Free() (vsthost.cpp:548)
==568699==    by 0x843EA7D: VSTPlugin::~VSTPlugin() (vsthost.cpp:356)
==568699==    by 0x843EB6C: VSTPlugin::~VSTPlugin() (vsthost.cpp:357)
==568699==    by 0x843CB7F: csoundModuleDestroy (vst4cs.cpp:606)
==568699==    by 0x4A605C4: csoundDestroyModules (csmodule.c:918)
==568699==    by 0x48CD3D0: reset (csound.c:3346)
==568699==    by 0x48CE32D: csoundDestroy (csound.c:1424)
==568699==    by 0x10A840: main (csound_main.c:332)
==568699==  Block was alloc'd at
==568699==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==568699==    by 0x269E3FB5: ??? (in /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.lv2/Pianoteq_7.so)
==568699==    by 0x843ECFD: VSTPlugin::Instantiate(char const*) (vsthost.cpp:474)
==568699==    by 0x843CA2B: vstinit (vst4cs.cpp:110)
==568699==    by 0x48E672A: init0 (insert.c:255)
==568699==    by 0x48F209F: musmon (musmon.c:301)
==568699==    by 0x4A6340B: csoundStart (main.c:562)
==568699==    by 0x10A68F: main (csound_main.c:326)
==568699==

gogins commented 2 years ago

See if this happens with another means of stopping the performance.

gogins commented 2 years ago

I fixed this but there is still a problem. Try:

[x] xanadu.csd. No problem.
[x] Examine addresses in valgrind log. No help.
[ ] Incrementally add plugins.
[x] Track atexit calls. There is only one.

gogins commented 2 years ago

I suspect this is due to the vst3 opcodes using a different way of freeing memory when the opcode module is destroyed. Looks like the atexit handler for static memory destructors. In the other opcodes, all contents of the std collections with allocated objects are destroyed in csoundModuleDestroy. The empty collections can then be deleted properly at exit. I will try changing the vst3 opcodes to follow the same pattern.

However, remove the vst3 plugins module does not remove the crash.

gogins commented 2 years ago

The other examples for the Webkit opcodes do not crash.

gogins commented 2 years ago

All right, I have zeroed in on the cause:

deativate: ip, nxt = 0x55d5f35c6db0 , 0x55d5f9ab9bc0
deativate: ip, nxt = 0x55d5f9ab9bc0 , 0x55d5fa2a5030
deativate: ip, nxt = 0x55d5fa2a5030 , 0x55d5f9f94360
deativate: ip, nxt = 0x55d5f9f94360 , (nil)
csoundModuleDestroy_webkit_opcodes: csound: 0x55d5e62c6940...
csoundModuleDestroy_webkit_opcodes: clear...
CsoundWebKit::~CsoundWebKit...
CsoundWebKit::~CsoundWebKit.
csoundModuleDestroy_webkit_opcodes: clear.
csoundModuleDestroy_webkit_opcodes: csound: 0x55d5e62c6940.
csoundModuleDestroy (vst3_opcodes): csound: 0x55d5e62c6940 thread: 140437922747008...
csoundModuleDestroy (vst3_opcodes): csound: 0x55d5e62c6940.

csound command: Segmentation fault
file:///home/mkg/webkit-opcodes/csound.js:38:35: CONSOLE JS ERROR TypeError: callback_error is not a function. (In 'callback_error(-32002, "AJAX Error")', 'callback_error' is undefined)
Segmentation fault (core dumped)
mkg@xenakis:~/webkit-opcodes$

It is that the Csound proxy continues to function but there is nothing on the C++ side to call any more.

gogins commented 2 years ago

1 Here's the source code of exit: kevin.vanzonneveld.net/techblog/article/… – splattne Feb 15 '09 at 9:24 That code doesn't appear to stop events on elements from firing, just the window's events. So you'd also need to loop through all the elements on the page doing something similar. It all sounds like a very odd requirement though. – andynormancx Feb 15 '09 at 9:31 3 the whole die concept is a bit broken - the flow should be capable of handling any and all eventualities, whether that reqire try-catch or not. – annakata Feb 15 '09 at 9:39 4 You can't stop XMLHttpRequest handlers. The current code does not stop intervals or timeouts from executing. Demo: jsfiddle.net/skibulk/wdxrtvus/1 You might consider this as a fix: stackoverflow.com/a/8860203/6465140 – skibulk Aug 9 '16 at 16:39

gogins commented 2 years ago

After reworking memory management for C++ objects in plugins, I think I have found the smoking gun:

Thread 607 "MHD-worker" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec37fe700 (LWP 1098226)]
0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in  ()
#1  0x00007fffde9a905f in jsonrpc::HttpServer::callback(void*, MHD_Connection*, char const*, char const*, char const*, char const*, unsigned long*, void**) ()
    at /usr/lib/x86_64-linux-gnu/libjsonrpccpp-server.so.0
#2  0x00007fffdd7227e1 in  () at /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#3  0x00007fffdd724838 in  () at /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#4  0x00007fffdd7268de in  () at /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#5  0x00007fffdd72a897 in  () at /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#6  0x00007fffdd72c4d1 in  () at /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#7  0x00007fffdd72c5ad in  () at /usr/lib/x86_64-linux-gnu/libmicrohttpd.so.12
#8  0x00007ffff7bf4609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#9  0x00007ffff7b1b293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

What needs to happen here is for jsonrpc::HttpServer to abort pending requests, perhaps as part of StopListening.

gogins commented 2 years ago

I am doing all heap object memory management for my Csound plugins with the new PluginBase.hpp now. We will see if this helps or if it needs to be improved.

gogins commented 2 years ago

These are my plugins and their status:

[x] ampmidid, not needed.
[ ] clang-opcodes, could be tricky, there are two objects with heap allocations, the lower half JIT compiler and the upper half Clang compiler.
[x] CMask opcodes, not needed.
[x] csound-extended/MVerb, redone.
[x] csound-vst.
[x] csound-vst3-opcodes, probably not needed, slightly different pattern that looks OK.
[x] doppler, probably not needed, managed in noteoff.
[x] mixer, probably not needed, uses Csound global variable.
[ ] plugins/AbletonLinkOpcodes, probably needs something.
[x] plugins/chua, not needed.
[x] plugins/fluidSynth, probably not needed, uses Csound global variable.
[x] plugins/jackops, probably not needed, uses Csound global variable.
[ ] plugins/linear_algebra, might need something.
[x] plugins/stk/stkops, done and lightly tested.
[x] padsynth_gen, not needed.
[x] signal flow graph, probably not needed, uses Csound global variable.
[x] webkit-opcodes.

Before booking this stuff in I need to create a plugin-base branch of Csound to ensure that it builds on all platforms.

gogins commented 2 years ago

Might want to fix these as long as I'm on it:

==1192983== Conditional jump or move depends on uninitialised value(s)
==1192983==    at 0x72E7B89: csoundModuleInit (rtalsa.c:1944)
==1192983==    by 0x4A607EF: csoundInitModule (csmodule.c:666)
==1192983==    by 0x4A6090A: csoundInitModules (csmodule.c:721)
==1192983==    by 0x4A6299D: csoundCompileArgs (main.c:306)
==1192983==    by 0x4A63C5C: csoundCompile (main.c:567)
==1192983==    by 0x10A68F: main (csound_main.c:326)
==1192983==  Uninitialised value was created by a stack allocation
==1192983==    at 0x72E78F0: csoundModuleInit (rtalsa.c:1898)
==1192983== 
==1192983== Conditional jump or move depends on uninitialised value(s)
==1192983==    at 0x663F1FF: csoundModuleInit (rtpa.c:886)
==1192983==    by 0x4A607EF: csoundInitModule (csmodule.c:666)
==1192983==    by 0x4A6090A: csoundInitModules (csmodule.c:721)
==1192983==    by 0x4A6299D: csoundCompileArgs (main.c:306)
==1192983==    by 0x4A63C5C: csoundCompile (main.c:567)
==1192983==    by 0x10A68F: main (csound_main.c:326)
==1192983==  Uninitialised value was created by a stack allocation
==1192983==    at 0x663F0B0: csoundModuleInit (rtpa.c:876)

gogins commented 2 years ago

valgrind is not helping here:

csound command: Segmentation fault
==1192983== Jump to the invalid address stated on the next line
==1192983==    at 0x3E0E2F00: ???
==1192983==    by 0x4C72A26: __run_exit_handlers (exit.c:108)
==1192983==    by 0x4C72BDF: exit (exit.c:139)
==1192983==    by 0x10AA69: signal_handler (csound_main.c:233)
==1192983==    by 0x1E25AA19: ??? (in /usr/lib/x86_64-linux-gnu/libjavascriptcoregtk-4.0.so.18.19.6)
==1192983==    by 0x4C1B41F: ??? (in /lib/x86_64-linux-gnu/libpthread-2.31.so)
==1192983==    by 0x3E0E2F2F: ???
==1192983==    by 0x4C72A26: __run_exit_handlers (exit.c:108)
==1192983==    by 0x4C72BDF: exit (exit.c:139)
==1192983==    by 0x4C500B9: (below main) (libc-start.c:342)
==1192983==  Address 0x3e0e2f00 is not stack'd, malloc'd or (recently) free'd
==1192983==

gogins commented 2 years ago

Also there is this:

[Switching to Thread 0x7fffc67f4700 (LWP 1419463)]

--Type <RET> for more, q to quit, c to continue without paging--c
Thread 23 "smpmgr-2" hit Hardware watchpoint 1: *0x00007fffd0bc6f60

Old value = <unreadable>
New value = 0
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
#1  0x00007fffd9f26fbb in  () at /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
#2  0x00007fffd9f28e5c in  () at /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
#3  0x00007fffd9f6428c in  () at /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
#4  0x00007fffd9f64b8c in  () at /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
#5  0x00007fffda00931f in  () at /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
#6  0x00007fffda0089ff in  () at /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
#7  0x00007ffff7bf3609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#8  0x00007ffff7b1a293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) info thread
  Id   Target Id                                        Frame 
  1    Thread 0x7ffff6607280 (LWP 1419441) "csound"     0x00007fffd9fe3903 in ?? () from /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
  2    Thread 0x7fffd99dd700 (LWP 1419442) "csound"     0x00007ffff7ad83bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=0x7fffd99dcc50, rem=0x0)
    at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
  3    Thread 0x7fffd37f7700 (LWP 1419443) "csound"     futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fffd37f6c50, clockid=<optimized out>, expected=0, 
    futex_word=0x5555559b8398) at ../sysdeps/nptl/futex-internal.h:320
  4    Thread 0x7fffd2ede700 (LWP 1419444) "JUCE Timer" futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fffd2eddbb0, clockid=<optimized out>, expected=0, 
    futex_word=0x555555a3a2a8) at ../sysdeps/nptl/futex-internal.h:320
  22   Thread 0x7fffc5ff3700 (LWP 1419462) "smpmgr-1"   0x00007fffd9fe4042 in ?? () from /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
* 23   Thread 0x7fffc67f4700 (LWP 1419463) "smpmgr-2"   __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
  24   Thread 0x7fffc6ff5700 (LWP 1419464) "smpmgr-3"   __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
  25   Thread 0x7fffc77f6700 (LWP 1419465) "smpmgr-4"   0x00007fffd9fe3aac in ?? () from /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
  26   Thread 0x7fffd25d8700 (LWP 1419466) "smpmgr-5"   0x00007fffd9fe3a0b in ?? () from /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
  27   Thread 0x7fffd1d29700 (LWP 1419467) "smpmgr-6"   0x00007fffd9fe38da in ?? () from /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
  28   Thread 0x7fffd14d1700 (LWP 1419468) "smpmgr-7"   0x00007fffd9fe38ab in ?? () from /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.so
(gdb)

I'm checking for a newer version of the Pianoteq plugin. I have the latest version, and there is not much in the settings that coul affect a crash, possibly using multi-threaded rendering.

gogins commented 2 years ago

I fixed up Csound's OpcodeBase.hpp header for c++11.

gogins commented 2 years ago

I've done a lot but there obviously is more that could be done. I am still getting the same error. I will now concentrate on:

[ ] Debugging the origin of the unreachable address.
[ ] Trying to shutdown the XMLHttpRequest responses.

gogins commented 2 years ago

It does seem clear that the problem occurs after the VST plugin modules have been unloaded. I am not sure that the unloading contributes to the crash, but it might if there are static non-POD objects in the module.

gogins commented 2 years ago

CsoundWebKit::~CsoundWebKit: terminate Web process...
CsoundWebKit::~CsoundWebKit.
####### csoundModuleDestroy_clang_opcodes: reset jit_compiler: currently 0x3d363a10
####### JITCompiler::~JITCompiler: deleting JITCompiler 0x3d363a10.
####### JITCompiler::~JITCompiler: execution session has ended.
####### csoundModuleDestroy_clang_opcodes: reset jit_compiler: reset to (nil)
csoundModuleDestroy (vst3_opcodes): csound: 0x623f3f0 thread: 102813632...
csoundModuleDestroy (vst3_opcodes): csound: 0x623f3f0.
==1430398== Invalid write of size 8
==1430398==    at 0x8246BAA: VSTPlugin::Free() (vsthost.cpp:544)
==1430398==    by 0x824702D: VSTPlugin::~VSTPlugin() (vsthost.cpp:351)
==1430398==    by 0x824713C: VSTPlugin::~VSTPlugin() (vsthost.cpp:352)
==1430398==    by 0x82425BD: module_destroy (OpcodeBase.hpp:479)
==1430398==    by 0x82425BD: csoundModuleDestroy (vst4cs.cpp:595)
==1430398==    by 0x4A60A24: csoundDestroyModules (csmodule.c:918)
==1430398==    by 0x48CD3F0: reset (csound.c:3346)
==1430398==    by 0x48CE34D: csoundDestroy (csound.c:1424)
==1430398==    by 0x10A840: main (csound_main.c:332)
==1430398==  Address 0x2a2f1848 is 200 bytes inside a block of size 1,160 free'd
==1430398==    at 0x483D1CF: operator delete(void*, unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1430398==    by 0x248FFE2D: ??? (in /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.lv2/Pianoteq_7.so)
==1430398==    by 0x8246BA5: Dispatch (vsthost.h:217)
==1430398==    by 0x8246BA5: VSTPlugin::Free() (vsthost.cpp:543)
==1430398==    by 0x824702D: VSTPlugin::~VSTPlugin() (vsthost.cpp:351)
==1430398==    by 0x824713C: VSTPlugin::~VSTPlugin() (vsthost.cpp:352)
==1430398==    by 0x82425BD: module_destroy (OpcodeBase.hpp:479)
==1430398==    by 0x82425BD: csoundModuleDestroy (vst4cs.cpp:595)
==1430398==    by 0x4A60A24: csoundDestroyModules (csmodule.c:918)
==1430398==    by 0x48CD3F0: reset (csound.c:3346)
==1430398==    by 0x48CE34D: csoundDestroy (csound.c:1424)
==1430398==    by 0x10A840: main (csound_main.c:332)
==1430398==  Block was alloc'd at
==1430398==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1430398==    by 0x248FAFB5: ??? (in /home/mkg/Pianoteq 7/x86-64bit/Pianoteq 7.lv2/Pianoteq_7.so)
==1430398==    by 0x8246D8D: VSTPlugin::Instantiate(char const*) (vsthost.cpp:469)
==1430398==    by 0x82421F1: vstinit (vst4cs.cpp:105)
==1430398==    by 0x48E674A: init0 (insert.c:255)
==1430398==    by 0x48F20BF: musmon (musmon.c:301)
==1430398==    by 0x4A6386B: csoundStart (main.c:562)
==1430398==    by 0x10A68F: main (csound_main.c:326)
==1430398== 
==1430398== Jump to the invalid address stated on the next line
==1430398==    at 0x3E6EBF60: ???
==1430398==    by 0x4C72A26: __run_exit_handlers (exit.c:108)
==1430398==    by 0x4C72BDF: exit (exit.c:139)
==1430398==    by 0x4C500B9: (below main) (libc-start.c:342)
==1430398==  Address 0x3e6ebf60 is not stack'd, malloc'd or (recently) free'd
==1430398==

gogins commented 2 years ago

Trying to isolate the hanging allocation...

[x] vst4cs, no help.
[x] Webkit opcodes, no help.
[x] Clang opcodes, not applicable.

I hate to say it, but WebKit and LLVM/Clang are huge hunks of software... I'm letting this rest.

gogins / csound-webserver-opcodes

Crash on early return. #10