Random Crashes during longer sessions

klopstock-dviz commented 3 years ago

Describe the bug App crashes repeatadly, sometimes 2 or 3 times per day; other times 2 or 3 times per hour

To Reproduce Nothing particular

Screenshots Logs on the terminal chougar@pc-argon:~$ antimicrox free(): invalid pointer Abandon (core dumped) chougar@pc-argon:~$ antimicrox free(): invalid pointer Abandon (core dumped) chougar@pc-argon:~$ antimicrox Erreur de segmentation (core dumped) chougar@pc-argon:~$ antimicrox free(): invalid pointer Abandon (core dumped)

Configuration Version of antimicrox: 3.1.3 Used package: github sources System version: LMDE 3

Additional context I share my profile, perhaps it is the source of the problem: total war.gamecontroller.amgp.tar.gz

additional info I use an xbox 360 controller The crashes happen only when I use the controller I am not sure if it is systematic but i have noticed the crashes happen when i fully push the directionnal stick

noisecode3 commented 2 years ago

I have more information that can help. the freeze only happens if my ps4 is using Bluetooth. it seems related to the crash.

TheFeelTrain commented 2 years ago

I do not get the deadlock error as far as I can tell. It is definitely related to mouse input, but I use a wired Xbox One controller. So I don't think it has anything to do with PlayStation or Bluetooth.

noisecode3 commented 2 years ago

well that data could matter and the operating system could manage memory and threads a bit different every time you run the program. It can still be related to the same problem. SDL2 is used. you never know, I don't think speculation in it self helps tho.

leycec commented 2 years ago

Wowzers! This has been open unresolved for a year now. I'm seeing this as well with a DS4 connected via wired USB. :sob:

The crashes appear to be non-deterministic; there doesn't appear to be an underlying correlation to input device, activity, or frequency. AntiMicroX just crashes "out of the blue." I agree with @TheFeelTrain: this is unlikely to be correlated to any specific device (e.g., PlayStation DS4) or communication technology (e.g., Bluetooth).

Interestingly, AntiMicro itself suffers a similar issue. So, this issue almost certainly predates the AntiMicroX fork. It's likely that AntiMicro(X)'s usage of one or more lower-level libraries (e.g., Qt 5, SDL2) is the culprit here. Something fundamentally unsafe with memory is being done.

Unfortunately, it doesn't seem like this will be resolved in either project anytime soon. Non-deterministic crashing every several hours is a hard blocker to fun, so... in my case, I'm reluctantly reverting back to QJoyPad. It ain't sexy, but it works reliably.

TheFeelTrain commented 2 years ago

@leycec I haven't tried it myself but since the crash happens when controlling the mouse, I've thought about running QJoyPad exclusively for mouse input, then AntiMicroX for everything else (just unmap that stick). I think it would actually work.

I would just use QJoyPad on its own but it's missing the ability to map mouse scroll which is really hard to go back to.

leycec commented 2 years ago

Huh. That's... a great idea! I love me some AntiMicroX when it actually works. So, that's a useful compromise between "No AntiMicroX: sad face" and "Full AntiMicroX: everything broke."

Speaking of broke, AntiMicroX actually just spewed out this fullbore stacktrace after yet another segmentation fault:

❌ERROR  Received SIGSEGV (segmentation fault)
❗WARN   Stack trace:
❗WARN   antimicrox(+0x81071) [0x55783a489071]   
❗WARN   /lib64/libpthread.so.0(+0x12690) [0x7f9dd13b2690]   
❗WARN   /usr/lib64/libQt5Gui.so.5(_ZN7QRegionpLERK5QRect+0xe5) [0x7f9dd113d985]     
❗WARN   /usr/lib64/libQt5Widgets.so.5(+0x17b7df) [0x7f9dd19c57df]   
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN7QWidget6updateERK5QRect+0xfa) [0x7f9dd19de4fa]    
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN7QWidget6updateEv+0x3a) [0x7f9dd19de59a]   
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN7QWidget5eventEP6QEvent+0x1f6) [0x7f9dd19f43b6]    
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent+0x7f) [0x7f9dd19afd1f]  
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent+0x118) [0x7f9dd0aa5f58]     
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN14QWidgetPrivate22propagatePaletteChangeEv+0x85) [0x7f9dd19e45e5]  
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN14QWidgetPrivate17setPalette_helperERK8QPalette+0x58) [0x7f9dd19e47f8]     
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN7QWidget10setPaletteERK8QPalette+0x6a) [0x7f9dd19e77ca]    
❗WARN   /usr/lib64/libQt5Widgets.so.5(+0x213f3c) [0x7f9dd1a5df3c]   
❗WARN   /usr/lib64/libQt5Widgets.so.5(+0x2142c5) [0x7f9dd1a5e2c5]   
❗WARN   antimicrox(+0xf26ed) [0x55783a4fa6ed]   
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN7QObject5eventEP6QEvent+0x2b2) [0x7f9dd0ad2792]   
❗WARN   /usr/lib64/libQt5Widgets.so.5(_ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent+0x7f) [0x7f9dd19afd1f]  
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent+0x118) [0x7f9dd0aa5f58]     
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN23QCoreApplicationPrivate16sendPostedEventsEP7QObjectiP11QThreadData+0x18d) [0x7f9dd0aa95ed]  
❗WARN   /usr/lib64/libQt5Core.so.5(+0x2d7053) [0x7f9dd0afa053]  
❗WARN   /usr/lib64/libglib-2.0.so.0(g_main_context_dispatch+0x26b) [0x7f9dcf5741cb]     
❗WARN   /usr/lib64/libglib-2.0.so.0(+0x55485) [0x7f9dcf574485]  
❗WARN   /usr/lib64/libglib-2.0.so.0(g_main_context_iteration+0x2f) [0x7f9dcf57454f]     
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0x64) [0x7f9dd0af9ac4]   
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE+0x123) [0x7f9dd0aa4843]    
❗WARN   /usr/lib64/libQt5Core.so.5(_ZN16QCoreApplication4execEv+0x8d) [0x7f9dd0aacecd]  
❗WARN   antimicrox(+0x5d396) [0x55783a465396]   
❗WARN   /lib64/libc.so.6(__libc_start_main+0xcd) [0x7f9dd030a7ed]   
❗WARN   antimicrox(+0x66ada) [0x55783a46eada]

Love the Unicode❗characters. Like... we get it. Horrifying bad stuff just happened.

As expected, the badness does indeed stem from some sort of harmful usage of Qt 5. From what I can make of this, AntiMicroX seems to be spamming Qt's event loop with an unholy number of widget events.

Does AntiMicroX support a headless daemon mode – say, something like a --daemon or --headless CLI option that just disables the Qt GUI entirely and runs as a quiet background daemon process? If not, adding that would absolutely be the fastest way to resolve this issue.

Since Qt is the problem, let's just cut Qt out of the picture entirely. You only ever need to run the AntiMicroX GUI when initially configuring a controller profile. Once that's nailed to the floor, the GUI is entirely superfluous and only gets in the way.

zpangwin commented 2 years ago

@leycec I haven't tried it myself but since the crash happens when controlling the mouse, I've thought about running QJoyPad exclusively for mouse input, then AntiMicroX for everything else (just unmap that stick). I think it would actually work.

Never noticed the correlation before but, like you were hypothesizing, I do have the mouse mapped for my right-stick and I do generally have my view (mouse-controls) go nuts when AMX does crash. I don't get crashes as frequently as I did when the ticket was first opened so I think there have been some significant improvements. These days, I might get 1-2 crashes during a 5-8 hour gaming session and Alt+tabbing and restarting AMX fixes it. It is extremely rare that I get a second crash the same night. Some nights, no crash at all. I'll try to see if I can get a fresh core dump next time it happens... but I admit that making sense of them is over my head.

Does AntiMicroX support a headless daemon mode – say, something like a --daemon or --headless CLI option that just disables the Qt GUI entirely and runs as a quiet background daemon process? If not, adding that would absolutely be the fastest way to resolve this issue.

Having a --headless CLI option is an excellent idea (I'd vote it up if it were feature request). I only really use the gui for initial setup / tweaking but after that I only rely on systray icon to confirm it's still running / I started it. That said, I am going to guess that depending on the way things were built back from original AntiMicro project that this may be an extremely challenging thing to do (I haven't looked at the codebase yet.. Got it to build on Fedora and was thinking about trying to study it but haven't had time)

noisecode3 commented 2 years ago

That is a good idea but require major rewrite I think because qt handle almost everything here(Okej I dont know everything just read little of AntiMicro code). When we do use big libs like this we get all the bugs from it. And its hard to fix and diagnose. The behavior of the crash CAN have a pattern when used on the same system for a while the change when we update libs and recompile. First we need to look for mistakes like not deleting objects or some abstraction of data that has gone wrong or whatever we can find. I can help next month. I think to go forward from here after that we need to inform some qt lib developers about this. let someone that have been or are involved with qt framework to take a look at what happen here. Then maybe is not for me to decide but move away a little bit from qt.

pktiuk commented 2 years ago

Hello guys
Have you checked flag --daemon?

  --daemon, -d               Launch program as a daemon. Use only on
                                      Linux.

Does it also crash?

leycec commented 2 years ago

Oh, boy... There's already an option for that. :sweat_smile:

I'll be firing up all future instances of antimicrox with that option enabled. I'm hip-deep into the first of a thousands-of-hours-long JRPG franchise that sprawls languorously across more than ten titles. (Yup, it's Falcom's Kiseki.)

Seriously hard data is incoming.

TheFeelTrain commented 2 years ago

That doesn't seem to work at all, it instantly crashes for me.

thefeeltrain@archlinux ~> antimicrox -d                                                                          (base) 
❗WARN  QCommandLineParser: option not defined: "next"
malloc_consolidate(): unaligned fastbin chunk detected
fish: Job 1, 'antimicrox -d' terminated by signal SIGABRT (Abort)

Edit: With gdb

(gdb) run
Starting program: /usr/bin/antimicrox -d
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7ffff2c79640 (LWP 1852776)]
[New Thread 0x7fffebf16640 (LWP 1852777)]
[New Thread 0x7fffdb175640 (LWP 1852778)]
❗WARN  QCommandLineParser: option not defined: "next"
[Detaching after fork from child process 1852779]
malloc_consolidate(): unaligned fastbin chunk detected
Couldn't read debug register: No such process.
(gdb) [Thread 0x7fffdb175640 (LWP 1852778) exited]
[Thread 0x7fffebf16640 (LWP 1852777) exited]
[Thread 0x7ffff2c79640 (LWP 1852776) exited]
[Inferior 1 (process 1852772) exited normally]
bt
No stack.

Seems like this malloc guy is always causing issues...

leycec commented 2 years ago

Superficially, antimicrox -d does successfully daemonize itself on my Gentoo machine:

$ antimicrox --daemon
❗WARN   QCommandLineParser: option not defined: "next"
❗WARN   QCommandLineParser: option not defined: "map"
❗WARN   QCommandLineParser: option not defined: "display"
$ pidof antimicrox
15229

Pragmatically, however, the daemonized antimicrox process doesn't appear to actually do anything. My DualShock 4 (DS4) remains blissfully unmapped – just as if I'd never run antimicrox at all.

It's back to the crash-prone Qt slave mines for me. Urgh!

pktiuk commented 2 years ago

@leycec
You can try to use flag --profile for changing flas used by app. (in case of any further questions please use discussion module)

leycec commented 2 years ago

Fascinating. I mostly have further concerns – like, antimicrox -d should probably either:

Default to the same profile that the antimicrox GUI defaults to or...
Print a fatal error if the --profile option isn't also passed.

In any case, thanks for the continual heads up! Let's see where this dark tunnel goes.

zpangwin commented 2 years ago

ok, just got a crash that I can provide a coredump for. Fedora 34 Cinnamon, while playing Satisfactory. Profile I was using is available here and I launched it via my DE's system menu.

In-game, the behavior when it crashed was that it jerked my view out of whack and started spinning me in circles (equivalent to if the mouse was stuck "moving" in one direction"). I don't recall for certain nor know if it is relevant, but I think the direction was counter-clockwise (e.g. moving to the left).

$ dnf list installed antimicrox
Installed Packages
antimicrox.x86_64                     3.2.0-1.fc34                      @updates

Based on notes here, I see I have some core dump files that were just generated

$ $ ls -acl /var/lib/systemd/coredump|grep 'Dec 28 17:'
drwxr-xr-x. 1 root root     970 Dec 28 17:21 .
-rw-r-----+ 1 root root 6762408 Dec 28 17:21 core.antimicrox.1000.ccfc9220a74e43408ff8d07322839de9.800972.1640730116000000.zst

And running gdb, I get the following (attached zst file but including inline for convenience):

$ cd /var/lib/systemd/coredump
$ coredumpctl gdb $(which antimicrox) ./core.antimicrox.1000.ccfc9220a74e43408ff8d07322839de9.800972.1640730116000000.zst
           PID: 800972 (antimicrox)
           UID: 1000 (fd)
           GID: 1000 (fd)
        Signal: 6 (ABRT)
     Timestamp: Tue 2021-12-28 17:21:56 EST (15min ago)
  Command Line: antimicrox --show
    Executable: /usr/bin/antimicrox
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-glib-io.github.antimicrox.antimicrox-800972.scope
          Unit: user@1000.service
     User Unit: app-glib-io.github.antimicrox.antimicrox-800972.scope
         Slice: user-1000.slice
     Owner UID: 1000 (fd)
       Boot ID: ccfc9220a74e43408ff8d07322839de9
    Machine ID: a501bc0e702b417f893bb79e6a211ca7
      Hostname: tinyfedora.local
       Storage: /var/lib/systemd/coredump/core.antimicrox.1000.ccfc9220a74e43408ff8d07322839de9.800972.1640730116000000.zst (present)
     Disk Size: 6.4M
       Message: Process 800972 (antimicrox) of user 1000 dumped core.

                Stack trace of thread 800991:
                #0  0x00007f9a722652a2 raise (libc.so.6 + 0x3d2a2)
                #1  0x00007f9a7224e8a4 abort (libc.so.6 + 0x268a4)
                #2  0x00007f9a722a7a97 __libc_message (libc.so.6 + 0x7fa97)
                #3  0x00007f9a722af70c malloc_printerr (libc.so.6 + 0x8770c)
                #4  0x00007f9a722b43fc malloc (libc.so.6 + 0x8c3fc)
                #5  0x00007f9a728746d1 _ZN10QArrayData8allocateEmmm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf46d1)
                #6  0x00007f9a728f1074 _ZN7QString11reallocDataEjb (libQt5Core.so.5 + 0x171074)
                #7  0x00007f9a728f170f _ZN7QString6appendEPK5QChari (libQt5Core.so.5 + 0x17170f)
                #8  0x00007f9a72aa7faf _ZN18QTextStreamPrivate9putNumberEyb (libQt5Core.so.5 + 0x327faf)
                #9  0x00007f9a72aa830b _ZN11QTextStreamlsEi (libQt5Core.so.5 + 0x32830b)
                #10 0x000055db6aeabb61 _ZN9JoyButton17getActiveZoneListEv (antimicrox + 0x169b61)
                #11 0x000055db6aeb9fd5 _ZN21JoyControlStickButton20getActiveZoneSummaryEv (antimicrox + 0x177fd5)
                #12 0x000055db6aea54dd _ZN9JoyButton28buildActiveZoneSummaryStringEv (antimicrox + 0x1634dd)
                #13 0x00007f9a72a5a3e9 _Z10doActivateILb0EEvP7QObjectiPPv (libQt5Core.so.5 + 0x2da3e9)
                #14 0x00007f9a72a5d68e _ZN6QTimer7timeoutENS_14QPrivateSignalE (libQt5Core.so.5 + 0x2dd68e)
                #15 0x00007f9a72a50edf _ZN7QObject5eventEP6QEvent (libQt5Core.so.5 + 0x2d0edf)
                #16 0x00007f9a73a0e443 _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt5Widgets.so.5 + 0x1ae443)
                #17 0x00007f9a72a267d8 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt5Core.so.5 + 0x2a67d8)
                #18 0x00007f9a72a76ea3 _ZN14QTimerInfoList14activateTimersEv (libQt5Core.so.5 + 0x2f6ea3)
                #19 0x00007f9a72a777ac _ZL19timerSourceDispatchP8_GSourcePFiPvES1_ (libQt5Core.so.5 + 0x2f77ac)
                #20 0x00007f9a714f54cf g_main_context_dispatch (libglib-2.0.so.0 + 0x554cf)
                #21 0x00007f9a715494f8 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa94f8)
                #22 0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #23 0x00007f9a72a77bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #24 0x00007f9a72a251e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #25 0x00007f9a728682ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #26 0x00007f9a728694c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #27 0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #28 0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800976:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a71df5f42 _xcb_conn_wait.part.0 (libxcb.so.1 + 0xdf42)
                #2  0x00007f9a71df78fc xcb_wait_for_event (libxcb.so.1 + 0xf8fc)
                #3  0x00007f9a61bb60d7 _ZN14QXcbEventQueue3runEv (libQt5XcbQpa.so.5 + 0x6e0d7)
                #4  0x00007f9a728694c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #5  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800972:
                #0  0x00007f9a722b0829 _int_free (libc.so.6 + 0x88829)
                #1  0x00007f9a722b390b _int_realloc (libc.so.6 + 0x8b90b)
                #2  0x00007f9a722b4cd7 realloc (libc.so.6 + 0x8ccd7)
                #3  0x00007f9a72874770 _ZN10QArrayData19reallocateUnalignedEPS_mm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf4770)
                #4  0x00007f9a728f110a _ZN7QString11reallocDataEjb (libQt5Core.so.5 + 0x17110a)
                #5  0x00007f9a728f183b _ZN7QString6appendE5QChar (libQt5Core.so.5 + 0x17183b)
                #6  0x00007f9a72aa7183 _ZN11QTextStreamlsEc (libQt5Core.so.5 + 0x327183)
                #7  0x000055db6aeafa8a _ZN9JoyButton13getActionNameEv (antimicrox + 0x16da8a)
                #8  0x000055db6aec02d6 _ZN31JoyControlStickButtonPushButton13generateLabelEv (antimicrox + 0x17e2d6)
                #9  0x000055db6ae29502 _ZN17FlashButtonWidget12refreshLabelEv (antimicrox + 0xe7502)
                #10 0x00007f9a72a50f49 _ZN7QObject5eventEP6QEvent (libQt5Core.so.5 + 0x2d0f49)
                #11 0x00007f9a73a0e443 _ZN19QApplicationPrivate13notify_helperEP7QObjectP6QEvent (libQt5Widgets.so.5 + 0x1ae443)
                #12 0x00007f9a72a267d8 _ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent (libQt5Core.so.5 + 0x2a67d8)
                #13 0x00007f9a72a29d46 _ZN23QCoreApplicationPrivate16sendPostedEventsEP7QObjectiP11QThreadData (libQt5Core.so.5 + 0x2a9d46)
                #14 0x00007f9a72a78117 _ZL23postEventSourceDispatchP8_GSourcePFiPvES1_ (libQt5Core.so.5 + 0x2f8117)
                #15 0x00007f9a714f54cf g_main_context_dispatch (libglib-2.0.so.0 + 0x554cf)
                #16 0x00007f9a715494f8 g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa94f8)
                #17 0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #18 0x00007f9a72a77bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #19 0x00007f9a72a251e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #20 0x00007f9a72a2d724 _ZN16QCoreApplication4execEv (libQt5Core.so.5 + 0x2ad724)
                #21 0x000055db6adaa8b8 main (antimicrox + 0x688b8)
                #22 0x00007f9a7224fb75 __libc_start_main (libc.so.6 + 0x27b75)
                #23 0x000055db6adb4bde _start (antimicrox + 0x72bde)

                Stack trace of thread 800978:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a7154948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f9a61caf3ed dconf_gdbus_worker_thread (libdconfsettings.so + 0x73ed)
                #4  0x00007f9a71523c42 g_thread_proxy (libglib-2.0.so.0 + 0x83c42)
                #5  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800977:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a7154948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f9a714f2c51 glib_worker_main (libglib-2.0.so.0 + 0x52c51)
                #4  0x00007f9a71523c42 g_thread_proxy (libglib-2.0.so.0 + 0x83c42)
                #5  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800980:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a7154948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f9a72a77bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f9a72a251e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f9a728682ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #6  0x00007f9a61a7bb6b _ZN22QDBusConnectionManager3runEv (libQt5DBus.so.5 + 0x1bb6b)
                #7  0x00007f9a728694c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #8  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #9  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800990:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a7154948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f9a72a77bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f9a72a251e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f9a728682ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #6  0x00007f9a728694c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #7  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #8  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800979:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a7154948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f9a714f4a93 g_main_loop_run (libglib-2.0.so.0 + 0x54a93)
                #3  0x00007f9a60350d9a gdbus_shared_thread_func.lto_priv.0 (libgio-2.0.so.0 + 0x110d9a)
                #4  0x00007f9a71523c42 g_thread_proxy (libglib-2.0.so.0 + 0x83c42)
                #5  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 800981:
                #0  0x00007f9a7231d5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f9a7154948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f9a714f2c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f9a72a77bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f9a72a251e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f9a728682ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #6  0x00007f9a728694c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #7  0x00007f9a73369299 start_thread (libpthread.so.0 + 0x9299)
                #8  0x00007f9a72328353 __clone (libc.so.6 + 0x100353)

GNU gdb (GDB) Fedora 11.1-5.fc34
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/antimicrox...
Reading symbols from .gnu_debugdata for /usr/bin/antimicrox...
(No debugging symbols found in .gnu_debugdata for /usr/bin/antimicrox)

warning: Can't open file /memfd:xorg (deleted) during file-backed mapping note processing
[New LWP 800991]
[New LWP 800976]
[New LWP 800972]
[New LWP 800978]
[New LWP 800977]
[New LWP 800980]
[New LWP 800990]
[New LWP 800979]
[New LWP 800981]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfos, use: dnf debuginfo-install antimicrox-3.2.0-1.fc34.x86_64
--Type <RET> for more, q to quit, c to continue without paging--
Core was generated by `antimicrox --show'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f9a722652a2 in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7f9a397fa640 (LWP 800991))]
(gdb) quit

zpangwin commented 2 years ago

antimicrox-coredump.tar.gz

zpangwin commented 2 years ago

For anyone looking for a temporary workaround until the crashes are resolved and assuming you don't mind launching from a terminal, you can create a bash script with the following which will launch to tray and continue to relaunch any time it crashes until you either hit Ctrl+C to cancel the script or right-click the AMX tray icon and choose exit.

#!/bin/bash

while true; do
    antimicrox --tray && exit
    sleep 1s;
done

You might get some minor 1-2s hiccups between crash and relaunch but other than that should be fairly low-effort and not require Alt-Tabbing out of the game to relaunch it.

To use, just create a text file named e.g. antimicrox-relauncher.sh, add the text, then set it ot executable e.g. chmod ug+x antimicrox-relauncher.sh. Then run from terminal, e.g. ./antimicrox-relauncher.sh

script explainer: on the line launching AMX, the && will only run the exit command if AMX terminates successfully (e.g. you tell it to close). Otherwise if you kill the process or it crashes it will have a non-zero exit code and the exitstatement will not run. In this case, it will sleep for 1 second, then relaunch AMX. It repeats this logic in an infinite loop (hence the need to Ctrl+C to kill the script when done.

noisecode3 commented 2 years ago

hey I think I found the problem, its string in the FlashButtonWidget::refreshLabe method also there is a depreciated method of Qt in FlashButtonWidget::paintEvent I changed fm.with() to fm.horizontalAdvance() . My solution is silly cus im just testing but I comment out the setText(generateLabel()); it results in buttons have no text in the ui. but think I'll find a work around for that later later. That should be the method that crashes. I will test 2 more day then I can fork and show my fix. If funny I have never fixed someone else's code only my own.

noisecode3 commented 2 years ago

I found there some read-write problems. This docs can help later and others fix this https://doc.qt.io/qt-5/qmutex.html I want to try fix this. It takes too much time to test this can you all help me test is this fork also crash for you? There is no text on the flashy buttons don't worry. Use you're profile like normal

https://github.com/noisecode3/antimicrox

pktiuk commented 2 years ago

It takes too much time to test this can you all help me test is this fork also crash for you?

You can create a release in your repo and then CI will build and upload packages automatically. (AppImage would be the best for this kind of testing)

BTW it you are dealing with code, online docs generated by doxygen may be useful: https://antimicrox.github.io/doxygen/

noisecode3 commented 2 years ago

Yes, I tried Travis CI, Im too broke to get the free plan... I make the appimage later tonight. It should be fine trying the normal build from my repo to see where it crashes next to make some kind of sens of it in the end. Please post log like @zpangwin

zpangwin commented 2 years ago

Disclaimer: before people go crazy and try to build / use the test branch as a replacement, keep in mind that it removes all the text on buttons and such so it's not really viable except for loading an existing profile. It's only intended for verifying a theory about what is causing the crashes.

I buillt the master branch from @noisecode3 on Fedora 34 Cinnamon in order to try to out the test build. I'm hesitant to say handling the function from the test "is the fix" for crashes when the ticket doesn't define a specific behavior that produces them (e.g. is the scope of the ticket for all runtime crashes? if so, is this the only issue?)...

But from my initial testing, this approach definitely seems very promising .

I had a fairly long session tonight using this build and no crashes (not from antimicrox anyway... the game I'm playing is still in Early Access tho). I generally get at least one crash from the normal build of AMX during a shorter session than this. I saved logs of build output; if anyone wants them let me know (I don't really think they're especially useful but you never know...). Thus far haven't had any segfaults / crashes... so no gdb coredumps at this point. I'll test it out some more over the next several days and post back if I'm able to capture anything.

~~Not sure how difficult it will be to adapt this without removing all the text in the app, but like I said seems promising.~~ Edit: it sounds like I was very mistaken about how an actual patch based on the concept from the test would be done.

here is how I did the test build for anyone who wants a native build on Fedora 34 (there is also an AppImage version now):

$ sudo -i
// just winging it here to get build depends on Fedora based on notes here:
//      https://github.com/AntiMicroX/antimicrox/blob/master/BUILDING.md
// likely not all of these are actually needed and this could be slimmed
// down slightly. i might also be leaving out stuff i already had installed.
// but i'm not bothering with perfect doco for that right now
# dnf groupinstall -y "Development Tools"
# dnf groupinstall -y "C Development Tools and Libraries"
# dnf install -y git make cmake gcc cmake extra-cmake-modules \
    qt5-qttools qt5-qttools-common qt5-qttools-devel qt5-qttools-static \
    SDL2 SDL2-static SDL2-devel libXi libXi-devel libXtst libXtst-devel \
    libX11 libX11-common libX11-devel itstool gettext gettext-devel
# exit
$ git clone https://github.com/noisecode3/antimicrox noisecode3_antimicrox
$ cd noisecode3_antimicrox/

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

$ git log -1
commit fe17acc935405ca9933f860584d1684ba9d21087 (HEAD -> master, origin/master, origin/HEAD)
Author: Martin Bångens <marbangens@gmail.com>
Date:   Thu Jan 6 15:20:22 2022

    read-write problem

$ mkdir build && cd build
$ cmake ..
$ cmake --build .
$ cd bin
$ ./antimicrox --log-level debug --tray

noisecode3 commented 2 years ago

Oh no, this is a test not fix. Don't think that. That not a problem we just have to make sure it locks that string before it reads from it. Generating that string is first priority then the Gui. There is timers and threads it makes it seem random. The problem is not random no. If you don't know how to build this you can try the AppImage. Test it and build it on different distros that's good.

Test some more days

zpangwin commented 2 years ago

Oh no, this is a test not fix. Don't think that.

My bad... was tired last night and phrased that very poorly. I probably should have said something like "the concept behind this test seems promising for finally fixing the crashes". ~~Will edit~~ Have edited my earlier post to hopefully avoid giving the wrong impression to anyone else dropping in.

That not a problem we just have to make sure it locks that string before it reads from it. Generating that string is first priority then the Gui.

Ah ok I think I had misunderstood before. Not familar with Qt and very rusty with C/C++ but it sounds at least vaguely similar to what happens in other languages when threaded code tries to access some object / value that is not thread-safe.

The problem is not random

Again, poor phrasing on my part. What I mean was that this ticket is written in such as way that there is no recreatable steps for generating the crash and that from a user's perspective, it occurs randomly during usage. So even if this test this leads to a fix, in theory at least, this ticket could still apply to other general runtime crashes that don't have well-defined steps / aren't easily reproducible.

If you don't know how to build this you can try the AppImage

Ah I either didn't see the AppImage or it wasn't up when I had looked. I mostly listed out my build steps so anyone else on Fedora knew how to resolve dependencies.

Anyway, very nice find with this approach and thanks for the test branch!

noisecode3 commented 2 years ago

Thank you for helping out. AppImage is now in the release link, I had it in the repo before but that was a bad idea. Test fest :)

zpangwin commented 2 years ago

Update: I've had 2 more fairly long sessions and 1 shorter session since my previous comment, with no crashes whatsoever. All of this on a native build from commit fe17acc935405ca9933f860584d1684ba9d21087.

I see there are some newer commits from yesterday but no new releases yet so I am assuming that you are already working on it and that, based on commit messages, you appear to have found some crashes that I did not run into. I'm trying to look through the changes... but thinking I need to go and spend some time learning Qt framework (and maybe refreshing myself on C++) first lol.

No rush from my end; whenever you think it's ready, just let me know when and which commit you'd like tested and I'll be happy to help out as I get time.

noisecode3 commented 2 years ago

Try this now, there is still problems and that's not the final fix, https://github.com/noisecode3/antimicrox/releases/download/3.2.3/AntiMicroX-x86_64.AppImage I need those crash bt logs from you :)

zpangwin commented 2 years ago

ok, I'll upgrade to that one. I had tried building off commit b422b3cda23ecb9f2ffca0931028332be7947f59 - "test without qDebug" yesterday (still no crashes on a 4-5 hr session) but I see there was another commit efab3ad852fb3d3432030238f784cc972e81ec48 - "more lock's for testing" later the same day... I'll check for whatever the latest commit is and rebuilt with that before i start (I generally prefer distro packages/building but I can try to test AppImage later too if you want). There's also one thing I want to try ruling out on the original version (I started using antimicrox --log-level debug --tray in my launcher script at some point along the way while my last crash report shows antimicrox --show so I just want to do a control to confirm that doesn't make a difference. Pretty sure I've had crashes with original version and --tray. I don't think --log-level debug should matter but figured it's better to play it safe).

Update 1:

Was able to confirm the issue still happens under my control test with the original code (antimicrox --log-level debug --tray) so I can rule out the difference between arguments used in my last crash report and what my script currently has and get back to testing the test branch.

Since I did have some debug statements in the terminal just prior to that crash that I did not capture previously, I have attached those as well. But since this is for the original code base (not the test build) and we already have some crash reports captured for that, I'm just going to add as attachments and not bother with any inline codeblocks. I also checked under ~/.config/antimicrox but I didn't have any log files there some captured output was just me saving the terminal output into a file (it's cut off at the top bc my terminal apparently didn't have a high enough scroll buffer).

Anyway, the last few lines of the debug output were:

[22:31:43.092] 🐞DEBUG   Throtted value at start of function is: -21848 Calculated value of throttle is: -21848 (file ir/build/BUILD/antimicrox-3.2.0/src/joyaxis.cpp:260)
[22:31:43.092] 🐞DEBUG   Throtted value at start of function is: -21848 Calculated value of throttle is: -21848 (file ir/build/BUILD/antimicrox-3.2.0/src/joyaxis.cpp:260)
[22:31:43.092] 🐞DEBUG   Raw value is less than 32767 and greather than -32767 (29792) (file ir/build/BUILD/antimicrox-3.2.0/src/joyaxis.cpp:561)
[22:31:43.092] 🐞DEBUG   Throtted value at start of function is: 29792 Calculated value of throttle is: 29792 (file ir/build/BUILD/antimicrox-3.2.0/src/joyaxis.cpp:260)
malloc(): unaligned tcache chunk detected
[22:31:43.092] 🐞DEBUG   Throtted value at start of function is: 29792 Calculated value of throttle is: 29792 (file ir/build/BUILD/antimicrox-3.2.0/src/joyaxis.cpp:260)
./antimicrox-relauncher.sh: line 64: 884397 Aborted                 (core dumped) antimicrox --log-level debug --tray

amx-w-debug-logging-crashdump.tar.gz antimicrox-terminal-output.txt

Update 2:

Started testing commit efab3ad852fb3d3432030238f784cc972e81ec48 - "more lock's for testing" (native build from source) for about 3-4 hrs last night but wasn't able to get any crashes. Will continue as I get time and see if I can capture anything.

Update 3:

Still testing commit efab3ad852fb3d3432030238f784cc972e81ec48 - "more lock's for testing". After another ~ 8 hrs, still not a single crash.

noisecode3 commented 2 years ago

It has be running sens friday morning for me. I have not played all the time only about 12 hours I think. I did occasionally stop what I was doing and try some input from time to time. No that dose not matter, use it like normal trying different flags is also good. I need new crash report from my test repo. tcache is Thread Local Cache. This seem to work I'll make proper clean fork with fix next week.

zpangwin commented 2 years ago

Ok, just when I was starting to think I couldn't get any crashes from test build, I finally got one on a build from commit efab3ad852fb3d3432030238f784cc972e81ec48 - "more lock's for testing".

# AMX_TEST_BUILD="/home/fd/Code/Forks/noisecode3_antimicrox/builds/2022-jan-13_efab3ad8/bin/antimicrox"
# coredumpctl gdb "$AMX_TEST_BUILD"
           PID: 1381102 (antimicrox)
           UID: 1000 (fd)
           GID: 1000 (fd)
        Signal: 11 (SEGV)
     Timestamp: Sun 2022-01-16 16:43:58 EST (16min ago)
  Command Line: /home/fd/Code/Forks/noisecode3_antimicrox/builds/2022-jan-13_efab3ad8/bin/antimicrox --log-level debug --tray
    Executable: /home/fd/Code/Forks/noisecode3_antimicrox/builds/2022-jan-13_efab3ad8/bin/antimicrox
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-39e9309c-02f8-42fd-b6a0-13b8d3ab7435.scope
          Unit: user@1000.service
     User Unit: vte-spawn-39e9309c-02f8-42fd-b6a0-13b8d3ab7435.scope
         Slice: user-1000.slice
     Owner UID: 1000 (fd)
       Boot ID: 3939191ea1424fc190a4eb0d3f6126ef
    Machine ID: a501bc0e702b417f893bb79e6a211ca7
      Hostname: <redacted>
       Storage: /var/lib/systemd/coredump/core.antimicrox.1000.3939191ea1424fc190a4eb0d3f6126ef.1381102.1642369438000000.zst (present)
     Disk Size: 5.0M
       Message: Process 1381102 (antimicrox) of user 1000 dumped core.

                Stack trace of thread 1381122:
                #0  0x0000000000452038 _ZL25termSignalSegfaultHandleri (antimicrox + 0x52038)
                #1  0x00007f6952903a20 __restore_rt (libpthread.so.0 + 0x13a20)
                #2  0x00007f69517f52a2 raise (libc.so.6 + 0x3d2a2)
                #3  0x00007f69517de8a4 abort (libc.so.6 + 0x268a4)
                #4  0x00007f6951837a97 __libc_message (libc.so.6 + 0x7fa97)
                #5  0x00007f695183f70c malloc_printerr (libc.so.6 + 0x8770c)
                #6  0x00007f69518443fc malloc (libc.so.6 + 0x8c3fc)
                #7  0x00007f6951e046d1 _ZN10QArrayData8allocateEmmm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf46d1)
                #8  0x00007f6951e80cf7 _ZN7QStringC1EiN2Qt14InitializationE (libQt5Core.so.5 + 0x170cf7)
                #9  0x00007f695200b501 _ZN5QUtf816convertToUnicodeEPKci (libQt5Core.so.5 + 0x2fb501)
                #10 0x00007f6951e85087 _ZN7QString15fromUtf8_helperEPKci (libQt5Core.so.5 + 0x175087)
                #11 0x00007f6951e850fc _ZN7QString16fromAscii_helperEPKci (libQt5Core.so.5 + 0x1750fc)
                #12 0x000000000044d4ee _ZN7QStringC1EPKc (antimicrox + 0x4d4ee)
                #13 0x000000000045211b _ZL25termSignalSegfaultHandleri (antimicrox + 0x5211b)
                #14 0x00007f6952903a20 __restore_rt (libpthread.so.0 + 0x13a20)
                #15 0x00007f69517f52a2 raise (libc.so.6 + 0x3d2a2)
                #16 0x00007f69517de8a4 abort (libc.so.6 + 0x268a4)
                #17 0x00007f6951837a97 __libc_message (libc.so.6 + 0x7fa97)
                #18 0x00007f695183f70c malloc_printerr (libc.so.6 + 0x8770c)
                #19 0x00007f69518443fc malloc (libc.so.6 + 0x8c3fc)
                #20 0x00007f6951e046d1 _ZN10QArrayData8allocateEmmm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf46d1)
                #21 0x00007f6951e80cf7 _ZN7QStringC1EiN2Qt14InitializationE (libQt5Core.so.5 + 0x170cf7)
                #22 0x00007f695200b501 _ZN5QUtf816convertToUnicodeEPKci (libQt5Core.so.5 + 0x2fb501)
                #23 0x00007f6951e85087 _ZN7QString15fromUtf8_helperEPKci (libQt5Core.so.5 + 0x175087)
                #24 0x00007f6951e850fc _ZN7QString16fromAscii_helperEPKci (libQt5Core.so.5 + 0x1750fc)
                #25 0x000000000044d4ee _ZN7QStringC1EPKc (antimicrox + 0x4d4ee)
                #26 0x000000000045211b _ZL25termSignalSegfaultHandleri (antimicrox + 0x5211b)
                #27 0x00007f6952903a20 __restore_rt (libpthread.so.0 + 0x13a20)
                #28 0x00007f69517f52a2 raise (libc.so.6 + 0x3d2a2)
                #29 0x00007f69517de8a4 abort (libc.so.6 + 0x268a4)
                #30 0x00007f6951837a97 __libc_message (libc.so.6 + 0x7fa97)
                #31 0x00007f695183f70c malloc_printerr (libc.so.6 + 0x8770c)
                #32 0x00007f69518443fc malloc (libc.so.6 + 0x8c3fc)
                #33 0x00007f6951e046d1 _ZN10QArrayData8allocateEmmm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf46d1)
                #34 0x00007f6951e80cf7 _ZN7QStringC1EiN2Qt14InitializationE (libQt5Core.so.5 + 0x170cf7)
                #35 0x00007f695200b501 _ZN5QUtf816convertToUnicodeEPKci (libQt5Core.so.5 + 0x2fb501)
                #36 0x00007f6951e85087 _ZN7QString15fromUtf8_helperEPKci (libQt5Core.so.5 + 0x175087)
                #37 0x00007f6951e850fc _ZN7QString16fromAscii_helperEPKci (libQt5Core.so.5 + 0x1750fc)
                #38 0x000000000044d4ee _ZN7QStringC1EPKc (antimicrox + 0x4d4ee)
                #39 0x000000000045211b _ZL25termSignalSegfaultHandleri (antimicrox + 0x5211b)
                #40 0x00007f6952903a20 __restore_rt (libpthread.so.0 + 0x13a20)
                #41 0x00007f69517f52a2 raise (libc.so.6 + 0x3d2a2)
                #42 0x00007f69517de8a4 abort (libc.so.6 + 0x268a4)
                #43 0x00007f6951837a97 __libc_message (libc.so.6 + 0x7fa97)
                #44 0x00007f695183f70c malloc_printerr (libc.so.6 + 0x8770c)
                #45 0x00007f69518443fc malloc (libc.so.6 + 0x8c3fc)
                #46 0x00007f6951e046d1 _ZN10QArrayData8allocateEmmm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf46d1)
                #47 0x00007f6951e80cf7 _ZN7QStringC1EiN2Qt14InitializationE (libQt5Core.so.5 + 0x170cf7)
                #48 0x00007f695200b501 _ZN5QUtf816convertToUnicodeEPKci (libQt5Core.so.5 + 0x2fb501)
                #49 0x00007f6951e85087 _ZN7QString15fromUtf8_helperEPKci (libQt5Core.so.5 + 0x175087)
                #50 0x00007f6951e850fc _ZN7QString16fromAscii_helperEPKci (libQt5Core.so.5 + 0x1750fc)
                #51 0x000000000044d4ee _ZN7QStringC1EPKc (antimicrox + 0x4d4ee)
                #52 0x000000000045211b _ZL25termSignalSegfaultHandleri (antimicrox + 0x5211b)
                #53 0x00007f6952903a20 __restore_rt (libpthread.so.0 + 0x13a20)
                #54 0x00007f69517f52a2 raise (libc.so.6 + 0x3d2a2)
                #55 0x00007f69517de8a4 abort (libc.so.6 + 0x268a4)
                #56 0x00007f6951837a97 __libc_message (libc.so.6 + 0x7fa97)
                #57 0x00007f695183f70c malloc_printerr (libc.so.6 + 0x8770c)
                #58 0x00007f69518443fc malloc (libc.so.6 + 0x8c3fc)
                #59 0x00007f6951e046d1 _ZN10QArrayData8allocateEmmm6QFlagsINS_16AllocationOptionEE (libQt5Core.so.5 + 0xf46d1)
                #60 0x00007f6951e80cf7 _ZN7QStringC1EiN2Qt14InitializationE (libQt5Core.so.5 + 0x170cf7)
                #61 0x00007f695200b501 _ZN5QUtf816convertToUnicodeEPKci (libQt5Core.so.5 + 0x2fb501)
                #62 0x00007f6951e85087 _ZN7QString15fromUtf8_helperEPKci (libQt5Core.so.5 + 0x175087)
                #63 0x00007f6951e850fc _ZN7QString16fromAscii_helperEPKci (libQt5Core.so.5 + 0x1750fc)

                Stack trace of thread 1381105:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a82c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f694123f3ed dconf_gdbus_worker_thread (libdconfsettings.so + 0x73ed)
                #4  0x00007f6950ab3c42 g_thread_proxy (libglib-2.0.so.0 + 0x83c42)
                #5  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381103:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6951385f42 _xcb_conn_wait.part.0 (libxcb.so.1 + 0xdf42)
                #2  0x00007f69513878fc xcb_wait_for_event (libxcb.so.1 + 0xf8fc)
                #3  0x00007f69411460d7 _ZN14QXcbEventQueue3runEv (libQt5XcbQpa.so.5 + 0x6e0d7)
                #4  0x00007f6951df94c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #5  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381104:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a82c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f6950a82c51 glib_worker_main (libglib-2.0.so.0 + 0x52c51)
                #4  0x00007f6950ab3c42 g_thread_proxy (libglib-2.0.so.0 + 0x83c42)
                #5  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381106:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a84a93 g_main_loop_run (libglib-2.0.so.0 + 0x54a93)
                #3  0x00007f693b708d9a gdbus_shared_thread_func.lto_priv.0 (libgio-2.0.so.0 + 0x110d9a)
                #4  0x00007f6950ab3c42 g_thread_proxy (libglib-2.0.so.0 + 0x83c42)
                #5  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #6  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381121:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a82c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f6952007bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f6951fb51e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f6951df82ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #6  0x00007f6951df94c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #7  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #8  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381108:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a82c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f6952007bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f6951fb51e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f6951df82ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #6  0x00007f6951df94c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #7  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #8  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381107:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a82c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f6952007bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f6951fb51e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f6951df82ca _ZN7QThread4execEv (libQt5Core.so.5 + 0xe82ca)
                #6  0x00007f694100bb6b _ZN22QDBusConnectionManager3runEv (libQt5DBus.so.5 + 0x1bb6b)
                #7  0x00007f6951df94c6 _ZN14QThreadPrivate5startEPv (libQt5Core.so.5 + 0xe94c6)
                #8  0x00007f69528f9299 start_thread (libpthread.so.0 + 0x9299)
                #9  0x00007f69518b8353 __clone (libc.so.6 + 0x100353)

                Stack trace of thread 1381102:
                #0  0x00007f69518ad5bf __poll (libc.so.6 + 0xf55bf)
                #1  0x00007f6950ad948c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa948c)
                #2  0x00007f6950a82c03 g_main_context_iteration (libglib-2.0.so.0 + 0x52c03)
                #3  0x00007f6952007bb8 _ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2f7bb8)
                #4  0x00007f6951fb51e2 _ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE (libQt5Core.so.5 + 0x2a51e2)
                #5  0x00007f6951fbd724 _ZN16QCoreApplication4execEv (libQt5Core.so.5 + 0x2ad724)
                #6  0x0000000000457383 main (antimicrox + 0x57383)
                #7  0x00007f69517dfb75 __libc_start_main (libc.so.6 + 0x27b75)
                #8  0x000000000043528e _start (antimicrox + 0x3528e)

GNU gdb (GDB) Fedora 11.1-5.fc34
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/fd/Code/Forks/noisecode3_antimicrox/builds/2022-jan-13_efab3ad8/bin/antimicrox...
(No debugging symbols found in /home/fd/Code/Forks/noisecode3_antimicrox/builds/2022-jan-13_efab3ad8/bin/antimicrox)
[New LWP 1381122]
[New LWP 1381105]
[New LWP 1381103]
[New LWP 1381104]
[New LWP 1381106]
[New LWP 1381121]
[New LWP 1381108]
[New LWP 1381107]
[New LWP 1381102]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/home/fd/Code/Forks/noisecode3_antimicrox/builds/2022-jan-13_ef'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000452038 in termSignalSegfaultHandler(int) ()
[Current thread is 1 (Thread 0x7f691d9ad640 (LWP 1381122))]
Missing separate debuginfos, use: dnf debuginfo-install SDL2-2.0.20-1.fc34.x86_64 adwaita-qt5-1.4.1-2.fc34.x86_64 at-spi2-atk-2.38.0-2.--Type <RET> for more, q to quit, c to continue without paging--
fc34.x86_64 at-spi2-core-2.40.3-1.fc34.x86_64 atk-2.36.0-3.fc34.x86_64 bzip2-libs-1.0.8-6.fc34.x86_64 cairo-1.17.4-3.fc34.x86_64 cairo-gobject-1.17.4-3.fc34.x86_64 dbus-libs-1.12.20-3.fc34.x86_64 dconf-0.40.0-3.fc34.x86_64 fontconfig-2.13.94-2.fc34.x86_64 freetype-2.10.4-3.fc34.x86_64 fribidi-1.0.10-4.fc34.x86_64 gdk-pixbuf2-2.42.6-1.fc34.x86_64 glib2-2.68.4-1.fc34.x86_64 glibc-2.33-20.fc34.x86_64 graphite2-1.3.14-7.fc34.x86_64 gtk3-3.24.30-1.fc34.x86_64 harfbuzz-2.7.4-3.fc34.x86_64 jasper-libs-2.0.33-1.fc34.x86_64 jbigkit-libs-2.1-21.fc34.x86_64 keyutils-libs-1.6.1-2.fc34.x86_64 kf5-karchive-5.85.0-1.fc34.x86_64 kf5-kimageformats-5.85.0-1.fc34.x86_64 krb5-libs-1.19.2-2.fc34.x86_64 lcms2-2.12-1.fc34.x86_64 libICE-1.0.10-6.fc34.x86_64 libSM-1.2.3-8.fc34.x86_64 libX11-1.7.2-3.fc34.x86_64 libX11-xcb-1.7.2-3.fc34.x86_64 libXau-1.0.9-6.fc34.x86_64 libXcursor-1.2.0-5.fc34.x86_64 libXdamage-1.1.5-5.fc34.x86_64 libXext-1.3.4-6.fc34.x86_64 libXfixes-6.0.0-1.fc34.x86_64 libXi-1.7.10-6.fc34.x86_64 libXrender-0.9.10-14.fc34.x86_64 libXtst-1.2.3-14.fc34.x86_64 libadwaita-qt5-1.4.1-2.fc34.x86_64 libaom-3.1.1-2.fc34.x86_64 libavif-0.9.0-2.fc34.x86_64 libbrotli-1.0.9-4.fc34.x86_64 libcap-2.48-2.fc34.x86_64 libcloudproviders-0.3.1-3.fc34.x86_64 libcom_err-1.45.6-5.fc34.x86_64 libdatrie-0.2.13-1.fc34.x86_64 libdav1d-0.9.2-1.fc34.x86_64 libepoxy-1.5.9-1.fc34.x86_64 libffi-3.1-28.fc34.x86_64 libgcc-11.2.1-1.fc34.x86_64 libgcrypt-1.9.3-3.fc34.x86_64 libglvnd-1.3.3-1.fc34.x86_64 libglvnd-glx-1.3.3-1.fc34.x86_64 libgpg-error-1.42-1.fc34.x86_64 libicu-67.1-7.fc34.x86_64 libjpeg-turbo-2.0.90-3.fc34.x86_64 libjxl-0.6.1-6.fc34.x86_64 libmount-2.36.2-1.fc34.x86_64 libpng-1.6.37-10.fc34.x86_64 libstdc++-11.2.1-1.fc34.x86_64 libtiff-4.2.0-1.fc34.x86_64 libuuid-2.36.2-1.fc34.x86_64 libwayland-client-1.19.0-1.fc34.x86_64 libwebp-1.2.1-1.fc34.x86_64 libxcb-1.13.1-7.fc34.x86_64 libxkbfile-1.1.0-6.fc34.x86_64 libxklavier-5.4-18.fc34.x86_64 libzstd-1.5.0-1.fc34.x86_64 lz4-libs-1.9.3-2.fc34.x86_64 openexr-libs-2.5.5-1.fc34.x86_64 openssl-libs-1.1.1l-2.fc34.x86_64 pango-1.48.10-2.fc34.x86_64 pcre-8.44-3.fc34.1.x86_64 qgnomeplatform-0.8.3-2.fc34.x86_64 qt5-qtbase-5.15.2-30.fc34.x86_64 qt5-qtbase-gui-5.15.2-30.fc34.x86_64 qt5-qtimageformats-5.15.2-3.fc34.x86_64 qt5-qtwebengine-5.15.5-1.fc34.x86_64 qt5-qtx11extras-5.15.2-3.fc34.x86_64 sssd-client-2.5.2-2.fc34.x86_64 systemd-libs-248.9-1.fc34.x86_64 xcb-util-keysyms-0.4.0-15.fc34.x86_64 xz-libs-5.2.5-5.fc34.x86_64 zlib-1.2.11-26.fc34.x86_64
(gdb) quit

coredump-text.txt

terminal-output.txt

antimicrox-relauncher.sh.txt

profile used - mostly I was just using the first control set and IIRC only thumbsticks (mapped to WASD and mouse respectively) and a couple buttons mapped to "E" / left mouse button around the time of the crash.

Hope that's helpful. If you need any add'l let me know. I'll keep using this build until I hear otherwise or see some a new commit and if I capture any more, I'll attach them.

noisecode3 commented 2 years ago

Its basally the same problem but with a bunch of other problems in the Gui. But for me it runs without crashing now. There is potential memory leaks and null pointer problems as well as it reads libSDL2 variables on another thread. That will take time. I don't think I will fix all of that by myself anytime soon..

zpangwin commented 2 years ago

I was thinking about this a little more today and started to wonder about if we possibly could approach the problem from another direction... but I'm just not sure if I understand everything correctly.

From what we've been going over in this thead so far, my understanding/assumptions of the current state of things is (and I apologize if I am grossly over-simplifying this / have things very wrong in my head):

While the issue could theoretically occur at any time after the app is started, in practice, it is more prevalent (or at least more frustrating to users) when the crash happens while in-game as opposed to happening while creating a profile.
The root cause of the crashes is related to setting text elements in Qt / how some Qt methods handle what appears to be code that is not thread-safe while those text elements are set... In other words, the crashes do not occur when we comment out / don't invoke sections related to text labels like in the earlier proof-of-concept code we tested (the old Jan 6 commit fe17acc935405ca9933f860584d1684ba9d21087 - read-write problem ).
Fixing every place with mem leaks / null pointer / threading / etc issues that can possibly cause crashes is a very large task and unlikely to be completed any time soon.

But assuming, I'm not entirely wrong here, then what do you think about a stopgap measure to reduce the scope of the issue and make point #1 less of a problem while fixes from point #3 are being worked? What I had in mind really hinges on point #2 about the crashes only happening when we deal with labels, but if so, could we add a conditional to refreshLabel()so that the code where the problems live never even gets invoked in cases where we don't really even need to display anything on the screen (e.g. app is closed to system tray / minimized / not in focus)? That would essentially avoid the problem in most cases similarly to the old test build that just commented things out but still allow new fixes like the Mutex/thread-safety stuff to be tested and the button texts to be displayed when the app is in-focus (such as people creating a new profile) e.g. very roughly something like:

flashbuttonwidget.cpp

void FlashButtonWidget::refreshLabel()
{
    // AFAIK these are just made-up example functions;
    // they'd need to be created or replaced by other Qt API calls
    if(appInSysTray() || !appHasFocus()) {
        return;
    }
    setText(generateLabel());
    qDebug() << "label has been set: " << generateLabel();
}

Again, only as a stopgap to reduce the scope of the problem / in addition to other fixes. Thoughts?

pktiuk commented 2 years ago

@zpangwin

I think this is a very good idea worth investigation.

noisecode3 commented 2 years ago

Again, only as a stopgap to reduce the scope of the problem / in addition to other fixes. Thoughts?

The problem is called data race. Take look at this https://www.youtube.com/watch?v=5sZo3SrLrGA

pktiuk commented 2 years ago

I have no Idea why refreshLabel() is called every time button is pressed or released. (It doesn't make much sense to me)

I disabled it and this change doesn't seem to affect the application at all.

@zpangwin
Would you like to test my build from here (again)?

zpangwin commented 2 years ago

I have no Idea why refreshLabel() is called every time button is pressed or released. (It doesn't make much sense to me)

Ah the joys of inherited codebases lol. I once had a project where the guy that handed it off to me had written it in VB.NET but had lost the original source and gave me decompiled sources.... and I had to migrate it to java and add concurrency. Fun stuff.

@zpangwin Would you like to test my build from here (again)?

Sure.

I had been playing around with Qt's isActiveWindow and isVisible but had much worse luck with the labeling part in my own test branch. Partly in mine, some of the labels didn't always get set correctly and partly bc's Qt's implementation of isVisible reports a window that is covered by another window as still being visible as long as it isn't actually minimized.

Anyway, just rebuilt from your branch (commit 5c52abfe) and did a bit of testing for making sure labels in the GUI are set correctly, that they get updated if you change mappings, switch virtual desktops, etc. Unlike mine, yours had no issue with the labels. Will take me awhile to do more thorough testing to see if I can generate crashes or not though.

Side note: while getting my builds going, I realized I was not getting any output for qDebug() statements on the terminal and eventually found creating a specific config file resolved the issue. Is that something I could create a ticket / PR for adding a note for in BUILDING.md (diff here)? I think this only applies when it is being built and the user doesn't have QtCreator installed so probably doesn't impact the release binaries, but still nice to have a note.

noisecode3 commented 2 years ago

hey, I have to say something when I see this. I had this problem with a plugin for music. I didn't think there was a problem reading some GUI value directly from my audio thread... What I got was this random glitchy behavior then a crash :) You will have to eventually sync threads at some point and there is also atomic. Its definitely better to not generate label all the time, but I think without synchronization, it will bury the bug deeper in a specific CPU :) You can take advantages of this and target a specific CPU. Don't think AntiMicroX need or should do that :)

pktiuk commented 2 years ago

@zpangwin

Is that something I could create a ticket / PR for adding a note for in BUILDING.md (diff here)? I think this only applies when it is being built and the user doesn't have QtCreator installed so probably doesn't impact the release binaries, but still nice to have a note.

Yes, it is definitely worth a PR. It may save a lot of headache for someone trying to debug it in the future

zpangwin commented 2 years ago

I've been testing a build from commit 5c52abfe - Disable updating button labels on press on and off for the last week (estimating I've had easily 20+ hrs in-game with it running - if not closer to 30) and have not gotten a single crash with it so far.

pktiuk commented 2 years ago

We can find this issue fixed (I will prepare the next release with this fix soon).
Thanks to @noisecode3 and @zpangwin. :bow:

pktiuk commented 2 years ago

FYI
I just published new release with this fix. You can download new binaries from here: https://github.com/AntiMicroX/antimicrox/releases/tag/3.2.2

AntiMicroX / antimicrox