Crashing on Mac Ventura 13.0 frequently

[greenfishgit22]

Hi,

First of all a huge thank you for this excellent application. Been an rn for more than 2 years now. Finally broading my self into ECG interpretation. I find your application to be more than excellent for this purpose.

I've noticed on Ventura 13.0 the application is crashing every 2-3 minute in usage.

Here's a debug log: https://pastebin.pl/view/c88d2012

I'm just a user, if you need any additional information, please let me know.

Thank you again.

[tanjera]

Thank you so much for the bug report and debug log! This helps a ton!

I definitely noticed stability issues on different OS/platforms I test it on but I've had a hard time pinning down and recreating most of the issues. But to hear that it's crashing every 2-3 minutes is horrendous and something I'd love to fix- I also didn't realize there was a new major MacOS version (I think my only Mac laptop is maxed out w/ Monterey/12) so I'll need to look into that as being a factor that needs supporting.

Going to read through the debug log, check out some of the dependencies to see if they have updated versions, and I'll post an update here if/when I find something.

Also- so glad II is helping you with ECG interpretation! 😄 And if you get into any of the tough cardiology concepts (specific diseases and their morphology, axis alterations, etc.) and need some more good reading, I can point you to some of the resources I used when modeling the rhythms and waveforms.

[tanjera]

Pertinent portions of the debug log, notes for my future self:

Process:               Infirmary Integrated [8619]
Identifier:            com.infirmary-integrated
Version:               2.4.5 (2.4.5)

... most up-to-date version ...

"captureTime" : "2023-01-27 13:19:25.4028 +0100",
"procLaunch" : "2023-01-27 13:09:27.0393 +0100",

... crashed @ ~10 min of runtime 😞

Crashed Thread:        75  .NET ThreadPool Worker

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Application Specific Information:
abort() called

Thread 75 Crashed:: .NET ThreadPool Worker
0   ???                                 0x7ff8923da9a8 ???
1   libsystem_kernel.dylib              0x7ff80311b30e __pthread_kill + 10
2   libsystem_pthread.dylib             0x7ff803152f7b pthread_kill + 263
3   libsystem_c.dylib                   0x7ff80309cca5 abort + 123
4   libcoreclr.dylib                       0x10928d6fb PROCAbort + 43
5   libcoreclr.dylib                       0x10928d62e PROCEndProcess(void*, unsigned int, int) + 318
6   libcoreclr.dylib                       0x1094da557 UnwindManagedExceptionPass1(PAL_SEHException&, _CONTEXT*) + 1111
7   libcoreclr.dylib                       0x1094da5a3 DispatchManagedException(PAL_SEHException&, bool) + 67
8   libcoreclr.dylib                       0x10943673a IL_Throw(Object*) + 554
9   ???                                    0x111c87b28 ???
10  ???                                    0x10f94ce55 ???
11  ???                                    0x112fd0380 ???
12  ???                                    0x112fd0131 ???
13  libcoreclr.dylib                       0x10957c5e9 CallDescrWorkerInternal + 124
14  libcoreclr.dylib                       0x1093c837f DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 223
15  libcoreclr.dylib                       0x1093df4f8 ThreadNative::KickOffThread_Worker(void*) + 136
16  libcoreclr.dylib                       0x109390898 ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 296
17  libcoreclr.dylib                       0x109390e00 ManagedThreadBase::KickOff(void (*)(void*), void*) + 32
18  libcoreclr.dylib                       0x1093df5ca ThreadNative::KickOffThread(void*) + 170
19  libcoreclr.dylib                       0x109292507 CorUnix::CPalThread::ThreadEntry(void*) + 407
20  libsystem_pthread.dylib             0x7ff803153259 _pthread_start + 125
21  libsystem_pthread.dylib             0x7ff80314ec7b thread_start + 15

... abort called from ??? ... ??? was called by .net CLR, at least 1 function called by CLR would be Avalonia as window manager... other threads have ~4-20 internal/??? calls in the stack trace... small # of function calls indicates either the abort was called by Avalonia for a window management exception, or the base function of an II (Infirmary Integrated) window or control... from memory, I don't remember implementing any exceptions being handled in window or control constructors... leading me to believe the exception is from the window management functionality. Before tossing blame on Avalonia, will read through the rest of the log.

MALLOC 340.2M 71

Memory use consistent with stability tests on Windows 10, Linux (kernel ?), Mac 12... no obvious memory leaks/misuse.

II 2.4.5-1 was compiled using Avalonia 0.10.7 ... very out of date ... need to try updating upstream packages, recompiling, then testing to recreate problem.

[tanjera]

Okay @greenfishgit22, I think I have an update ready for you. I've fixed what I think is the problem and compiled a new version. I updated the libraries that Infirmary uses to run on different platforms, and there were definitely updates that should improve stability on all platforms including MacOS. I tested the new build on my Windows, Linux, and Mac machines, but I only have MacOS Monterey, so I'm not sure if you'll have different results on Ventura.

The new update for you is at https://github.com/tanjera/infirmary-integrated/releases/download/v2.4/infirmary-integrated-2.4.6-macos.tar.gz

It'll probably be another day before I get the link up and running on the website, so I just wanted to share it here first. If you can just use that version and update me if it's fixed or if it still remains problematic, that would help a ton!

Thank you so much!

[greenfishgit22]

Thank you so much for the bug report and debug log! This helps a ton!

I definitely noticed stability issues on different OS/platforms I test it on but I've had a hard time pinning down and recreating most of the issues. But to hear that it's crashing every 2-3 minutes is horrendous and something I'd love to fix- I also didn't realize there was a new major MacOS version (I think my only Mac laptop is maxed out w/ Monterey/12) so I'll need to look into that as being a factor that needs supporting.

Going to read through the debug log, check out some of the dependencies to see if they have updated versions, and I'll post an update here if/when I find something.

Also- so glad II is helping you with ECG interpretation! 😄 And if you get into any of the tough cardiology concepts (specific diseases and their morphology, axis alterations, etc.) and need some more good reading, I can point you to some of the resources I used when modeling the rhythms and waveforms.

Okay @greenfishgit22, I think I have an update ready for you. I've fixed what I think is the problem and compiled a new version. I updated the libraries that Infirmary uses to run on different platforms, and there were definitely updates that should improve stability on all platforms including MacOS. I tested the new build on my Windows, Linux, and Mac machines, but I only have MacOS Monterey, so I'm not sure if you'll have different results on Ventura.

The new update for you is at https://github.com/tanjera/infirmary-integrated/releases/download/v2.4/infirmary-integrated-2.4.6-macos.tar.gz

It'll probably be another day before I get the link up and running on the website, so I just wanted to share it here first. If you can just use that version and update me if it's fixed or if it still remains problematic, that would help a ton!

Thank you so much!

Hello sir,

Very impressed by your prompt response and more importantly you managed to compile a new version so fast for me to try!

I've noticed the crashes happens more frequently if i'm up and running 12-ECG in the background and in the same time reading a pdf/website browsing. The difference compared to the last time is that when the application crashes there's no log, because I have to "force-quit" the non responding application. Since i'm not a developer like yourself i'm not familar with all the terms used in software language. I think you guys call this a hard crash.

• Crashes if i'm doing background tasks like putting the application in the background while reading a pdf
• More prone to crash in ECG-12 with color-scheme- grid
• Have to force quit the application

Update: I can see the ECG trace in grid mode, the full QRS complex then the whole app freezes.

Update; holy jesus look at memory usage!!! >30GB <----------

https://postimg.cc/DSTzKhVz

I'm using macbook air m1 2020 model with ventura 13.0

Thank You :)

[tanjera]

I actually think this is progress... there may have been multiple problems causing crashes, and that may have fixed one and exposed another. On the bright side, your information that you provided (specifically the exorbitant memory usage) and that the program creeps to a halt are both signs of a memory leak. I'm just thankful you were able to pin it down to the 12-lead ECG.

I'll have to do a bit of profiling and debugging and find and fix the memory leak. It'll probably take about a week with my current schedule to complete all that. In the meantime, a workaround worth trying would be to pause the 12 lead ECG. It's probably accruing memory usage while the tracings "draw" off-screen- using the "Pause Device" option should maybe stop that. Either way, it definitely needs fixing.

Thank you so much for the update!

[tanjera]

Troubleshooting profiling notes:

~ 18 minute memory profiling with 12 lead ECG and cardiac monitor open; Visual Studio (Windows)

• Consistent memory usage over time when no changes implemented
• Significant (~60 MB) jump in usage applying grid to ECG background @ ~1080p
  • Massive (~500 MB) jump in usage applying grid to ECG background @ ~4K
• Minor jump in usage (~10 MB) when rhythm changes applied (on clear and re-draw of future buffer?)
• Consistent memory usage in general when running physiology model and drawing routines, no progressive increase
• Major changes (~ 500 MB) with window manager (Avalonia) usage including resizing window, maximizing window, etc.
  • Window size/resolution appears to directly influence memory usage, sizing ~1080 to ~4K alloc'd ~500 MB, then ~4K to ~1080 dealloc'd same memory amount
  • Window size -> memory usage: not experienced with cardiac monitor
  • Window size -> memory usage: only experienced if grid is applied to background (why is grid using so much memory??)
• Garbage collector generally working as expected, no obvious memory leaks or failure to dealloc.

Next steps:

• Profile (and debug?) why showing the grid utilizes so much memory?
• Profile on MacOS (profiling utility?? dotnet??)

[greenfishgit22]

I actually think this is progress... there may have been multiple problems causing crashes, and that may have fixed one and exposed another. On the bright side, your information that you provided (specifically the exorbitant memory usage) and that the program creeps to a halt are both signs of a memory leak. I'm just thankful you were able to pin it down to the 12-lead ECG.

I'll have to do a bit of profiling and debugging and find and fix the memory leak. It'll probably take about a week with my current schedule to complete all that. In the meantime, a workaround worth trying would be to pause the 12 lead ECG. It's probably accruing memory usage while the tracings "draw" off-screen- using the "Pause Device" option should maybe stop that. Either way, it definitely needs fixing.

Thank you so much for the update!

1. htop-infirmary running (no problem)

2. htop-infirmary running with Cardiac Monitor (no problem)

3. htop-infirmary running with 12 Lead ECG (memory starts to increase, especially if I start to minize your application when i'm reading a pdf or browsing a webpage.

Take your time. It's no rush for me. Like you said if I pause the ECG it won't crash. I've attached 3 screenshots from htop running your application.

Please let me know if you need any additional information from my side. Thanks again sir!

Regards Oscar

[tanjera]

Recreated the crash experience that @greenfishgit22 was having, but it does present slightly differently from before. Might be because I am on macOS 12.

Appears to be a memory overrun in SkiaSharp, a drawing library used by Avalonia for geometric drawings, which I indirectly use for creating the ECG tracings. The problem is definitely easy to recreate, and I notice significant performance issues when Infirmary is in the background (like @greenfishgit22 says, when tabbing to a different window to read, etc.)... leads me to think it may be the Avalonia window management functions with a bug versus an actual problem with SkiaSharp.

Either way... it appears the problem is in the foundational libraries Infirmary is built on... so it's not a bug I can fix. Since I can't fix it, I'll think on ways I can mitigate the problem, maybe by refreshing controls or pausing tracings when in the background... but I don't know if it will be successful. Worth a try to attempt to circumvent the bug when I'm next able to.

For now, will keep this issue open and tag it appropriately, but I won't hold up the development or release cycle because of it- I had been holding off on the next release in the hope I could fix this bug... but it will have to wait :(

Process:               Infirmary Integrated [899]
Path:                  /Volumes/VOLUME/*/Infirmary Integrated.app/Contents/MacOS/Infirmary Integrated
Identifier:            com.infirmary-integrated
Version:               2.4.6 (2.4.6)
Code Type:             X86-64 (Native)
Parent Process:        launchd [1]
User ID:               501

Date/Time:             2023-02-06 12:22:16.4056 -0500
OS Version:            macOS 12.6.3 (21G419)
Report Version:        12
Anonymous UUID:        A03E51A1-8F4D-63AF-D5B1-DB9AC6A8F6A3

Time Awake Since Boot: 130 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       UNKNOWN_0xD at 0x0000000000000000
Exception Codes:       0x000000000000000d, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [899]

VM Region Info: 0 is not in any region.  Bytes before following region: 4555083776
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      10f811000-10f821000    [   64K] r-x/r-x SM=COW  ...ry Integrated

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libSkiaSharp.dylib                     0x1b0189f9e 0x1b014f000 + 241566

Infirmary Integrated-2023-02-06-122227.ips.zip

[tanjera]

Also wanted to thank you Oscar @greenfishgit22 for help in figuring out the full extent of the issue. While I don't think I can fix the heart of the problem, the fact that you said it doesn't crash if you pause it helps me to think I may be able to auto-pause when in the background, and that could circumvent it.

So thank you for bringing the bug to my attention and helping me pin down the problem and possible solutions!

[tanjera]

May have to do with:

https://github.com/AvaloniaUI/Avalonia/issues/4557

Per issue thread, appears to be fixed (?) in Avalonia preview 4... not sure if this is true... can create a test build on the preview packages to try to replicate... don't recommend moving Infirmary to target preview releases though... bad idea.

[greenfishgit23222]

Hi! I had to create a new account due to login issues. I really hope you can resolve this issue. I've noticed the same "bug" or memory overrun also occurs with cardiac monitor if it's left running in the background. Really hope you can find a solution for this problem because your application is quite excellent. It's a treasure for an RN like my self or a physician going into cardiology.

Let me know in the future if you need help with testing.

Cheers

//Oscar

[greenfishgit23222]

A couple of suggestions:

• Pressing spacebar "pauses" the simulation
• Having more than one rythm running, would be nice to have one regular sinus rythm running as a base for comparison when you'r trying to diagnose the other arrhythmias.
• Tools to measure RR/PP intervals

Thanks as always

[tanjera]

These are great suggestions! I will post them as issues so that I have a line item to-do list for when I am able to work on them in the future. Thank you for sharing the suggestions!

[tanjera]

I implemented some of the ideas you mentioned Issues. The only one I can't think of a way to properly implement would be Having more than one rythm running, would be nice to have one regular sinus rythm running as a base for comparison when you'r trying to diagnose the other arrhythmias. ... I actually took a screenshot yesterday when I was teaching a class and used the screenshot as a comparison (compared SVT to NSR side-by-side) ... so this is definitely a viable use case for the program. I'm just not sure how to go about implementing it, or if it is a good thing to implement for a simulator. I'll have to think on this one.

[tanjera]

By the way, thank you for the good ideas :) I definitely like them. I am currently swamped with work and school so I may have to wait for my next school break before I have a chance to work on these ideas. That's why I put them in as issues... so I could think about them and keep notes until I have a chance to program them.

[tanjera]

Reviving this issue because it appears to be wider-spread than initially thought. Also experiencing the same crashing on Linux, primarily with the 12-lead ECG and especially with the grid background. Doesn't appear to be strictly an issue with the drawing library- may be positioning or compositing issues with the image box re: red grid.

Potential solutions:

• Replace grid line draw method with vector-based drawing (.svg) instead of raster (.png)
• Replace grid line container draw method with programmatically-created lines instead of image-based drawing routines
• Explore repositioning elements; may be stacked container resizing issues (similar to previously-fixed "crashing on resize" bug w/ IICM)
• Upgrade framework to AvaloniaUI v11 ... Not sure if this will even fix this, though