App crashes or stops decoding new telemetry

[K4KDR]

Using Ubuntu Linux 16.04 (64-bit) with commit ec70858 ...

This issue may not be unique or new to this commit - prior to current day I had not tested the app with more than a few packets.

Issue: When sending test packets at a ~2 second interval, after 6 or 7 packets the application with either crash & close or the time stamp will stop updating and no new packets are logged. If packets will never be received remotely close to this pace, perhaps this is not an issue. However, I wanted to report it.

Log file of last test attached.

minxss_beacon_decoder_debug.log

[jmason86]

I'll do similar "high speed" testing on my end to see if I can replicate. I'd like to resolve this even though it's a non-flight-like situation. At least for MinXSS, the fastest rate we would ever even possibly see is 3 second beacons, but that will be rare. Typically, it will be 9 seconds. Still, other satellite programs that fork this repo shouldn't have to worry about this, so I'll definitely take a look. Thanks for catching that!

[K4KDR]

Re-tested with 6-second gap between packets; application crashed/closed after 16 decodes.

Log file:

minxss_beacon_decoder_debug--6-second-test.log

[jmason86]

Hm... log files look normal. Could you copy paste the traceback if you have it (all the errors that print out in the console when it crashes). Have yet to try replicating it myself.

[K4KDR]

Update: as of commit c92170a, the GUI no longer crashes. However, after numerous tests with incoming packets at 3, 6, and even 10 second intervals, the "Last Packet At:" time stops updating and examination of the log & output files confirms that only a limited number of packets were decoded. The GUI can be "disconnected" normally and closes normally. The only feedback at the prompt where it was launched from is the same regardless of whether 1 packet is decoded or if it processes data until no longer incrementing the time stamp. That is:

QThread: Destroyed while thread is still running
Aborted

... additional info: I have confirmed that the Direwolf software modem continues to successfully decode each packet from the recording loop that is playing even after the GUI has stopped incrementing the packet counter.

Summary: the audio file with a test packet plays non-stop with a 10-second gap between packets, Direwolf decodes a packet every 10 seconds and provides it to the GUI over the KISS connection, but beyond some random (varying) number of packets, no additional data is placed in the LOG file and the "Last Packet At:" counter does not increment. However, the GUI is -not- frozen and can be ended normally.

[jmason86]

I'm able to replicate some parts of the issue. I can get the "Last Packet At" counter to stop and I noticed that it does update if I click on a different application and when I click back. That's weird and hopefully I can fix that.

I'm not able to replicate the ceasing of the data though. I've even tested with 0.2 second cadence and still the GUI continues to decode and display the telemetry, the log file shows the decoded data, and the output/ files continue to update.

I'm using the test_send_tcpip_packets in an infinite loop. Maybe that's an important difference. Your .wav file is ust a single packet that's being looped over and over rather than a series of packets? (I'm hoping that maybe there's a bad packet due to a bit flip or something in there that I'm not handling properly).

[K4KDR]

Yes, source of the test packet is a single packet recorded from MinXSS-1. A couple of observations that may help:

1, when looping, it continues to decode properly in Direwolf (even after the beacon decoder GUI stops indicating that any new packets are arriving)

2, I assume (with no basis) that since the same packet is being used over and over, if it is processed properly once, it should be processed properly repeatedly. This "is" the case in the Direwolf software modem (comment #1 above)

And while valuable, my concern with the test_send script is that it's not a real-world illustration of what the app will actually see. Not a criticism! But I do generally prefer to test as close as possible to the conditions that an app will actually be used with.

[jmason86]

Totally agreed that your test is the better one. I just don't have things set up to do the same kind of testing. I'm on macOS so would ideally find a Direwolf equivalent I could run on here. I am getting things set up on my Windows partition now though so I could run it from there.

I'm kind of stuck on resolving this until I have a way to replicate this error. The log files and description and inconsistent behavior are not narrowing it down enough that I know where the bug is. Working on getting everything up in Windows so hopefully I can reproduce it and find this bug.

[K4KDR]

Please don't forget that the KISS connection is over TCP, so you can absolutely run the GUI on one platform but play sample packets on a completely different computer on the same local network. Just remember to disable any software firewalls on both platforms. FYI the typical audio file -> to software modem combo's used by most are: LInux: Audacity > Direwolf, Windows: Audacity > UZ7HO Soundmodem (although Direwolf works extremely well on Windows, too). For "live" over-the-air signal processing, replace "Audacity" with the SDR software of your choice. Please forgive if the above is common knowledge in your circle.

[K4KDR]

Running commit 07ed428 from a command prompt on Ubuntu 16.04 (64-bit) with an audio recording of test packets looping every 6 seconds, the GUI crashed & closed on multiple tests after varying number of packets. The following text was displayed in the terminal window:

python3: ../3rdparty/harfbuzz/src/harfbuzz-myanmar.c:535: void HB_MyanmarAttributes(HB_Script, const HB_UChar16*, hb_uint32, hb_uint32, HB_CharAttributes*): Assertion `i == boundary' failed.
Aborted

[jmason86]

Some googling on that error reveals that it is from an open source Linux font shaper. That means that part is out of my control and since I don't have a Linux machine, I won't be able to replicate it.

However, it would be good to know what in my code is calling that; maybe I could just change the font or something like that to prevent it from becoming an issue. Was there anything more printed in the terminal window? Usually there's a longish traceback that would show which lines of my code invoked this.

[K4KDR]

No, afraid that was the sum total except for one line that has always appeared regardless of whether the app ran properly or not. It did not think it relevant, but I should not have omitted it. It prints immediately when the app is first run:

k4kdr@:~/MinXSS_Beacon_Decoder$ python3 minxss_beacon_decoder.py 
libpng warning: iCCP: known incorrect sRGB profile

[jmason86]

Hm shoot. Yeah I see that message every time too. Something to do with colors but I have no suspicions that it's causing trouble.

I'll try googling around on that error some more (but the results so far haven't been very promising). My best suggestion at this point is to try

1. opening the code directory as a project in PyCharm
2. setting your project interpreter to the python3 environment you've made
3. run menu > edit configuration to look like this (except with your python3 environment set instead of my beacon_decoder one.
4. running it in debug mode (just click the bug symbol in the top right), which should then provide the full traceback

[K4KDR]

Struggling a bit with PyCharm, but will keep trying. Thanks for the detailed how-to.

[K4KDR]

Ok, I believe that PyCharm is operating properly for me. I opened & ran the app w/ DEBUG and repeated my test w/ a recording that looped every 6 seconds. Using commit 2cdf5f1 , there was no crash but the app stopped ingesting packets evidenced both by the time stamp on the main tab of the GUI as well as in the log file. At the time the app stopped updating the time stamp, the following line appeared in the CONSOLE window at the bottom of the PyCharm:

OpenType support missing for "Ubuntu", script 9

... and here is the entire CONSOLE window from PyCharm from program launch (via DEBUG) to graceful program exit:

/usr/bin/python3.5 /snap/pycharm-community/85/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 38187 --file /home/k4kdr/MinXSS_Beacon_Decoder/minxss_beacon_decoder.py
pydev debugger: process 19665 is connecting

Connected to pydev debugger (build 182.4505.26)
libpng warning: iCCP: known incorrect sRGB profile
  OpenType support missing for "Ubuntu", script 9
<html><head><title>MinXSS HAM Radio Beacon Upload</title></head><body>
<form method='post' action='fileupload.php' enctype='multipart/form-data'[>
Select File: <input type='file' name='filename' size='10' />
<input type='submit' value='Upload' /></form><br/>Uploaded file '2018-10-16T00_47_54.449581_K4KDR_37.7784_-77.6093.dat'<br/></body></html>

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

The log file from this one test is attached. It appears 8 packets were processed before the app stopped accepting them.

minxss_beacon_decoder_debug.log

[jmason86]

Thanks! That was quite helpful and I think has pointed us in the right direction. It seems that pyside2 does have hooks to load fonts but there's a disconnect somewhere. Either my application is asking your system for the OpenType Ubuntu font, or your system is saying it wants to use that font but pyside2 is missing that font file.

This made me look at some of your previous screenshots and it looks like it is indeed using a font that I didn't specify. I set everything to use Copperplate Gothic Light, which used to be native to Windows but other operating systems have to install it. I had to for my mac. It may not even be supported by OpenType, only TrueType (whatever that means) so I may need to change the font in the app to something more widely available.

Could you try installing Copperplate Gothic Light on your Ubuntu machine and then testing the app for crashes? All you should have to do is install is download, unzip, double click on the .tff file.

[K4KDR]

The font has been installed and certainly does change the app's appearance!

... as before, I fed packet audio to direwolf at an approx. 4 second interval & the decoder GUI was connected via KISS. Sometimes, the app would just stop updating; in those cases, nothing was logged in PyCharms. In other cases (no difference in test scenario), the app would crash. The following was the console output in PyCharms:

/usr/bin/python3.5 /snap/pycharm-community/85/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 35693 --file /home/k4kdr/MinXSS_Beacon_Decoder/minxss_beacon_decoder.py
pydev debugger: process 11310 is connecting

Connected to pydev debugger (build 182.4505.26)
libpng warning: iCCP: known incorrect sRGB profile
python3.5: ../3rdparty/harfbuzz/src/harfbuzz-myanmar.c:535: void HB_MyanmarAttributes(HB_Script, const HB_UChar16*, hb_uint32, hb_uint32, HB_CharAttributes*): Assertion `i == boundary' failed.

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

[jmason86]

Hm, that's the same error you were getting before. This seems to be the code that's crashing, but I note a line number discrepancy. Your error is saying line 535 should be assert(i == boundary); but that is line 539 in the linked code. That discrepancy is suspicious.

In fact, that link above is the only place I could find the harfbuzz-myanmar.c code that is crashing, which is a file that doesn't even exist in the official HarfBuzz repo anymore. Perhaps you installed this OpenText implementation awhile ago and it contained this bug?

1. Do you expect your system to be doing something with fonts from Myanmar?
2. Are you willing to try re-installing HarfBuzz?

[K4KDR]

The additional 2 lines are the

#!/usr/bin/env python3

... and a blank line at the beginning of the minxss_beacon_decoder.py file so that it can be run from the file explorer.

Don't know what harfbuzz is, but installed successfully from:

https://github.com/harfbuzz/harfbuzz

(before installing harfbuzz, installed all dependencies listed in install instructions)

Using commit 37d2642 , ran minxss_beacon_decoder.py using "debug" in pycharm-community.

Test packets were played from the audio recording as usual & began showing up in the GUI. The GUI crashed/closed with the following in pycharm-community:

/usr/bin/python3.5 /snap/pycharm-community/85/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 38751 --file /home/k4kdr/MinXSS_Beacon_Decoder/minxss_beacon_decoder.py
pydev debugger: process 10869 is connecting

Connected to pydev debugger (build 182.4505.26)
libpng warning: iCCP: known incorrect sRGB profile

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

With the app closed, the only measure of how much data was ingested was the LOG file (attached). The packet count was 37 - much larger than previous tests.

minxss_beacon_decoder_debug--2018-10-22.log

[jmason86]

Those additional two lines are in different code files though, so that still doesn't explain it.

This is certainly progress! It didn't crash due to that bug in the harfbuzz myanmar code. We've now gotten to that code 139 error, which you've also seen before in PyCharm.

I did some googling and this stackoverflow answer has a good suggestion, indicating that the problem may actually be PyCharm. So, two things to try:

1. PyCharm > Settings > Build, Execution, Deployment > Python Debugger: make sure PyQt compatible is checked, and change the drop down menu from "Auto" to "PySide"
   • This may still not work because we're actually using PySide2, which came out somewhat recently but is now the official Qt for Python. Sidenote: I've put in the request to the PyCharm devs to add support for PySide2.
2. Skip PyCharm altogether and try any of your more usual methods of running the app.

[K4KDR]

PyCharm setting changed from Auto to PySide & ran DEBUG from inside PyCharm-Community again. Unfortunately not getting consistent results... this time the harfbuzz error message returned.

As you suggested (thank you!), to simplify, I removed the two lines I had added to minxss_beacon_decoder.py so that my code was exactly as provided in the repository. Running from a command line (entire sequence shown below), the results are different once again (log file attached-- 17 packets decoded). Very sorry that I cannot produce the same error repeatedly - I realize that makes things difficult. Will test on Windows this evening as a check against these results in linux.

k4kdr@:~/MinXSS_Beacon_Decoder$ echo $LD_LIBRARY_PATH 
/home/k4kdr/.local/lib/python3.5/site-packages/PySide2/Qt/lib/
k4kdr@:~/MinXSS_Beacon_Decoder$ python3 minxss_beacon_decoder.py 
libpng warning: iCCP: known incorrect sRGB profile
Segmentation fault
k4kdr@:~/MinXSS_Beacon_Decoder$ 
k4kdr@:~/MinXSS_Beacon_Decoder$ git show --oneline -s
37d2642 Updated version in the UI for the new fileupload for windows
k4kdr@:~/MinXSS_Beacon_Decoder$ 

minxss_beacon_decoder_debug--2018-10-23.log

[jmason86]

I've been caught up at a conference but am still passively pondering this.

Had a chance to test on Windows?

[K4KDR]

After updating to the current commit (d70dc73), the application ran perfectly on Windows-7 (64-bit) using the same audio test file running in a continuous loop. This is great news since I would imagine that the majority of contributors would be using a Windows computer.

Specifically, Audacity was used to play the sample packets (from MinXSS-1) out to a Virtual Audio device. This made the audio stream available to the UZ7HO HS_SoundModem set to 9600 FSK. A KISS port connection was made available in HS_SoundModem and MinXSS_Beacon_Decoder used that KISS connection to receive demodulated data from HS_SoundModem.

If you replace the Audacity audio player with SDR software (I use HDSDR, but there are many others), this is how the majority of contributors would receive & decode packets from MinXSS-2.

Very sorry if the multiple versions of Python or any of the other rather extensive array of test software on my Ubuntu 16.04 computer gave the false impression that there was an issue with running the GUI long-term. It's not something that I would want to do (as my linux machine is working well in all other regards), but on a clean Ubuntu install, fully updated, with a single installation of Python, I would imagine that the application would run perfectly.

[jmason86]

Sweet!! Okay, I'm going to guess that this is largely a problem with that harfbuzz thing and assume that's not universal. If it pops up again on more people's machines, we can reopen this issue.