jfabernathy commented 1 year ago

Platform: Ubuntu 22.10 arm64 Raspberry Pi 4 4GB
MythTV version: v32 latest as of 1/5/2023
Package version:
Component: mythtv-setup

What steps will reproduce the bug?

I was trying to install MythTV and came to the part where I'm setting up the backend with mythtv-setup and when I select New Video Source it never responds. I can't exit, and have to switch to a console login (Ctl-Alt-F3) and reboot.

I have found that if I run mythtv-setup via ssh -X ..., I can run New Video Source without issue.

I have it all working but not sure how to figure out what is happening with Video Sources running on the backend itself.

Here's some particulars. This is Ubuntu 22.10 on a Raspberry Pi 4 4GB. I saw the discussions on Mythtv Forum where some folks thought the video looked fine and wanted to have a complete Ubuntu desktop so they wanted to avoid the Raspberry PI OS install. On Ubuntu the ppa:mythbuntu/32 repository worked fine.

So I thought it was worth a test. I'm surprised how well the video looks running on top of Wayland and GDM inside Gnome 43~. The install and configure of Ubuntu 22.10 is slow as ever but when you're updated it runs pretty well and the Mythfrontend playback of recordings looks very good. But this hang up in mythtv-setup when NOT using ssh -X is a problem and I don't know how to provide the right debug information since it's locking me out. Maybe running from another PC on the network an ssh session with 'journalctl -f' and turning verbose on mythtv-setup??

How often does it reproduce? Is there a required condition?

What is the expected behaviour?

What do you see instead?

Additional information

jfabernathy commented 1 year ago

mythtv-setup.log this is the latest run where it hangs but I have a log. it was captured with: mythtv-setup.real --loglevel debug -v all 2>&1 | tee mythtv-setup.log

jfabernathy commented 1 year ago

I tried another test. To eliminate Wayland as an issue, I install Ubuntu 22.10 Server and then Ubuntu-mate-desktop (X11). mythtv-setup worked until I got to the same place, Video-Sources. When I hit [enter] on New Video Source the screen went black instead of just freezing as it did on Ubuntu Desktop.

Again I could reboot and use ssh -X from another computer to finish setting up mythtv. Then everything worked as normal.

kmdewaal commented 1 year ago

Just curious... Is it really only the New Video Source selection that causes the freeze/black and do all the other buttons/choices/menu's work OK?

jfabernathy commented 1 year ago

I went through the General and Tuners without any issues. I don't do Record Profiles and if I go into Video Sources I can delete all sources and then it asks to confirm, even though I have not sources at that time. It's just the New source button that nothing happens after its pressed. If I avoid New sources and go to Connections I can do some things but you need the Video Source to complete it. Same with Channel editor. You can setup all your Storage Directories.

I don't know if this is Ubuntu 22.10 or ARM64 or Wayland? I don't think it's Wayland because I get the same problem with Mate DE.

jfabernathy commented 1 year ago

okay, getting somewhere. I just built a fresh Ubuntu 22.10 desktop on a NUC PC 11th gen Core i7. I installed mythtv from the PPA and started configuring with mythtv-setup. I went down the line and all the configuration worked until Video Sources. The minute I clicked on (New Video Source) it froze just like it did on my Raspberry Pi 4 with Ubuntu 22.10 ARM64.

I also installed EndeavourOS (Archlinux) on both the RPI 4 and NUC PC and installed Mythtv from source. Both worked correctly in the mythtv-setup Video Sources menu as well as everywhere else.

jfabernathy commented 1 year ago

On the Ubuntu 22.10 NUC PC, I installed ssh and used it to ssh -X jim@192.168.0.100 which is the IP of the NUC PC and completed the mythtv-setup and Video Source setup worked via ssh.

Next I went back to mythtv-setup on the NUC without ssh and tried to delete the Video source I had setup via ssh and tried to add it back. Same freeze-up. So ssh -X is a workaround until we find this.

jfabernathy commented 1 year ago

BTW, I could reproduce all this on a Virtual Machine. I use Libvirt KVM/QEMU

paul-h commented 1 year ago

The last error in the log relates to running tv_find_grabbers baseline. If you run that command as the same user as you are using to run mythtv-setup do it run OK?

jfabernathy commented 1 year ago

Wouldn't tv_find_grabbers baseline require that XMLTV be installed? In this test I have not installed XMLTV. my tuners were both setup with EIT and that would be the Video Source I would choose if I had gotten that far.

paul-h commented 1 year ago

It's part of XMLTV so yes it needs to be installed to get the list of available grabbers. Are you saying it used to work OK without installing it? Ideally mythtv-setup would handle it better and just show the grabbers that don't require XMLTV and allow you to carry on, I don't know if it used to work that way and something has changed or maybe it was always assumed XMLTV would be installed?

jfabernathy commented 1 year ago

It's never been required to have XMLTV installed in the past. And pulling from github source it still works that way for 'aarch64' now. Let me install XMLTV and retest.

jfabernathy commented 1 year ago

I just installed XMLTV and then reran mythtv-setup. Still hangs/freezes at Video Sources (New Video Source).

Can you suggest a better debug command than what I used? That log is huge. Here's what I used: mythtv-setup.real --loglevel debug -v all 2>&1 | tee mythtv-setup.log

paul-h commented 1 year ago

OK it was the last thing in the log it must be a red herring.

Can you run it under gdb and when it hangs do a Ctrl-C and then get a full backtrace? That may give a clue where it is hanging.

jfabernathy commented 1 year ago

Ctrl-C never worked. I had to Ctrl-F3 to get to a console login outside of X11 and login. I can see mythtv-setup and mythtv-setup.real are running. Not sure how I can get you the data you need at that point. Not familiar with backtrace producing. I'm more of a hardware geek, or at least I was

paul-h commented 1 year ago

I meant use gdb mythtv-setup.real to start it in a terminal then when it freezes switch back to the terminal then do the Ctrl-C to get back to gdb then you can print a backtrace or whatever.

Alternatively if it's easier you can run mythtv-setup as normal then attach gdb to it. To do that when setup is running get it's PID. One way is pidiof mythtv-setup.real then in another terminal window attach gdb to it using gdb --pid=PID where PID is the one you found earlier. When setup freezes switch to the gdb terminal and press Ctrl-C and print the BT or whatever.

To get the full BT in gdb I normally use something like thread apply all bt full. To get useful BT's you may have to install a separate package with debug symbol how you do that depends on which version of MythTV you have. Older ones have just one debug package newer versions now have individual packages.

jfabernathy commented 1 year ago

Okay, please bear with me. I'm leaning as I go. So booted Ubuntu 22.10 and opened a terminal. I had to stop mythtv-backend.service before I ran setup because the prompt in mythtv-setup that pops up about needing to stop the backend before proceeding froze up.

So with mythtv-backend.service stopped, I then ran: gdb mythtv-setup.real I did some setup stuff then deleted all Video Sources and then selected New Video Source and it froze as before. I did Alt-Tab to show the tasks running and switched to the gdb console session and Ctrl-C to get a gdb prompt. At that point I did bt It displayed: `Thread 1 "mythtv-setup.re" received signal SIGINT, Interrupt. 0x0000fffff571e754 in __GI___poll (fds=0xaaaaaae6d880, nfds=6, timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41 41 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory. (gdb) bt

0 0x0000fffff571e754 in GI_poll (fds=0xaaaaaae6d880, nfds=6,

timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41

1 0x0000ffffef1dba7c in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0

2 0x0000ffffef1840f4 in g_main_context_iteration ()

from /lib/aarch64-linux-gnu/libglib-2.0.so.0

3 0x0000fffff5d60698 in QEventDispatcherGlib::processEvents

log.txt log.txt (QFlags) () from /lib/aarch64-linux-gnu/libQt5Core.so.5

4 0x0000fffff5cfe1b8 in QEventLoop::exec(QFlags) () from /lib/aarch64-linux-gnu/libQt5Core.so.5

5 0x0000fffff5d0724c in QCoreApplication::exec() ()

from /lib/aarch64-linux-gnu/libQt5Core.so.5

6 0x0000aaaaaaab3fcc in main ()

(gdb) `

If this does not help, let me know what I should do differently.

jfabernathy commented 1 year ago

log.txt

jfabernathy commented 1 year ago

This is MythTV Version : v32.0+fixes.202301091618.e677dd354f~ubuntu22.10.1

kmdewaal commented 1 year ago

Looks to me that the issue could be in another thread. When running under gdb, and after the Ctrl-C, can you give the gdb command thread apply all bt full and post the result?

jfabernathy commented 1 year ago

Just did that. Here's a long log. attached log2.txt

paul-h commented 1 year ago

Can you please install the MythTV debug symbols so we get a little more context in the BT.

I don't have access to a machine running the same version of MythTV from the PPA's so not sure what you need to install.

If you have the mythtv-dbg package available then install that. I suspect you wont and will need to install the individual debug packages which is new to me. If you run apt-cache search mythtv it should list the available packages look for ones with something like dbgsym in the name. For now I would just install a few like mythtv-common-dbgsym, libmyth-dbgsym and possibly mythtv-backend-dbgsym (not sure where mythtv-setup.real is in).

Hopefully that will give use a little more info on where those threads are waiting?

jfabernathy commented 1 year ago

I found mythtv-dbg and installed it.

jfabernathy commented 1 year ago

Spoke too soon mythtv-dbg is not up to date and it will not let me install

`Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: mythtv-dbg : Depends: mythtv-backend (= 2:32.0+fixes.20220325.f69ce764b7-0ubuntu2) but 2:32.0+fixes.202301091618.e677dd354f~ubuntu22.10.1 is to be installed or mythtv-frontend (= 2:32.0+fixes.20220325.f69ce764b7-0ubuntu2) but 2:32.0+fixes.202301091618.e677dd354f~ubuntu22.10.1 is to be installed E: Unable to correct problems, you have held broken packages. `

jfabernathy commented 1 year ago

I can build from source with certain options if it would help

paul-h commented 1 year ago

Not sure what is going on with the packages from the PPA. Looks like the individual debug packages should be there. https://code.launchpad.net/~mythbuntu/+archive/ubuntu/32/+build/25467074

If you want to install from source you just need to add --compile-type=debug to your normal run of configure.

jfabernathy commented 1 year ago

faster to build from source with the debug symbols than figure out what to install. Only 8 minutes to build. I'll repeat the gdb instructions from before and post the results.

jfabernathy commented 1 year ago

This time when gdb asked about downloading symbols from some Ubuntu website I said yes. But gdb also said that mythtv-setup.real had no debug symbols. Not sure why after I build from source and installed into /usr instead of /usr/local. Anyway the log file is attached gdb.txt

jfabernathy commented 1 year ago

I tried something else. This time I did gdb mythtv-setup I left the .real off and it seem to find symbols. Anyway the log file is more readable to me at least. see attached gdb.txt

paul-h commented 1 year ago

When you compile from source the executable is mythtv-setup. In Ubuntu they rename that executable to mythtv-setup.real and add a script called mythtv-setup which does the stuff like prompting you to shutdown the BE before it runs the real mythtv-setup executable and then attempts to restart the BE when it exits.

So basically when you want to use gdb it wants the actual executable so on Ubuntu you use mythtv-setup.real but nearly everywhere else you want to use mythtv-setup. Hope that makes some sense. It's got nothing to do with your problem though.

The BT's look a little better but I'm non the wiser :( To my untrained eye I don't see much in the way of MythTV code being run it's mostly external library stuff. There is some SDDP and logging stuff going on and some MythSystemLegacyIOHandler but little else.

jfabernathy commented 1 year ago

What is most puzzling to me is that when you run mythtv-setup in a terminal it freezes and it's doing something based on htop showing it's consuming 8% of the CPU. But if you run it via ssh -X on the same system; basically looping back on it self. the same command runs successfully. Same software, same system. One is using Gnome-terminal and the other using the console that you access from ssh. I guess the QT stuff isn't in play the same way with ssh -X, or is it.

paul-h commented 1 year ago

It's definitely strange :) When it locks up it's just the mythtv-setup ui that becomes unresponsive other apps are still OK?

We could try one more time to get a BT this time we will also increase the logging so hopefully we can get an idea of what it's trying to do when it locks up.

Using your compiled mythtv-setup run it under gdb gdb mythtv-setup

When gdb has loaded run mythtv-setup with these verbose flags (or some other combination that best shows what it's doing) run -v most:debug

When setup freezes press Ctrl-C in gdb and print the backtrace.

Post the entire gdb session so we get both the logging and the bt. There are options to turn off pagination and also log to a file in gdb if needed.

jfabernathy commented 1 year ago

Hope I have it all. I now can do this all on a VM on my laptop. makes it easier to work on. gdb.txt

paul-h commented 1 year ago

Not quite what I was expecting. I was trying to get both the gdb log and the mythtv-setup log at the same time but because you have set debuginfod enabled the mythtv-setup logging is going to a file somewhere else.

You can probably turn that off with set debuginfod enabled off before you run mythtv-setup.

jfabernathy commented 1 year ago

gdb.txt

Not sure there's any more setup console log. Do we need to include --logpath? I tried that on one run and it didn't produce a file

jfabernathy commented 1 year ago

gdb.txt mythtv-setup.20230111142304.3148.log Okay, I got a console log file and gdb log. AND keep in mind I do not have XMLTV installed so no tv_find_grabbers

jfabernathy commented 1 year ago

I must not have included the bt in the last gdb.txt gdb.txt mythtv-setup.20230111144408.3283.log

kmdewaal commented 1 year ago

It must be something in the Qt - Wayland - OpenGL area. I cannot reproduce the problem as described but I do have an issue with loss of input focus when using the wayland mode of operation.

If the MythTV window goes from completely obscured to completely visible then it does NOT get input focus and you cannot do anything with it. This is what also happens if the MythTV is full screen and it is alt-tabbed back and forth. I think this is similar to the problem described here. The workaround for me is to use MythTV in a window, and an expose of a partially obscured window does restore input focus.

Here it works OK if I use the xcb platform option: [klaas@kasus mma-satip]$ mythtv-setup -platform xcb

However, in the mythtv-setup log in the previous post it looks like the platform is already xcb.

If you give a wrong platform then the following message is given: Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, wayland-xcomposite-egl, wayland-xcomposite-glx, xcb. It might be interesting to try them all and see what happens.

It could just be that this issue can only be solved in the hardest possible way and that is understanding the code and fixing it....

jfabernathy commented 1 year ago

Most of the platforms I tested just exited. 2 actually opened the Setup menu, xcb and eglfs. However eglfs had no keyboard control. Couldn't move the arrows or click with a mouse. xcb worked except for the Video Source problem we've been chasing.

EDIT: I also logged into the Ubuntu desktop with X11 instead of Wayland. Same results.

The simplest workaround for me is to run mythtv-setup from an ssh session. From a Gnome Terminal I did: ssh -X jim@localhost Then run mythtv-setup That works everytime.

paul-h commented 1 year ago

@jfabernathy You said earlier that it was just the edit video sources screen and all the other screens work OK. If that is true we should be looking at what is different about that screen. One thing would be the running or attempting to run the external tv_find_grabbers which does temporarily disable key press handling.

If you do what @kmdewaal was suggesting and run mythtv-setup in a window and try swapping focus to another window and/or cover the window with another window then switch back to the setup window does that bring it back to life again?

jfabernathy commented 1 year ago

@paul-h My current test setup does not have XMLTV installed so no tv_find_grabber. I tested a while ago with XMLTV install in another setup and it made no difference. However we don't have a log with that installed. I'll try that.

How to I run mythtv-setup in a window? is that an argument? I'll look. EDIT --geometry 1080x768 works

paul-h commented 1 year ago

Whether you have XMLTV installed or not it still tries to run it so the steps taken are the same. I did briefly look at the code around that and nothing stood out as being wrong although I didn't dig too deep.

You can probably run in fullscreen and Alt-TAB (or whatever your window manager uses to switch windows/apps) to get the same effect.

jfabernathy commented 1 year ago

I ran mythtv-setup on a RPI4 without debug symbols since my Laptop was tied up. Just Straight up PPA install. But with --geometry 1080x768. When I got to the part where New Video Source hung, I played with other windows and came back to mythtv-setup. Nothing happened. Still no response. Alt-tab out and ctrl-c to exit.

jfabernathy commented 1 year ago

Maybe this gives new information or not. I installed xmltv on my Ubuntu 22.10 VM with mythtv compiled with debug on. This is a gdb run with -v most:debug and --logpath. gdb.txt mythtv-setup.20230112000441.3402.log It looks like the setup completed tv_find_grabbers and then....

paul-h commented 1 year ago

Well it's consistent with the other logs.

Nothing in the BT's you have provided to me suggest why it's frozen. The main thread #1 looks to be in the main QT event loop and to me looks idle waiting for something to happen. There 2 or 3 threads waiting on the logger but I think that is normal. Most others are waiting for something to do.

kmdewaal commented 1 year ago

I can reproduce the problem. I did a clean install of Ubuntu 22.04 with fixes/32 on my Intel i7-7700 desktop with Intel graphics and then with mythtv-setup I can add a Video Source, no problem. Everything default, so with Gnome and Wayland. Then I did a clean install of Ubuntu 22.10 with fixes/32 on the same machine and then I do get the blank screen after clicking the New Video Source button. To be continued.

paul-h commented 1 year ago

There is another report of the same problem on the forum with similar logs stopping in the same place and also using Ubuntu 22.10. https://forum.mythtv.org/viewtopic.php?f=36&t=5236

kmdewaal commented 1 year ago

I have not found it but I did come a bit closer. @paul-h was very close with suspecting tv_find_grabbers. It is however not the tv_find_grabbers command itself that causes the problem but the MythSystemLegacy class that is used to execute that command. Replacing tv_find_grabbers with another command else gives the same problem.

This are the lines where it fails, in libs/libmythtv/videosource.cpp:300-305:

            MythSystemLegacy find_grabber_proc("tv_find_grabbers", args,
                                                kMSStdOut | kMSRunShell);
            find_grabber_proc.Run(25s);
            LOG(VB_GENERAL, LOG_INFO,
                loc + "Running 'tv_find_grabbers " + args.join(" ") + "'.");
            uint status = find_grabber_proc.Wait();

When this is commented out so that the find_grabber_proc is not created and not executed then adding the Video Source works OK.

This are the last lines in the log after the New Video Source is selected:

2023-01-15 20:28:34.874398 I [127373/127373] CoreContext mythinputdevicehandler.cpp:192:IgnoreKeys  InputHandler: Locking input devices
2023-01-15 20:28:34.874410 N [127373/127373] CoreContext mythmainwindow.cpp:2157:PauseIdleTimer  Suspending idle timer
2023-01-15 20:28:34.938044 I [127373/127373] CoreContext mythinputdevicehandler.cpp:194:IgnoreKeys  InputHandler: Unlocking input devices
2023-01-15 20:28:34.938054 N [127373/127373] CoreContext mythmainwindow.cpp:2162:PauseIdleTimer  Resuming idle timer

My best guess at this moment is that the input events (from the keyboard in this case) that do arrive in MythTV are not routed to the GUI code. Yes, this is a vague statement but that matches my understanding.

Note the following:

The problem is present with master, fixes/32 and fixes/31 on Ubuntu 22.10, compiled with --compile-type=debug
The MythSystemLegacy code dates back from the times of Isaac Richards and Gavin Hurlbut and there have not been any relevant changes since, other than some modernization.

For me this rules out a regression in the MythTV code.

It is also unlikely that Ubuntu 22.10 is too new with respect to Qt and kernel versions; I use Fedora 37 and that uses later versions of Qt and the kernel.

The MythSystemLegacy is used all over the place so it would not surprise me if there are many more issues with Ubuntu 22.10, especially in MythMusic and in the DVD archiving because they also have a UI.

Help is definitely appreciated, especially insights about how events are passed on in MythTV and what possibly can have changed in Ubuntu.

kmdewaal commented 1 year ago

The function MythSystemLegacy::HandlePreRun in mythsystemlegacy.cpp is called before the command is executed and the HandlePostRun afterwards. When the following lines in HandlePreRun are removed:

    // This needs to be a send event so that the MythUI m_drawState change is
    // flagged immediately instead of after existing events are processed
    // since this function could be called inside one of those events.
    if (GetSetting("DisableDrawing"))
    {
        QEvent event(MythEvent::kPushDisableDrawingEventType);
        QCoreApplication::sendEvent(gCoreContext->GetGUIObject(), &event);
    }

it works OK. I do not see anything going wrong on the screen with this hack, not even when the complete HandlePreRun and HandlePostRun are both removed. However, there must be a reason why the code is the way it is....

paul-h commented 1 year ago

That's a good question I don't know why we disable ui drawing when running external scripts. It does make some sense to disable mouse and key presses to prevent the users accidentally moving the focus or activating buttons while waiting for the script to exit but don't remember why we also disable drawing. It could be no longer require when using MythUI but was required when using the old Qt based mythtv-setup?

There are a bunch of flags you can pass to MythSystemLegacy to turn on/off those settings. They are in mythsystem.h https://github.com/MythTV/mythtv/blob/master/mythtv/libs/libmythbase/mythsystem.h#L34

paul-h commented 1 year ago

Ah! I'd forgot about this :) https://github.com/MythTV/mythtv/commit/c1250a665ab8a5a5

MythTV / mythtv

mythtv-setup hanging at Video Sources #690

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behaviour?

What do you see instead?

Additional information

0 0x0000fffff571e754 in GI_poll (fds=0xaaaaaae6d880, nfds=6,

1 0x0000ffffef1dba7c in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0

2 0x0000ffffef1840f4 in g_main_context_iteration ()

3 0x0000fffff5d60698 in QEventDispatcherGlib::processEvents

4 0x0000fffff5cfe1b8 in QEventLoop::exec(QFlags) () from /lib/aarch64-linux-gnu/libQt5Core.so.5

5 0x0000fffff5d0724c in QCoreApplication::exec() ()

6 0x0000aaaaaaab3fcc in main ()

MythTV / mythtv

mythtv-setup hanging at Video Sources #690

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behaviour?

What do you see instead?

Additional information

0 0x0000fffff571e754 in __GI___poll (fds=0xaaaaaae6d880, nfds=6,

1 0x0000ffffef1dba7c in ?? () from /lib/aarch64-linux-gnu/libglib-2.0.so.0

2 0x0000ffffef1840f4 in g_main_context_iteration ()

3 0x0000fffff5d60698 in QEventDispatcherGlib::processEvents

4 0x0000fffff5cfe1b8 in QEventLoop::exec(QFlags) () from /lib/aarch64-linux-gnu/libQt5Core.so.5

5 0x0000fffff5d0724c in QCoreApplication::exec() ()

6 0x0000aaaaaaab3fcc in main ()

0 0x0000fffff571e754 in GI_poll (fds=0xaaaaaae6d880, nfds=6,