search

MythTV / mythtv

The official MythTV repository
https://www.mythtv.org
GNU General Public License v2.0
700 stars 344 forks source link

Move backend config from mythtv-setup to mythbackend internal web server #239

Closed stuarta closed 11 months ago

stuarta commented 3 years ago

It's cumbersome to have to use mythtv-setup (an X application) on a headless backend server. We should be able to perform all required setup via the webserver in the backend

Original trac ticket https://code.mythtv.org/trac/ticket/8262

stuartm commented 3 years ago

If we're being honest, it's cumbersome to use even with the current GUI.

Count me in on this task.

bennettpeter commented 1 year ago

All pages are done. Some items are not complete, ones that require hardware I do not have. Some capture cards types still need to be done, and channel scan needs to be done.

At the moment it is a straight replacement for mythtv-setup. It still requires the backend to be restarted after updates. Supporting updates while the backend is scheduling and recording has not been started.

As an interim measure, I am working on ensuing that setup is not done while recordings are active, by presenting messages to the user and giving the user the ability to stop the scheduler.

bennettpeter commented 1 year ago

I have added code to require that the scheduler be disabled before any changes can be saved. Also added buttons for disabling the scheduler and for restarting the backend after updates are complete.

kmdewaal commented 1 year ago

As discussed and promised, here a few comments/observations on today's version of the webserver.

(1) The "Backend Setup" button does select the "Initial Setup Wizard". I think it would be more consistent if there is a "Initial Setup Wizard" button that then starts a "Initial Setup Wizard" window. This is the kind of consistency I have graduallly built into mythtv-setup in the pages that I have changed over the years.

(2) The "Initial Setup Wizard" does sometimes overwrite the ~/.mythtv/config.xml file. It changes the LocalHostName setting back from what I defined it to be to "my-unique-identifier-goes-here". This then changed, on my laptop, the hostname from "kasus" to "kasus.home.lan"; this is then used as key in the settings table and this then effectively means all settings are lost, until you figure out what happens. This happened to me twice but it does not happen always.

(3) The "Backend Status" button shows a nameless page with various status and system information. OK. I think it would be more consistent and clear if the top of the page showed both the icon and the "Backend Status" text. This should be done on all pages; it makes that the user has always a clear view of where he is.

(4) The "Backend Status" page shows in the "Machine Info" field disk space: free, in use etc. I suggest to add a storage unit after the number. It looks like the values are given in MiB but today one MiB is not that much anymore. The free space as shown by mythfrontend is given in GiB so changing it to GiB is what I recommend.

(5) The "Backend Settings" icon does remind me of a sound mixer/equalizer. For me this is not a visual cue that this is the mythtv-setup configuration. The icon for system configuration, information and setup in Windows looks like a gear but that icon is used here for the "Backend Setup" also known as "Initial Setup Wizard" page. Maybe the "gear" icon can be used for the "MythTV Setup" items and I think the "Initial Setup Wizard" could be added to that list.

(6) The "Backend Settings" icon does create a popup menu with title "MythTV Setup" with which one of the mythtv-setup main menu choices can be made: General, Capture Cards etc. When the choice is made then a page is shown with the title "MythTV Setup" and below that then the selected choice. This is OK. What is inconsistent is that the icon is called "Backend Settings" (as shown when the cursor is near the button) but the page is "MythTV Setup".

(7) It would be easier if the "Backend Settings" a.k.a. "MythTV Setup" would show a secondary menu with a fixed position on the screen instead of a floating popup menu that disappears once a choice has been made. This would be easier if the main menu is horizontal on the top of the window instead of vertical on the left side; you can then easy make second row of buttons. This is how the menu's of my DD-WRT router are organized. I can provide a screenshot if that is appreciated. I do realize that a horizontal row of buttons is terribly old-fashioned but it does allow for texts on buttons in addition to, or replacing the, icons.

(8) The "TV Guide" button does show the TV Guide but this is not shown on the top of the page. It is of course fairly obvious that this is the guide but it would be consistent if there is "TV Guide" then shown on top of the page.

(9) The vertical scrolling of the TV guide is OK, but there is no horizontal scrolling of the TV Guide. This must be done with a calendar-like popup window. If horizontal scrolling is not possible then buttons to shift two hours, which is the width of the guide on the screen, and buttons for 4 hours, 8 hours and 24 hours shift back and forth would be an alternative. Although a bit old fashioned in a world where swiping is the norm.

N.B. I actually did do GUI's for a living many moons ago but it is not my hobby anymore. I do appreciate the effort you put in this and I hope my comments do help to improve it.

To be continued....

paul-h commented 1 year ago

A couple of notes.

The icons used come from the ones provide by PrimeNG the Angular UI library we chose to use and were chosen from what is available that looked vaguely appropriate. There may be more suitable icons available let us know if you spot something from here https://primeng.org/icons#list

The original purpose of the "Initial Setup Wizard" was to help new users. We would allow the backend to startup without any database or config at all (at the moment that is not possible). If the user then went to the web setup it would bootstrap into the "Initial Setup Wizard" to allow the new user to easily do some initial setup like choose the language, create the MySql database and setup access to it etc. When that initial setup is saved the backend could then be re-started with the DB and some other essential initial config now in place and then the more in depth set up stuff could be done. Peter didn't like that plan and said it wasn't needed so not sure what the current plan is.

bennettpeter commented 1 year ago

On 4/3/23 16:35, kmdewaal wrote:

As discussed and promised, here a few comments/observations on today's version of the webserver.

(1) The "Backend Setup" button does select the "Initial Setup Wizard". I think it would be more consistent if there is a "Initial Setup Wizard" button that then starts a "Initial Setup Wizard" window. This is the kind of consistency I have graduallly built into mythtv-setup in the pages that I have changed over the years.

(2) The "Initial Setup Wizard" does sometimes overwrite the ~/.mythtv/config.xml file. It changes the LocalHostName setting back from what I defined it to be to "my-unique-identifier-goes-here". This then changed, on my laptop, the hostname from "kasus" to "kasus.home.lan"; this is then used as key in the settings table and this then effectively means all settings are lost, until you figure out what happens. This happened to me twice but it does not happen always.

(3) The "Backend Status" button shows a nameless page with various status and system information. OK. I think it would be more consistent and clear if the top of the page showed both the icon and the "Backend Status" text. This should be done on all pages; it makes that the user has always a clear view of where he is.

(4) The "Backend Status" page shows in the "Machine Info" field disk space: free, in use etc. I suggest to add a storage unit after the number. It looks like the values are given in MiB but today one MiB is not that much anymore. The free space as shown by mythfrontend is given in GiB so changing it to GiB is what I recommend.

(5) The "Backend Settings" icon does remind me of a sound mixer/equalizer. For me this is not a visual cue that this is the mythtv-setup configuration. The icon for system configuration, information and setup in Windows looks like a gear but that icon is used here for the "Backend Setup" also known as "Initial Setup Wizard" page. Maybe the "gear" icon can be used for the "MythTV Setup" items and I think the "Initial Setup Wizard" could be added to that list.

(6) The "Backend Settings" icon does create a popup menu with title "MythTV Setup" with which one of the mythtv-setup main menu choices can be made: General, Capture Cards etc. When the choice is made then a page is shown with the title "MythTV Setup" and below that then the selected choice. This is OK. What is inconsistent is that the icon is called "Backend Settings" (as shown when the cursor is near the button) but the page is "MythTV Setup".

(7) It would be easier if the "Backend Settings" a.k.a. "MythTV Setup" would show a secondary menu with a fixed position on the screen instead of a floating popup menu that disappears once a choice has been made. This would be easier if the main menu is horizontal on the top of the window instead of vertical on the left side; you can then easy make second row of buttons. This is how the menu's of my DD-WRT router are organized. I can provide a screenshot if that is appreciated. I do realize that a horizontal row of buttons is terribly old-fashioned but it does allow for texts on buttons in addition to, or replacing the, icons.

(8) The "TV Guide" button does show the TV Guide but this is not shown on the top of the page. It is of course fairly obvious that this is the guide but it would be consistent if there is "TV Guide" then shown on top of the page.

(9) The vertical scrolling of the TV guide is OK, but there is no horizontal scrolling of the TV Guide. This must be done with a calendar-like popup window. If horizontal scrolling is not possible then buttons to shift two hours, which is the width of the guide on the screen, and buttons for 4 hours, 8 hours and 24 hours shift back and forth would be an alternative. Although a bit old fashioned in a world where swiping is the norm.

N.B. I actually did do GUI's for a living many moons ago but it is not my hobby anymore. I do appreciate the effort you put in this and I hope my comments do help to improve it.

To be continued....

Message ID: @.***>

Hi Klaas

Thanks for looking at this.

In my opinion the front page needs reworking.

The only parts I have worked on are the backend setup, on the sound mixer/equalizer icon. That is the part I mainly look for feedback on. I re-implemented the setup pages on mythtv-setup, even if they are not ideal. There was talk about redesigning the whole thing as a wizard, but I am at a loss on how that would look or work.

I see that nobody has worked on the front page, the guide or the "setup wizard" since I started on this about a year ago. These are all incomplete. If nobody is doing them, I can start on them once the setup is finished. I agree with all you are saying. The popup menu was a quick way to be able to navigate the setup pages until we get the home page sorted out.

I think backend setup needs one icon that has the "setup wizard" and the other setup  pages that are currently under the sound mixer icon.

I also would like to add some frontend type functionality, like list of recordings, upcoming recordings, the ability to schedule recordings. This will all take time.

Peter

kmdewaal commented 1 year ago

I have been using the Channel Editor to change the visibility of a few channels from Visible to Not Visible on my living room system. This was done completely OK and it is convenient to be able to do that while the backend is running. Suggestions for improvement:

bennettpeter commented 1 year ago

Thanks for the suggestion. I will look into it, it does not sound complicated.

paul-h commented 1 year ago

(2) The "Initial Setup Wizard" does sometimes overwrite the ~/.mythtv/config.xml file. It changes the LocalHostName setting back from what I defined it to be to "my-unique-identifier-goes-here". This then changed, on my laptop, the hostname from "kasus" to "kasus.home.lan"; this is then used as key in the settings table and this then effectively means all settings are lost, until you figure out what happens. This happened to me twice but it does not happen always.

I've just run into this as well while testing the latest version.

kmdewaal commented 1 year ago

In page "Backend Settings" / 5. Input Connections then expand for a capturecard. First setting is "Input Name" with default value None.

  1. This is (None) in mythtv-setup. The additional brackets indicate that this not a name but the absence of a name. Or so understand it. This does not matter much of course but it is a difference.
  2. The value of the "Input Name" field is most likely not used anywhere. I have somewhere on my "pre-ticket" list a note to figure out if it really can be removed.
  3. The help text shown below the "Input Name" field is not correct and is probably a cut&paste error from a capture card setting. In mythtv-setup there is no help text for this field.
kmdewaal commented 1 year ago

As discussed, I would trigger you when the "HDHomeRun Channel Import" functionality has been committed to master. This is done now and I expect it to work for ATSC but of course I have not tested that. To be honest, I do not know how you want to implement channel scanning. I am familiar with all the mythtv-setup code but there is something of a gap, if not an ocean, between the mythtv-setup code and the webapp...

gigem commented 1 year ago

Input name was needed on analog, capture cards. Input names would be something like Tuner, Composite and S-Video. I believe HD-PVRs are one such case that still might be in use where the choices for input name are S-Video and Component.

bennettpeter commented 1 year ago

I have incorporated all of the setup pages into the setup wizard under the cogwheel icon, and removed no longer needed icons and menu.

Fixed the problem of config.xml losing the custom id and the startup parameters. These are now in the setup page

bennettpeter commented 1 year ago

(2) The "Initial Setup Wizard" does sometimes overwrite the ~/.mythtv/config.xml file. It changes the LocalHostName setting back from what I defined it to be to "my-unique-identifier-goes-here". This then changed, on my laptop, the hostname from "kasus" to "kasus.home.lan"; this is then used as key in the settings table and this then effectively means all settings are lost, until you figure out what happens. This happened to me twice but it does not happen always.

Fix fixed

bennettpeter commented 1 year ago

In page "Backend Settings" / 5. Input Connections then expand for a capturecard. First setting is "Input Name" with default value None.

  1. This is (None) in mythtv-setup. The additional brackets indicate that this not a name but the absence of a name. Or so understand it. This does not matter much of course but it is a difference.
  2. The value of the "Input Name" field is most likely not used anywhere. I have somewhere on my "pre-ticket" list a note to figure out if it really can be removed.
  3. The help text shown below the "Input Name" field is not correct and is probably a cut&paste error from a capture card setting. In mythtv-setup there is no help text for this field.

This is a weirdness that exists in mythtv-setup. The column in the capturecard table in inputname. In the DVB capture card setup that field is called Delivery System and uses values obtained from the card. In my case it has ATSC and DVBC/B as options. In the case of other types of card the value is called input name and is entered in the input connections page. I don't know why there is this inconsistency. I also don't know if MythTV uses the value ATSC vs DVB/C for anything, but you have to select one. I don't know whether other devices supply a value to be selected, I think maybe HDPVR does.

The help text is the same text as used for DVB when you can select a value for input name. I suppose I could change the text from "If your card supports more than one delivery system then you can select here the one that you want to use." to "If your card supports more than one input name then you can select here the one that you want to use.", or change the title in input connections to "delivery system".

kmdewaal commented 1 year ago

On Tue, 9 May 2023 at 17:33, Peter Bennett @.***> wrote:

In page "Backend Settings" / 5. Input Connections then expand for a capturecard. First setting is "Input Name" with default value None.

  1. This is (None) in mythtv-setup. The additional brackets indicate that this not a name but the absence of a name. Or so understand it. This does not matter much of course but it is a difference.
  2. The value of the "Input Name" field is most likely not used anywhere. I have somewhere on my "pre-ticket" list a note to figure out if it really can be removed.
  3. The help text shown below the "Input Name" field is not correct and is probably a cut&paste error from a capture card setting. In mythtv-setup there is no help text for this field.

This is a weirdness that exists in mythtv-setup. The column in the capturecard table in inputname. In the DVB capture card setup that field is called Delivery System and uses values obtained from the card. In my case it has ATSC and DVBC/B as options. In the case of other types of card the value is called input name and is entered in the input connections page. I don't know why there is this inconsistency. I also don't know if MythTV uses the value ATSC vs DVB/C for anything, but you have to select one. I don't know whether other devices supply a value to be selected, I think maybe HDPVR does.

The help text is the same text as used for DVB when you can select a value for input name. I suppose I could change the text from "If your card supports more than one delivery system then you can select here the one that you want to use." to "If your card supports more than one input name then you can select here the one that you want to use.", or change the title in input connections to "delivery system".

It looks like I have fallen into a trap of my own making here. The first serious change I made in MythTV a few years ago was adding Delivery System support, specified in the dvbv5 api, for digital tuner cards. This was and is necessary because modern digital tuner cards can support for instance DVB-T2 (terrestrial) and DVB-C (cable). The card can tell you what its capabilities are and then you have to tell the card which delivery system has to be used. So I did need a field in the database table capturecard for that. I then got the guidance that changing the database schema was not a good idea because then the mythtv schema version needs updating and then every mythfrontend needs to be updated. There was the field inputname used by analog tuners but that had no use in digital tuners so that was then suggested to be used to store the Delivery System. Of course, many database schema changes have been made since.... So, wrapping this up, for analog cards it is an input selection (e.g. S-Video, Component or Composite) and for digital cards it is the Delivery System selection. The HDHomeRun needs a bit of thinking because there is also a model that supports both DVB-T2 and DVB-C but most of the time (not always...) it can figure out everything by itself.

Klaas.

bennettpeter commented 1 year ago

I have made the change to support HDHomerun Channel Import but I cannot test it. If you test, please let me know if there are any problems.

kmdewaal commented 1 year ago

First, the looks are much better now! I appreciate how the mythtv-setup setting pages can be selected now. I did have a quick go at trying to do a "HDHomeRun Channel Import" but that fails in the same way that a conventional "Full Scan" fails, with a "Programmer Error: Failed to handle tune complete." error. Which is a bit of code that the "HDHomeRun Channel Import" should not reach at all. I somehow have the feeling that this API stuff is about as low level as the C++ code that makes up mythtv-setup. Which means there are many lines of code to duplicate.... I can give a more step-by-step report maybe tomorrow. A look at the code might be useful as well, but I do no know what " the code" here is. Going through loads of generated javascript or so is definitely not my thing.....

Slightly OT for this thread... what is device are you using for analog capture / V4L2 ?

bennettpeter commented 1 year ago

The web based channel scan uses channelscanne_web.cpp which tries to duplicate what is done in channelscanner_gui.cpp

The client side source code is all in the /mythtv/mythtv/html/backend/src directory. Definitely do not look at the generated javascript.

The channelscan part is in src/app/config/settings/input-connections/channelscan/channelscan.component.ts and src/app/config/settings/input-connections/channelscan/channelscan.component.html.

The client uses these services to request the scan on the backend:

StartScan has 32 parameters. The scan runs in a separate thread, as it already does in mythtv-setup. The results are put in fields in the ChannelScannerWeb class, which has a single instance. GetScanStatus retrieves the results from the ChannelScannerWeb object to display them to the user. Only 1 scan can run in a backend at a time.

If the scan is not working, probably some changes need to be made in the ChannelScannerWeb (channelscanner_web.cpp). The StartScan method there is called from the API. I added a check for HDHRIMPORT, but maybe something else needs to be done.

bennettpeter commented 1 year ago

Slightly OT for this thread... what is device are you using for analog capture / V4L2 ?

USB Capture device. There are many brands available from Amazon. Those that advertise 3840x2160 input and 1920x1080 output, costing $5 and up, all seem to be clones of the same thing. These are USB 2 devices. Running lsusb with any of them mounted shows the device with ID 534d:2109.

See https://github.com/bennettpeter/MythTV-LeanCapture for the full code

bennettpeter commented 1 year ago

I fixed a copy/paste error in channelscanner_web.cpp, the clone of a VBOX line for HDHR was not done correctly. Try it again.

kmdewaal commented 1 year ago

2023-05-10_21-48_hdhomerun_import

This is succes! The message "Failed to find any new channels" is obviously not correct here. Before doing this with the webapp I did delete all channels in video source AT with mythtv-setup Channel Editor. I think a good next step is to add to the webapp Channel Editor at least the capability to select a video source, and a button to delete all channels from a video source. Then it is possible to do a scan, delete all channels, and do the scan again.

kmdewaal commented 1 year ago

2023-05-10_22-20_scan_existing_transports The "Scan of all existing transports" and the "Scan of single existing transport" is OK.

kmdewaal commented 1 year ago

2023-05-10_22-45_ful_scan_selection As the message says, for a Full Scan there must be a country selected in mythtv-setup. The country does select the channel table to use. There is a default but that does not need to be correct, so a country field would be useful. It may even be better to show a message to check the country selection in the first tab because that should be the selection that is used.

N.B. If I do change the country, do Save, and then go back to mythtv-setup then the first time the old country selection is still used; after starting mythtv-setup a second time the newly selected country is used. This is a bug in the cache/contextcache.xml mechanism; the values in the cache take precedence over the values in the database and only at program exit the values from the database are written to the cache..... Correct behavior is IMHO that at startup the values are read from the database and only when the database cannot be accessed the values from the contextcache are used.

bennettpeter commented 1 year ago

I have committed all changes to support web-app setup.

The backend can start without a database, so that the database can be set up using the web app.

If any of these conditions exist at backend startup, the backend will start in a web-server only mode:

There is a restart button at the top of the web pages for them to restart once the setup is done.

This changes a few things about mythbackend startup. In the cases mentioned above, the backend would previously do one of two things:

Now it will still output a message, but then instead of prompting the user or shutting down, it will start up in web-server only mode.

This still needs testing. Some thing I could not test, where I do not have the required hardware or live in the appropriate country.

Translation is done via the ngx-translate package. The default language in en_US. For testing I loaded one string, the title "MythTV Setup" in the Dutch language (Nederlands). I do not know how to organize the translations. The strings are in json files, one per language.

Please test this and let me know of any problems, questions or suggestions.

To run it use http://backend:6744 and click the cogwheel icon.

I plan to continue with the web app, replacing the home page with something sensible and creating pages that do what mythweb does today.

bennettpeter commented 1 year ago

I have replaced the dashboard with a tabbed view that contains backend status, channel edit, program guide, and I will be adding other tabs like Recordings, recording rules, videos, etc.

I have also removed the restriction on channel edit so that you can update channels while the scheduler is enabled.

jfabernathy commented 1 year ago

I just started testing this at Peter's request on the mailing list.

My interest in this app is for remote mythtv backend administration. So to test this more easily, I'm using KVM/QEMU virtual machines containing a server Linux distro with mythtv built from source. I started with Ubuntu 22.04 LTS server.

After building the server, I did all the work from an ssh terminal on my workstation. I have HDHomerun ATSC 1.0 tuners on my local network and all VM are on a bridge which makes them on the same subnet at the HDHR tuners.

I used ansible to get the dependencies and got the latest master and built, then installed mythtv. I then install mariadb-server.

I could not install Ubuntu server with the only user being mythtv, but ubuntu would not allow that. So I create user 'jim' on install and then on first boot created group mythtv and user mythtv, making sure mythtv had a password, was part of the sudo group and was using /home/mythtv as it's home directory.

All work on the backend was then done using ssh mythtv@192.168.0.140 on 2 terminal sessions, one for running the mythbackend command and one for everything else.

after several failed attempts I found that I needed to setup the database before I started with the webapp setup. So from the console I did:

sudo mysql -u root -p
create database mythconverg;
create user 'mythtv'@'%' identified by 'mythtv';
create user 'mythtv'@'localhost' identified by 'mythtv';
set password for 'mythtv'@'%' = password('mythtv');
set password for 'mythtv'@'localhost' = password('mythtv');
connect mythconverg;
grant all privileges on *.* to 'mythtv'@'%' with grant option;
grant all privileges on *.* to 'mythtv'@'localhost' with grant option;
flush privileges;
exit;
mysql_tzinfo_to_sql /usr/share/zoneinfo | sudo mysql -u root -p mysql

At this point I started mythbackend on one terminal and the google-chrome on the workstation to 192.168.0.140:6744 I could following most of the setup based on what I know from mythtv-setup. However, setting up the Input Connections is where I have issues. I first tried setting up two HDHR tuners for EIT scan with the Video source for EIT for over-the-air EPG which works in mythtv-setup. I could not force a scan. I could only import the HDHR channel list, but no luck. I also install xmltv and configured that as I would in mythtv-setup but full scan failed.

The console log on mythbackend showed

2023-05-31 13:19:05.484333 I  HDHRSH[4](1074C841): Added 4 devices from 1074C841
2023-05-31 13:19:05.504108 I  HDHRSH[4](1074C841): Connected to device(1074C841-0)
2023-05-31 13:19:05.576663 I  ChanImport: No new channels to process

The web app showed on the right No freq table for (atsc, , ) found Found 0 Transports
Failed to find any new channels!

This happens immediately.

If I use mythtv-setup to setup the Input Connections and then switch back to the webapp for everything else. It works as expected.

Where did I go wrong?

bennettpeter commented 1 year ago

Thank you for the feedback. I will look into this. The setup is still in testing so there probably are bugs/issues.

jfabernathy commented 1 year ago

I wrapped up my testing for a while and I've create a VM with btrfs so I could snapshot at every stage so I can go back to the start of configuration without having to recompile mythtv or setup xmltv.

At the current time I can use the webapp after setting up mythconverg and starting mythbackend. I can setup through everything except Input connections. That I still have to do with mythtv-setup. I do the channel scan there. Then switch back to webapp for the rest. Some things get reset in the process. After going back into webapp I completed storage directories and edited a few channel, But I could not get the backend to restart and show the update option. I found that somehow the master backend box got unchecked and the IP for the backend got cleared. After fixing that I exited webapp and cntrl-C'ed the mythbackend cmd and restarted it. Mythbackend remained working without errors and webapp now showed the dashboard on startup. I could then use it to schedule a recording.

billmeek commented 1 year ago

On Wednesday 31 May 2023 05:04:18 PM (-05:00), jfabernathy wrote:

I wrapped up my testing for a while and I've create a VM with btrfs so I could snapshot at every stage so I can go back to the start of configuration without having to recompile mythtv or setup xmltv.

At the current time I can use the webapp after setting up mythconverg and starting mythbackend. I can setup through everything except Input connections. That I still have to do with mythtv-setup. I do the channel scan there. Then switch back to webapp for the rest. Some things get reset in the process. After going back into webapp I completed storage directories and edited a few channel, But I could not get the backend to restart and show the update option. I found that somehow the master backend box got unchecked and the IP for the backend got cleared. After fixing that I exited webapp and cntrl-C'ed the mythbackend cmd and restarted it. Mythbackend remained working without errors and webapp now showed the dashboard on startup. I could then use it to schedule a recording.

I don't recall if you have an HDHR. I tested with mine using:

Input Connections Scan for Channels Scan Type HDHomeRun Channel Import

It ran in seconds and just to check, I looked in mythconverg.channel and saw mine.

-- Bill

jfabernathy commented 1 year ago

HDHomeRun CHannel import does find all the channels, but that information never makes it into the guide. xmltv grabber doesn't fill in the data. I tried OTA EIT and tv_grab_zz_sdjon_sqlite. I can get a working system only by using mythtv-setup for the Input Connection part.

I still have issues after doing the setup via the webapp of having it fail because it forgot the master backend checkbox and the IP of the backend.

kmdewaal commented 1 year ago

The Channel Editor of the MythTV Setup section does have the possibility to filter something from Source (presumably from the Video Source) but how this works eludes me. What I would like is to have a Video Source selection similar to what is in mythtv-setup. Here that would then be a dropdown list with first choice "All" and then the names of all video sources. The list of channels is then either all channels from all video sources or all channels of the selected video source. More/better/other filtering is not needed for me. What is also needed is the capability to delete all channels from a video source. This can be done in mythtv-setup in the Channel Editor page and that page can be used here as well, but it might be better to add a button in the Video Sources page; when a video source is expanded there is a "Delete this Video Source" button and you could put next to that a button "Delete all channels" and while you are at it also a button "Delete all transports". The "Delete all channels" should do the same as in mythtv-setup: delete all the channels but leave all the transports. The "Delete all transports" is something that is missing in mythtv-setup but that would be useful. It should delete all channels and all transports of that video source. I can help with the C++ code part if needed.

bennettpeter commented 1 year ago

The way the filter was working is that you type in the first few letters of the source name and it filters on that.

I have changed it now to a drop-down list and added the delete all channels for the source to the channel editor page.

There are filters on the Recordings page that also allow you to type in part of a title or part of a Rec Group to filter. You can click the funnel next to the filter input text to get various filter options, like "contains", "starts with", "equals", etc. that are applied to your input text.

I did not do anything with transports. I can implement the transport delete. The transport edit I also did not do. Is that needed?

Note that the delete channels actually flags them for delete, does not physically delete them.

kmdewaal commented 1 year ago

Peter, you are fast! Will look at it again soon. About editing and deleting transports, on second thought it is maybe better to not implement it. It is rather dangerous and I am probably the only one who would ever use it. The problem with deleting transports is that it also really deletes all associated channels. This breaks the model that channel data will be available as long as there is a recording. To implement it in the same way it would be needed to make also a "mark for delete" for transports, and that is not worth it.

jfabernathy commented 1 year ago

Peter, I've been playing with the latest version you did last night: v34.0~master.202306050718.e2f2f2ac74~ubuntu23.04.1

I delete all channels by selecting the Source "SD" and it marked them for deletion. This was done in the Channel Editor without going into Setup (gear). Not sure how and when they get deleted. The use case is when you move NUC and HDHR to new location and setup your xmltv again with new channels for the new location. In mythtv-setup I would do this from the Channel editor and then rescan all within channel editor.

Here I went into Update backend where it stops the backend and back to the Channel editor there and re-imported from the HDHR . Seemed to work but I could not select a Starting channel, none available. In the 2nd tuner section I could change the stating channel it had a complete list. I Saved and went back to the 1st tuner and Still on starting channel selections available. I moved on the next section, Storage directories and then back to Channel Editor and this time I could see Starting channels for the 1st tuner and I changed and saved. restarting the backend with the button got me working again. This test was with an updated version of the mythtv software and not a rebuild. I'll save that for when we get something more to test. I also want to try a test on a system other than Ubuntu where you have to build from source and setup all the stuff yourself. The web app should really help for that.

rcrdnalor commented 1 year ago

Hi Peter Thank You for providing this amazing web-setup!

I did a pristine Ubuntu 23.04 installation and added the mythbuntu ppa 34. I am using a DVBT-2 USB stick and when doing a channel scan in section

  1. Input Connections it returns immediately without any found channels. The log level of mythbackend is -v general:info,http:info,chanscan:debug The log shows 
    I HTTP6 mythhttpsocket.cpp:276 (Read) 127.0.0.1:60196: Processing: path '/Channel/' file 'StartScan'
I HTTP6 cardutil.cpp:1233 (SetDeliverySystem) CardUtil[2]: Set delivery system: DVB-T2
I HTTP6 mythhttpsocket.cpp:276 (Read) 127.0.0.1:60196: Processing: path '/Channel/' file 'GetScanStatus'
I HTTP6 mythhttpsocket.cpp:276 (Read) 127.0.0.1:60196: Processing: path '/Channel/' file 'GetChannelInfoList'

    within the same second. No entries from the 'chanscan' activities. It looks to me that 'GetScanStatus' returns too early an outdated value.

In contrast, when I start a full channel scan triggered by 'mythtv-setup' and it takes several minutes.

On the other hand, importing a 'm3u' file which I use in my production setup works, but I get on mythfrontend 'device busy' and the log file from mythbackend says

W CoreContext cardutil.cpp:1884 (GetStartChannel) CardUtil[7]: No start channel found on inputid 7
E CoreContext mythbackend_main_helpers.cpp:240 (setupTVs) Problem with capture cards. Card 7 failed init

Any help how to debug this is welcomed. It seems to me, that the services API for channel scan needs to be checked in more detail.

I really appreciate your great work and want to help you to add the last missing bits.

Roland

P.S.: I forgot to mention the installed version: 2:34.0~master.202306080719.1829328718~ubuntu23.04.1

bennettpeter commented 1 year ago

These are difficult for me to debug, as it works on my system. I only have one type of DVB card, ATSC and QAM. This is what I see with current master - it takes a really long time, going through all the channels https://imgur.com/a/yLOXqHg Did you plug the USB stick into the backend before starting mythbackend? The code caches the state of the device on startup and does not find it if you plug it in while mythbackend is running. Can you send a screen shot of what you see. Here is the log I get on the backend. There is lots of junk from the channel scan https://pastebin.com/D3ZqLkuC

kmdewaal commented 1 year ago

Testing DVB-T channel scan. Selecting Scan Type "Full Scan". The help text shows: For DVB-T/T2 and scan type 'Full Scan' select a country to get the correct set of frequencies. This is the help text from mythtv-setup and this indicates the problem later on.

After pressing the "Start Scan" button there appears the message: No freq table for (dvbt, ofdm, ) found This must be because there is no country defined; the country code is used to select the frequency table.

==> Issue: it is not possible to select the country. Correct behavior: create a menu with available countries; the current country as default. For a description see Wiki page https://www.mythtv.org/wiki/Channel_Scanning#Full_Scan

bennettpeter commented 1 year ago

As soon as you select full scan for DVBT or DVBT2 you should see the country list. it checks subtype for ['DVBT','DVBT2'] Can you run this api call in a browser, substitute the cardid for your DVB card: http://backend:6744/Capture/GetCardSubType?CardId=81 I get this result:

<CardSubType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serializerVersion="1.1" version="1.0">
<CardId>81</CardId>
<SubType>DVBT</SubType>
<InputType>DVBT</InputType>
<HDHRdoesDVBC>false</HDHRdoesDVBC>
<HDHRdoesDVB>false</HDHRdoesDVB>
</CardSubType>

This is how it looks when I test it (it doesn't find any channels for Germany since I do not have a signal, but it goes throgh the motions): Screenshot_2023-06-12_14-09-40

kmdewaal commented 1 year ago

This is what it says on my DVB-T/T2 box:

<CardSubType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serializerVersion="1.1" version="1.0">
<CardId>28</CardId>
<SubType>HDHOMERUN</SubType>
<InputType>HDHOMERUN</InputType>
<HDHRdoesDVBC>false</HDHRdoesDVBC>
<HDHRdoesDVB>true</HDHRdoesDVB>
</CardSubType>

This is what it says on my (older) DVB-C box:

<CardSubType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" serializerVersion="1.1" version="1.0">
<CardId>20</CardId>
<SubType>HDHOMERUN</SubType>
<InputType>HDHOMERUN</InputType>
<HDHRdoesDVBC>true</HDHRdoesDVBC>
<HDHRdoesDVB>true</HDHRdoesDVB>
</CardSubType>

Improving identification and detection of capabilities of the HDHomeRuns is also somewhere on my todo list but since it all just works I never got around to it. There are HDHomeRuns that can do both DVB-C and DVB-T/T2 and they should be configured with a Delivery System type of selection just like what is done for the /dev/dvb tuners.

bennettpeter commented 1 year ago

I just pushed a fix for HDHOMERUN. The code was not handling the DOESDVBC and DOESDVB flags correctly. That should be fixed now.

kmdewaal commented 1 year ago

HDHomeRun DVB-T2 "Full Scan" works OK now. Country can be selected and the correct channel list is used.

HDHomeRun DVB-C "Full Scan (Tuned)" does not yet work; messages from mythbackend are:

2023-06-12 22:30:21.693153 I  ChScan: Scan()
2023-06-12 22:30:21.693309 I  ChScan: ScanTransports()
2023-06-12 22:30:21.693349 E  DTVMux: Invalid modulation parameter unknown
2023-06-12 22:30:21.693357 E  ChScan: Failed to handle tune complete.
2023-06-12 22:30:21.693363 E  ChScanWeb: Programmer Error: Failed to handle tune complete.

The message E DTVMux: Invalid modulation parameter unknown is the indication of what is wrong but it requires a bit of debugging to figure out exactly what is going wrong here. I am fairly certain that I entered the correct values for all modulation parameters so I think not everything is propagated correctly to the ParseDVB... functions.

jfabernathy commented 1 year ago

With the version Peter committed his afternoon. I can now do full scan or import from HDHR tuners in the web app. If a user maintains his HDHR favorite via the HDHR web page, then using the import option is the quickest. Full scan gets all the channels but then you have to edit the channel to eliminate the ones you don't really need or get well. I maintain the HDHR tuner web page with only the channels I want the HDHomerun FireTV app to see and that makes the importing method my choice.

I tested this on an Ubuntu server (22.04) adding the ppa:mythbuntu/34 repository. Everything else was done on the web app.

I tested both EIT for over the air ATSC 1.0 USA channels and using xmltv grabber tv_grab_zz_sdjson_sqlite for Schedules Direct EPG. Both work as expected.

I also tested on a Raspberry PI 4 4GB running EndeavourOS (archlinux) with HDHR tuners. Same results.

If needed I could shutdown my production backend for a short while and change the nvme drive and built up an Ubuntu server to test that server's Hauppauge WinTV quadHD PCIe tuner card. Let me know if that is needed. Some coordination will be needed for help.

jfabernathy commented 1 year ago

On a fresh Debian 12 system I discovered that on the Input Connection page after you select Input name, source and full scan, you might need to hit save to avoid no channel found error.

bennettpeter commented 1 year ago

I re-tested this and it looks correct. The process is Select Scan Type: Full Scan Select Frequency table if needed Select channels if needed Click Start Scan It scans and finds channels Then it says "99 MPEG channels found" or similar message You have to select "Import All" and click submit. Then maybe there are more messages about channels found. Import each one, or tell it to ignore them Once that is all finished, then you can select the starting channel. You can't select starting channel until you have answered all the questions.

bennettpeter commented 1 year ago

The message E DTVMux: Invalid modulation parameter unknown is the indication of what is wrong but it requires a bit of debugging to figure out exactly what is going wrong here. I am fairly certain that I entered the correct values for all modulation parameters so I think not everything is propagated correctly to the ParseDVB... functions.

Please tell me what parameters you used and I can check if they are being handled.

If you want to check it yourself, using Chrome,:

jfabernathy commented 1 year ago

So here's what I did in a retest on a RPI4 Archlinux (EOS) system with latest git pull and rebuilt.

EDIT: BTW I'm using Firefox browser for flathub

jfabernathy commented 1 year ago

Just did a re-re-test and same results. I have to click save before Start Scan. I edited a few items in the above post to clean it up and be more accurate.

jfabernathy commented 1 year ago

Screenshot_20230613_101243 This file is a screen shot from Chrome when I didn't do the Save. It gets the 304 code responses forever. If I stop the scan and do a Save then Start the scan again I get a lot of 304 but I get some 200 codes as well.

jfabernathy commented 1 year ago

I went back to the Debian 12 server and re-re-tested. This system uses XMLTV instead of EIT but the results are the same I have to save before I click Start Scan.