Add initial support for multiple Ceton devices

[DanAustinGH]

All streaming features are working the Web UI and status pages currently only show the first device.

[arrmo]

Awesome - thanks! Will give this a try, but first ... any variable name changes (that I need to update in config.ini)?

[DanAustinGH]

The device_tuners is new, and as soon as you asked, I realized Ishould set it to the same as 'tuners' if only one device is configured. I wanted to make tuners, which is already mandatory, a list, but the core needs/expects it to be a single entry. This code passed the test of a single Ceton device configured. I will test what happens if device_tuners is not set.

[DanAustinGH]

Actually, I think I provided a config default, so it should not fail...

[DanAustinGH]

Confirmed that roll over to the 2nd device works. The UI cannot close, but that makes sense. I have not taught it about the tunerstatus dict yet

[arrmo]

OK, seems this is not pushed upstream yet ... just tried to grab it, didn't work ... LOL.

Should I try to pull from your repository for now? Need to figure out how to have 2 upstreams 🤣

[DanAustinGH]

And I missed the default device_tuners. Will push shortly after fixing the close button. Do not feel like you have to run this code yet, but I would super appreciate an extra set of eyes to look for anything obvious I missed or should account for...

It has been awhile since I looked at the UI code, and it does almost do the right thing when clicking close. It does make the correct API call and the code does try to stop the tuner, but I end up with a kernel soft lockup and an unkillable ffmpeg process with the source is the PCI device. Will need to play carefully here. Not likely related to fHDHR, but force rebooting my server did not go well. Dropped multiple disks from my 22TB media volume, which made the server unbootable.

Managed to get enough disks re-add to boot and looks like the data is intact. @arrmo Have you used the web UI to close tuners when streaming from the local ctn device?

[DanAustinGH]

I can push it, but at least the UI is very much not ready. My post above got delayed by 90 minutes while I tried to get my server back online...

[arrmo]

FYI, not dropped or forgotten - but the daughter has the tuner tied up tonight, can't take it down right now. Sorry!

[DanAustinGH]

No rush. I think I may have triggered a long standing kernel bug and it was a coincidence that the last interaction was with fhdhr. I still have 6 hours before my RAID restripe is finished, so I am not going to poke the bear. I'll merge this when you have time to help test.

[arrmo]

OK, fired it up - just had to add to config.ini,

device_tuners = 4
tuners = 4

And it's working! I like what you added to the API status output - nice 👍. And the fHDHR UI seems unchanged ... correct?

My HW transcode still seems to work (for stream_method = ffmpeg), so that's great. But it's broken now for direct? Or did you change that to "hardware" (or something else)? Also, if it's set to direct ... no output in the log file (at least at noob level) ... but perhaps because it bypasses transcoding now? Would like to get that back of course 😄. Also, even with direct, it's showing RTP in the ceton Status page? Just a few minor things ... overall very nice!

I think I may have triggered a long standing kernel bug

OK, that part scares me. Any more details?

Thanks again.

[arrmo]

And just as I hit send, looked at the status API ... LOL! ffmpeg is pulling direct, looked at the URL. Forgot we had changed it to direct only for HW => all good there. But it still now says RTP on the status (page)?

Thanks!

[DanAustinGH]

I'm a Centos user and had to shuffle with 8 going EOL in December. There are reports of some 5.X kernels being soft lock prone on AMD. Looking in logs I think I got super unlucky that the locked up caused an issue in MDADM, which bumped multiple drives out of my array (not directly bug related). It also looks like one of those drives was also a real failure, also at just the right time.

I am on an older kernel, so upgrading may help, but seems a new security feature is blocking that, so I have to boot to an older kernel to then upgrade. I am close to certain that it was just a pile of bad luck at just the right moment.

The UI displays RTP because we are now collecting the the 'Playing' status from the Ceton. I think we can display RTP/HW based on either the streamurl (contains dev or udp) in tunerstatus. So I think we only have UI cosmetics to deal with, but...

[DanAustinGH]

And if you had another device and added 4 more tuners the UI would show all 8 in the UI. I am still uncertain how I want to add the new device info to the UI status page...

[arrmo]

I am on an older kernel, so upgrading may help, but seems a new security feature is blocking that, so I have to boot to an older kernel to then upgrade. I am close to certain that it was just a pile of bad luck at just the right moment.

Sort of hope it's just that - bad luck ... not wishing bad luck on you of course! 🍀. And BTW, I'm on 5.13.0-35-generic #40-Ubuntu.

The UI displays RTP because we are now collecting the the 'Playing' status from the Ceton. I think we can display RTP/HW based on either the streamurl (contains dev or udp) in tunerstatus. So I think we only have UI cosmetics to deal with, but...

Funny - was thinking the same thing ... exactly as you say, use the url. Want me to take that one, or add to your changes? Just let me know, whatever you prefer.

And if you had another device and added 4 more tuners the UI would show all 8 in the UI. I am still uncertain how I want to add the new device info to the UI status page...

Perhaps just two "subsections" in the table ... with a heading row for the tuner "number" (0, 1, ... 17 ...)?

And BTW, it seems that "direct" now really is direct => no transcoding at all, just pulling packets and sending them out, right? I like it! So ffmpeg uses RTP (external) or /dev (internal), transcoding away. Nice 👍.

[arrmo]

And BTW, on the kernel - still struggling to get DKMS to rebuild / install ctn91xx on a kernel upgrade. Arrgh!

[DanAustinGH]

I'm hoping it was my bad luck. Not that I want it, but that is better than the plugin being the cause.

I have a few obligations on top of the not fully recovered server that will keep me from development until Friday or Saturday. I will merge this if you have time to start making UI tweaks.

The subsection idea would work. Likely easier that having dynamically sized columns for the device details and looping to fill those out per device.

Direct should work the way you describe. It did not work that way for me, so I thought I might have the broken and was looking forward to your thoughts. Now I need to poke and test that again. Would be nice if I did not break it....

[arrmo]

I'd say go ahead and merge - folks (myself included) can always pull an older commit, run that if issues 😜

I can take a shot at those items, NP at all. Just to understand ... did you capture a tuner "device" (vs. tuners inside a device) in to some variables? To make it a bit easier.

Thanks!

And do let me know what you find with direct 😄. I think what I said is right, and I don't see ffmpeg firing up.

[DanAustinGH]

Direct does not seem to work for me. It is not throwing any errors, and the core does show chunks downloaded and server, but my VLC client does not receive any packets. I am setting the fhdhr streaming to direct and the ceton streaming_method to direct. More diag to be done...

[DanAustinGH]

And it is working. This might be a quirk with my cable company but there are dozens of media streams in the network 'stream' . I had to manually pick the one media stream that matched my channel (this is in VLC).

[arrmo]

Was just about to mention that - so many streams. How did you select it in VLC BTW? I find sometimes it randomly messes up, but in general works 😆.

And for me - the downside ... watch your lost video frames, over a period of time - and compare to lower bit rate (HW) transcoding if you would. I find more errors in the direct case - thinking something is just slightly overloaded?

[DanAustinGH]

The closest I come to capturing the device is the tunerstatus [ceton_IP] key. I was planning to only use ffmpeg, but documenting what works and what does not might make it easier for anyone else that tries to use the plugin.

I noticed in VLC is under playback there is a 'Program' section with all the programs in the stream. I think we can force that by setting the Program in set_ceton_tuner, will play with that later...

[arrmo]

The closest I come to capturing the device is the tunerstatus [ceton_IP] key. I was planning to only use ffmpeg, but documenting what works and what does not might make it easier for anyone else that tries to use the plugin.

Capturing? You lost me there 😆. I was just meaning above ... watch the statistics if you would, see if you see the same as I do.

I noticed in VLC is under playback there is a 'Program' section with all the programs in the stream. I think we can force that by setting the Program in set_ceton_tuner, will play with that later...

I can explain that one 😜. Check out the Tuner section in the Ceton UI. It shows multiple programs under a tuner (frequency) => multiple channels on a frequency it seems. Need to make sure that the selected on in the UI (output video) matches to what is selected in VLC. Make sense? And good find BTW, thanks!

[DanAustinGH]

It might be multiple programs on a frequency, or something else the Ceton devs planned and did not quite finish. I had seen it before, but for RTP streams it did not seem to matter (maybe ffmpeg filtered it out). Handy for testing, but hopefully Plex/Emby just do the right thing if direct is chosen.

I really should get better at quoting. I was replying to:

did you capture a tuner "device" (vs. tuners inside a device) in to some variables? To make it a bit easier.

I was thinking to add a device{} and then loop over that in the UI, but had not got to it before 'The Event'

[arrmo]

It might be multiple programs on a frequency, or something else the Ceton devs planned and did not quite finish. I had seen it before, but for RTP streams it did not seem to matter (maybe ffmpeg filtered it out). Handy for testing, but hopefully Plex/Emby just do the right thing if direct is chosen.

Yep, exactly! I see this in the log, with ffmpeg running => it selects the program! Just not VLC 😆

Input #0, mpegts, from '/dev/ceton/ctn91xx_mpeg0_0':
  Duration: N/A, start: 77222.599311, bitrate: N/A
  Program 748

I was thinking to add a device{} and then loop over that in the UI, but had not got to it before 'The Event'

No worries! Will dig 👍

[arrmo]

OK, PR created - for the status part of it (will create a separate one for the device row). The changes are pretty simple to fix "Direct", 1) Check /dev in use (first) ... if it is, call it direct. If not, check RTP. Avoids direct + RTP being called RTP. 2) Fix the string used to check for /dev in use

Clear?

Thanks!

[arrmo]

BTW, to the Program item above - it seems it's pretty standard. And yes, multiple programs (streams) on a frequency. A few items I found ... but seems that ffmpeg is detecting the one that is in use, VLC not so much 😆 https://superuser.com/questions/343716/ffmpeg-how-to-demux-live-multi-program-transport-stream https://trac.ffmpeg.org/ticket/995 https://stackoverflow.com/questions/43845382/ffmpeg-retrieve-a-specific-program-from-ts-file

[arrmo]

FYI, https://wiki.videolan.org/VLC_command-line_help/ See ...

   Track settings:
      --program=<integer>        Program
          Choose the program to select by giving its Service ID. Only use this
          option if you want to read a multi-program stream (like DVB streams
          for example).

[DanAustinGH]

I had hoped I was overlooking a setting to auto-detect. I run my server headless, and all media playback is tested on Windows systems. In any case I rarely use VLC, normally just to allow me to launch multiple streams against fHDHR for testing without going through Emby. Having to manually set the program when using direct is slightly annoying, but is better than thinking direct is broken because VLC is not playing any media...

[arrmo]

Agreed! And I'm glad you found it, I was a tad confused ... sometimes it worked, not others. Always good now if I select the Program (and only a VLC thing it seems).