HDHomeRunDVR: Difficulties with audio-only streams (II)

[gibxxi]

Carrying on from the previous (now closed) issue: https://github.com/djp952/pvr.hdhomerundvr/issues/48 and the conversation started here: https://github.com/djp952/pvr.hdhomerundvr/issues/64 ...

We're treating radio stations on DVB-T/T2 as pure audio streams. They're not. They're MPEG2 streams, with video and audio, as I can see when tuning radio channels on my TV (Which has nothing to do with the HDHomeRun box).

https://i.imgur.com/YOme55I.jpg

https://i.imgur.com/x36CAoG.jpg

NOTE: The above images are not just showing static content. The synopsis text changes at regular intervals (at a guess, about once every 5-10 seconds).

The HDHomeRun VIEW Windows App, does not display the above in it's own app, you just get a black screen for radio channels.

It's my belief that the video portion may be being (purposefully) delayed to allow for DAB-based devices that have no video capabilities, so they latch onto the audio portion of the stream, ignoring / dropping the video portion, but that's just supposition on my part.

Here in the UK, digital radio stations are broadcast on multiple standard definition (SD) muxes. There are no radio channels on HD (H.264) muxes. NextPVR lists both TV channels and radio channels as "MPEG2" - i.e. a DVD-style MPEG2 stream with no differentation for either. There must be some kind of "flag" being broadcast signalling that it is a radio station, as the HDHomeRun is able to tell them apart.

Milkdrop 2 takes control of the graphics adapter whenever Kodi is told the source being played is "audio-only", it will not relinquish hold of that device until the stream is stopped. The sound portion of the stream plays for as long as there is buffered data to play, but since the video data has no output (because the graphics card is in use by Milkdrop), the audio cuts out as soon as it gets to a point where the audio will become too out of sync with the video which is not able to be rendered, therefore resulting in a freeze of Kodi as soon as any action is taken that will modify the play state of the current stream from the HDHomeRun. The skin i use shows buffer queue on screen for streamed audio, and it's not running out of buffered data (in as far as audio is concerned).

I have successfully managed to do "channel changes" when this ocurrs using the up / down arrow keys, and by keying in the channel number, e.g. "1" for BBC ONE (TV). The correct channel icon will appear in the music vis window, but no audio (or video) is played, again, for the reasons above stated above in my view.

This is the only explanation that makes logical sense to me (baseed on what I've witnessed). None of the other HDHomeRun PVR addons I've tried have any issue, with Milkdrop being active other than this one, and that may well be down to the fact they use the "VIEW" portion of the HDHomeRun box for playback, and not the "RECORD" portion as this add-on seems to do. Directly streaming from the HDHomeRun (Advanced Setting) also has no effect. The only thing that does is disabling Milkdrop which is a deal-breaker, long term.

The confusing thing would be that it's still the music vis screen that's being displayed, regardless. The skin I'm using has a setting I use to instantly "switch" to the music vis screen on music / audio playback, which might have a bearing, but I've not testing disabling that as yet. And I don't know who (Kodi itself, or this add-on) is making the determination that the stream is "audio-only".

The crux of this post, and henceforth creating another "issue" thread, after our previous conversation, is that I think the only workaround for this is that if the RECORD engine must be the engine of choice for playback, we need to treat all streams it provides as video streams, regardless. How we get to that point if the stream differs in any way from the regular TV channels, I do not know.

What I do know is, constantly having to switch on/off Milkdrop 2 to use PVR / listen to music is far from ideal.

One other thing I've noticed. Tuning channels (TV or radio) can take anywhere from 3 to 5 seconds for anything to appear in Kodi, even on the HTPC which is on the same powerline switch / segment as the HDHomeRun box itself. Another queer fact is that it doesn't ALWAYS happen, just 95% of the time.

Thoughts?

Dan / Gib.

[djp952]

Sorry for the delay in responding; I have to read this a couple times and absorb it, but one thing I can point out now is that I just pass along the MIME type reported by the HDHomeRun system (DVR or tuner). I haven't personally encountered a true audio-only stream, mine here are also MPEG video streams.

But, internally to Kodi if it doesn't find a valid video stream it may be triggering an audio-only scenario. It takes the MIME type I provide as a "hint" of sorts, it still does things on it's own to try and figure out how to best handle things.

Changes I've made can deal with a pure audio stream by not assuming certain things, and if I tell Kodi it's "audio/mpeg" it will pop open the music UI instead of the video UI.

Let me read this through a few times to see if there is anything I might be able to take an action on for you.

[gibxxi]

Hi Mike,

First up, it's not just Milkdrop / Milkdrop 2. I ran a test with the "Vortex" visualization, and the exact same behaviour ocurrs. I therefore expect any visualizers that come bundled with Kodi by default to illicit the same results (as yet untested by me).

Secondly, Considering we're streaming from the RECORD engine, it seems to only output an MPEG2 transport stream, as when initiating recordings, the nature of the TV station doesn't matter. Be it TV or radio, a recorded section will result in a "*.mpg" file on the NAS.

The Radio channels, are shown as "MP2" in the visualization screen for the skin i use, which is essentially correct. The audio "portion" of the stream, on European DAB radio is MPEG1 - Layer 2 (MP2). That however does not rule out the presence of a video portion coming from the HDHomeRun. I just cannot be sure of it's presence at the Kodi side of the link (yet), and / or whether it is coming from OTA data, or perhaps pseudo-generated in post-processing by the HDHomeRun to satisfy the MPEG2 standard.

Neither can I be absolutely sure that what I see on my TV is "video" data per-se or some other kind of stream data displayed as video on-screen.

What I can be sure of, is that the other HDHomeRun PVR addon (without DVR functionality), when using the "VIEW" portion of the HDHomeRun device, does not cause any issues, whereas what we're doing via the "RECORD" engine is causing mayhem.

I will attempt another recording over the weekend, and analyze it with the MediaInfo tool. Maybe that will yeild more info as to how standards-compliant, the mpeg2 transport stream the HDHomeRun RECORD engine outputs, really is.

Dan / Gib.

[djp952]

There is definitely a difference in what the RECORD engine throws out as opposed to the tuner on it's own. I can prove it rewrites some of the TS information, but I'm not nearly astute enough to know exactly why or if that can be problematic. If I had to guess, I would point at the extra packets you get at the head of a RECORD engine stream. There aren't many, and they are standards-compliant, but it could be screwing up ffmpeg somehow.

The observation that the built-in HDHomeRun addon works OK is a good one. It works by just sending the URL to the tuner to Kodi and let's Kodi do all the work. Not that it's a bad thing, just a difference. (Kodi's implementation can't handle seeking on the RECORD streams, which is the primary reason I had to make my own).

Can you get me a recorded .MPG file that exhibits this problem again? I'm about out of TODOs on the PVR right now with the new release pending, I have time to play.

I can even cook you up a custom version of the PVR to help assist you in your diagnoses, we can do things like force the media type a certain way or try screwing around with the PIDs in the stream.

If this can be narrowed down to a TS concern, and if it doesn't require writing an entire demuxer to solve, there is always the possibility the data can be screwed with enough as it's being read to trick Kodi/ffmpeg into doing our bidding. I don't think I have the skillset to do much more than maybe screw with the stream metadata, but if the condition can be detected easily enough, maybe!!

[gibxxi]

Hi Mike,

Will do m8. I will PM you a link via one of the forums to whatever I can conjure up.

The curious thing is that it doesn't always fail. I have had random (albeit limited) occasions where the two do play together, but this is a rarity rather than the norm, which leads me to believing that it's something about the stream and / or how it's delivered from the HDHomeRun box that causes the issue.

It's probably also the case that my kludge of a network setup is also contributing to the issue / confusion. Unfortunately, there's no way I can work around this part of the problem.

I do feel that a solution is within touching distance though. I have the DVR subscription, and I'd like to be able to use it, without having to go back to the days of using 3rd-party middleware like NextPVR like i did when i had my old TV Tuner card.

Watch this space.

Dan / Gib.

[gibxxi]

I took a sample, but the filesize is a bit too big to upload. Needless to say, what I've seen via MediaInfo is not what was expected...

General ID : 4161 (0x1041) Complete name : \**\HDHomeRun\5 Live Sport\5 Live Sport 20060515 [20191102-1200].mpg Format : MPEG-TS File size : 1.38 GiB Duration : 3 h 5 min Overall bit rate mode : Variable Overall bit rate : 1 064 kb/s FileExtension_Invalid : ts m2t m2s m4t m4s tmf ts tp trp ty

Audio ID : 1402 (0x57A) Menu ID : 5632 (0x1600) Format : MPEG Audio Format version : Version 1 Format profile : Layer 2 Codec ID : 3 Duration : 3 h 5 min Bit rate mode : Constant Bit rate : 96.0 kb/s Channel(s) : 1 channel Sampling rate : 48.0 kHz Frame rate : 41.667 FPS (1152 SPF) Compression mode : Lossy Stream size : 128 MiB (9%) Language : English descriptor_tag_extension : 6

Menu ID : 1400 (0x578) Menu ID : 5632 (0x1600) Duration : 3 h 5 min List : 1402 (0x57A) (MPEG Audio, English) / 7210 (0x1C2A) () / 7201 (0x1C21) () / 7270 (0x1C66) () Language : English

I've obfuscated the NAS name, but everything else is untouched. I intend to look for another radio channel that may not have the "Menu" element, like what was displayed in the screenshots for comparision. The Spectrum visualizer seemed to work without issue when tuning up Radio 5 Live today, but will need to do this again after a sizable break to determine it's not conflicting.

I will do some more testing and post back once that's done. It would seem, at this stage that since there is no video element present, that my initial supposition was incorrect. But, the "Menu" element, might be related to the "Red Button" data that gets broadcast alongside the channel, that on TV stations allows for modern TV-text and red-button services (for BBC channels), and that may be causing an issue.

I also need to get a long ethernet cable to connect the switch the HDHomeRun is connected to, directly to the router and rule out the Powerline adapters, but that may be a while down the road, due to lack of available funds at the moment.

EDIT: I also plan to attempt playback of the pre-recorded stream, and see if there's any difference. I've tried altering buffer sizes before, without much difference, but if all-else fails will try again. Currently, all buffering settings are at defaults.

Dan / Gib.

[djp952]

No worries. Since it's an MPEG-TS file you could also just crop it at a reasonable size, it won't affect the outcome (use a multiple of 188 bytes).

Ideally I would like to help compare a recorded stream from that channel against one straight from the tuner. You can grab a stream from the tuner with something like cURL or even wget pretty easily.

Based on your descriptions I think we may find a key difference in the transport stream metadata packets.

On Sun, Nov 3, 2019, 4:23 PM Dan Morgan notifications@github.com wrote:

I took a sample, but the filesize is a bit too big to upload. Needless to say, what I've seen via MediaInfo is not what was expected...

*General ID : 4161 (0x1041) Complete name : ******\HDHomeRun\5 Live Sport\5 Live Sport 20060515 [20191102-1200].mpg Format : MPEG-TS File size : 1.38 GiB Duration : 3 h 5 min Overall bit rate mode : Variable Overall bit rate : 1 064 kb/s FileExtension_Invalid : ts m2t m2s m4t m4s tmf ts tp trp ty

Audio ID : 1402 (0x57A) Menu ID : 5632 (0x1600) Format : MPEG Audio Format version : Version 1 Format profile : Layer 2 Codec ID : 3 Duration : 3 h 5 min Bit rate mode : Constant Bit rate : 96.0 kb/s Channel(s) : 1 channel Sampling rate : 48.0 kHz Frame rate : 41.667 FPS (1152 SPF) Compression mode : Lossy Stream size : 128 MiB (9%) Language : English descriptor_tag_extension : 6

Menu ID : 1400 (0x578) Menu ID : 5632 (0x1600) Duration : 3 h 5 min List : 1402 (0x57A) (MPEG Audio, English) / 7210 (0x1C2A) () / 7201 (0x1C21) () / 7270 (0x1C66) () Language : English*

I've obfuscated the NAS name, but everything else is untouched. I intend to look for another radio channel that may not have the "Menu" element, like what was displayed in the screenshots for comparision. The Spectrum visualizer seemed to work without issue when tuning up radio 5 Live today, but will need to do this again after a sizable break to determine it's not conflicting.

I will do some more testing and post back once that's done. It would seem, at this stage that since there is no video element present, that my initial supposition was incorrect. But, the "Menu" element, might be related to the "Red Button" data that gets broadcast alongside the channel, that on TV stations allows for modern TV-text and red-button services (for BBC channels), and that may be causing an issue.

I also need to get a long ethernet cable to connect the switch the HDHomeRun is connected to, directly to the router and rule out the Powerline adapters, but that may be a while down the road, due to lack of available funds at the moment.

Dan / Gib.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/djp952/pvr.hdhomerundvr/issues/65?email_source=notifications&email_token=AAFMMB7ISDPKFIWTKWZPM5DQR46MTA5CNFSM4JFM43WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC55FJY#issuecomment-549180071, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFMMB6HRBKZSKUGLSMGSNTQR46MTANCNFSM4JFM43WA .

[gibxxi]

Well I feel like an idiot Mike. I should of tried this before, but here goes:

Tried playing back the recorded stream via Video -> Files section: Success. Tried playing back the recorded stream via PVR -> "Recordings": Success.

Just to be sure, tried another internet-based audio stream (Digitally Imported via the AudioAddict Plugin): Success.

So I decided to do a last ditch change... I modified both buffers for the Zuki HDHomeRun plugin to maximum permissable values (Read buffer: 32Kb, Ring Buffer: 16MB), and guess what?

Success!

I did a restart of Kodi as I've had false starts like this before, whereby It's OK on one run, not on the next. Again, success.

Early days yet, but based on what I'm seeing, Milkdrop must be running out of audio to process with default settings, thus causing a stream freeze at the beginning.

Another thing I've noticed is that at default buffer levels, starting the stream with no visualizer enabled, then enabling Milkdrop 2 via the OSD after the fact also works.

Milkdrop does cause a micro-pause on start as it loads any and all presets from it's presets folder prior to selecting which one to start with, but at this stage it looks like a simple buffering issue.

I will monitor it over the coming days, but since "maxing out" the buffer settings in the addon doesn't seem to cause any ill-effects elsewhere (yet), I'm inclined to leave it as is.

Dan / Gib.

[djp952]

Awesome! Are you on Krypton or Leia? The setting should be called "DVR stream read chunk size" on Leia.

A little background for these settings to assist you:

• The stream ring buffer size should be pretty useless in and of itself, if you want you can probably turn that back down. This dictates how big of a "past data" buffer I keep to try and optimize small seeks. The buffer size has no bearing on the speed at which new data can be delivered. But bigger doesn't hurt anything either!

• The DVR Stream chunk size is, I bet, the secret sauce. This behaves differently on Leia than it did on Krypton. On Krypton, Kodi always asked for 4K at a time, this setting was a tuning parameter I had put in for whatever reason, but (until now) it wasn't useful to set it. On Leia, Kodi now asks the PVR how much data it should ask for at a time, so now this setting controls that. This works internally by setting up the minimum amount of data that has to be read before returning anything to Kodi.

So here's what I think happened when you bumped it up to 32K (again, kudos!). Instead of the default 4K of data being sent back to Kodi for the initial stream read, it got back 32K. There was probably enough TS data in the 32K for it to actually figure out what it was doing, as opposed to making a bad guess. There is a very very short timeout Kodi uses during this "what is it" phase, so I'm guessing the timer doesn't start until after the first packet is delivered, so viola - you fixed it!

I really didn't expect that setting to be of any value to anyone, but I'm very glad I left it there.

I'm thinking it may be a good idea to sacrifice a few milliseconds of "start stream" performance to ensure that more than 4K of data is ready for Kodi anytime a new stream is started. I'll do some math, but I think we can go notably higher than 32K, perhaps 128K or 256K. That's not a lot of data and I don't think anyone would notice.

Such a thing may also help with the momentary audio drop that happens when starting a stream (it's not just you, I get that a lot too).

LMK how things keep going on your end, I will play with this on my end. My preference would be to apply some effective change so that people can leave the defaults in place and still have things work right.

YOU DA MAN!!!

[djp952]

Did some tinkering, you may have found the best way to solve this, it doesn't look like a code change will help.

Kodi only asks me for the chunk size once when starting a stream so I can't do something dynamic like make it bigger the first time through. That said, I can force the stream to buffer a certain amount before indicating that the stream is "started", which really isn't doing much of anything. Regardless of how much data I have in there, Kodi still only asks for the "chunk size" as specified in the settings.

Based on my experimentation, there is no measurable benefit to buffering this additional data during startup, the total time to start a stream and complete the initial read is the same -- it just moves those milliseconds from one place to another. The key seems to be what you have uncovered -- making sure Kodi has enough data up front to make a "good choice" when it analyzes the stream.

If you don't mind, can you see if you can still resolve the problem with increasingly smaller chunk size values down to 8K? Obviously 4K isn't enough, and I think 32K is too high to be a default for most purposes, but if 8K solves it I can change the default. If you find 8K does not, but 16K does, I would like to send you a version that allows for a 12K value to see if that works. My reasoning for a 12K test is that a 96kbps audio stream would take 1 second to stream that much data, which seems like a good breaking point to hit. I don't think Video streams would be affected at all, the bitrate is plenty high enough that a change from 4K to 12K shouldn't be noticeable for most people.

Thank you again Dan for all your research. It's very hard to help without having direct access to the streams you do, and I am very thankful for the time and effort you have put into helping make this work as well as it possibly can within the confines of the Kodi world!

[gibxxi]

Hi Mike,

I've only briefly scanned your two replies at this point, as I'm off to work shortly, but one thing I wanted to say, from my experiences with Milkdrop / 2 in both Winamp and Kodi, is that it displays it's visualization slightly in retrospect. i.e: It needs a decent sized "back-buffer" of audio data to process to display itself.

This was my thinking after seeing other forms of playback (as mentioned above) work without issue. I realised that all other playback attempts would probably be using Kodi's own buffering system to play, whereby there would be enough data for Milkdrop to work.

Vortex doesn't work at all for me, on any audio sources. It doesn't crash, but there's simply nothing displayed. Since it was ported from XBOX, there may be an issue there, but that's not your addon's fault. Secondly, I don't use it anyway, so I can live without that.

I will read your replies properly, tonight, when I get home, but I would suggest leaving this issue open until I can post a proper response, and maybe up till the weekend, so I can do some more real-world testing. There are a couple of shows I plan to listen to over the next couple of days, where I can see if this is resolved properly. I've had false dawns before, so don't want to announce a glorious success only to be back at square one again.

I'll post another reply later. I'm on Kodi Leia 18.4, but the settings are still named as they always were for me. I might have to reset / remove the settings file and set it up again to see the settings name changes you mention.

Dan / Gib.

[djp952]

I did think of something I can do to help in code; I can strip out that useless metadata the RECORD engine always sends when a stream starts. I never bothered since it was innocuous, but it should be pretty easy overall. This would send more valid data to Kodi on the first read, which would increase it's chances of getting the packets it wants. I forget exactly how big this metadata is, but I think it's around 1K, which is pretty significant -- 25% of the first chunk (using the 4K default) is garbage.

Oh, and the strings for the settings are actually in a "strings.po" file, the settings.xml has the internal names only, replacing that would just cause you to lose your settings. Cosmetic regardless, the setting still does the right things no matter what it says it's name is ...

[gibxxi]

Hi Mike,

I did as you suggested and reset the "DVR Ring Buffer Size" back to 1MB, and tried 8KB on the "DVR Read Chunk Size", no go on that one. As with 4KB, you get about 1/4 to 1/2 second of audio, before the audio stops. Kodi isn't "frozen" at this point, and Milkdrop 2 is still active.

What causes the eventual freeze is any action that will force Milkdrop 2 to attempt to load a diffferent preset (e.g: Exiting Vis screen back to main menu, or attempting to stop the stream).

Set to 16KB seems to work as with my initial setting of 32KB. I'm running the Zuki HDHomeRun add-on from your own repo, so any updates to that branch will automatically be applied to my systems here on each Kodi startup.

Another thing i've noticed that can cause issues, is the "commoncache.db" database not being clear prior to tuning or playback, and I think this might of be partially responsible for the weird timing issues we saw on previous versions too. I no longer get those time discrepencies btw, but if Kodi crashes for some reason, due to issues unrelated to this add-on, and subsequently, the commoncache.db file is not empty, I've seen hangs due to that too. It's simply a case of deleting the file, and allowing Kodi to recreate it on launch.

Carry on and send me your 12KB mod when ready and i'll try that.

Dan / Gib.

P.S: Don't worry about the string labels, that was just my P.Poor memory lol.

[gibxxi]

I stand corrected, it's just hung again after exiting and re-launching a new Kodi session @16KB. This was the behaviour I saw last time I messed with the buffer settings. I'll try going up one step (again).

[gibxxi]

It's looking like 32KB is the minimum that will work consistently with Zuki and MilkDrop 2.

I notice that (since Kodi is set to switch instantly to Music Vis Screen on playback start), the "Buffer Progress Bar" displayed in this skin is only about 1/3rd to 1/2 full at stream start. It goes to 3/4 full after about 500ms (gradually) and floats about 3/4 to 4/5 for one or two minutes before eventually going full.

Please bear in mind that this might be down to network latency, Powerline QoS, etc. But as it stands, 32KB is the only setting that works consistently across Kodi sessions on my setup at this time.

Acronis backups ocurring might well throw a spanner in the works, as even though I have it's resource / network usage for all backup jobs set to "minimum / low", this will only have an effect on the PC the backup task is running on at the time, not overall network saturation. However, I can work around this issue if / when it occurs.

EDIT: For the reasons outlined above, i wouldn't nessecarily say that my experiences should be used as a benchmark to base this add-ons defaults on. My setup here (borne out of necessity rather than desire), is far from "mainstream" by any means. At least now, I know what works and what doesn't, and potentially why. I can work around this issue now (potentially).

Dan / Gib.

[djp952]

The powerline adapter is my main concern with such a large chunk size. The small chunk size is used to combat intermittent latency problems. The bigger you make it, the longer the PVR will wait to get that much data before sending anything back to Kodi, which could cause drop outs and buffering. Even on a fast network if I set that value super high that ultimately happens.

I looked into how Kodi is using this chunk size value, it turns out that it doesn't appear to be that important to it. It's primary function is to tell ffmpeg how big of a buffer to allocate.

Here is what I am going to try tonight here, I'm going to tell Kodi to always expect a 32K chunk, but send less (the configured chunk size) if that much isn't available yet. My expectation is that there will be zero impact to anything at all. If that turns out to be the case, I will then have the ability to wait for the first chunk to be a full 32K before it goes across, which I'm 90% certain will still solve your problem. If that pans out, then there would be 2 settings: the existing chunk size one and a new "initial chunk size" one that I can default higher without impacting startup performance at all. The default for that still will be too low for you (probably 8K), so you'd need to change it to 32K in your situation.

So, if that all works out and my assumptions are correct, we can solve your problem without the potential for momentary latency issues causing you new problems.

Also, if you can split that file you have into something reasonable, I would like to look at what TS packets might not be coming across until we hit that magic 32K boundary. It could still be something that SiliconDust and/or Kodi may want to address. I think 50MB would be a good size? Long enough to play back for a while, and certainly more than enough to examine the first 100K or so. Heck, if you can even get me the first 128K of the file I can work with it.

I'm going to find the old sample you sent me from "Live 5", I know I have it here somewhere, but I recall that particular one ended up working fine in the end. It would be a good comparison point, though.

BTW - This is fun, thank you. I greatly appreciate all of the detective work you're putting in to help me try to help you sir!

[djp952]

Hi Dan, I may need to extend an apology, after taking a look at how Krypton and Leia differ, this could all be the result of a mistake on my part. Krypton always used exactly a (you guessed it) 32K buffer size for ffmpeg for PVR addons. Leia made this configurable so ... I configured it.

I give it a 50/50 chance of success, but can you try this build? (2.4.1~alpha1) : https://1drv.ms/u/s!AgEGEEVzGNq-jINiovxGhs_WqdpjSg?e=LxYNbS

The difference here is that I am telling Kodi to always use 32K for the buffer size, but still enforcing the same minimum read chunk size, which should realign things with how Krypton worked. To test you will need to change the PVR setting back to 4K.

If this works, all I need to do is rename the setting back to what it was (and what you see, apparently). If it doesn't work, I would like to follow up with a build that adds the proposed "first chunk of data" being 32K then falling back to the setting.

I hope it works, even if it means I completely screwed the proverbial pooch :)

[gibxxi]

Hi Mike,

Downloaded and gave 2.4.1 - Alpha1 a quick try this morning. Unfortunately no joy.

The audio plays for longer, or at least it did with this test, at about 5 seconds or thereabouts. But the instant Kodi switched from EPG to Music Vis (factoring in the Milkdrop startup delay), I lose audio and we're back to the same scenario as before.

No more time this morning.

EDIT: I've tried all the settings up to 32KB Mike, and the only difference between this version, and 2.4.0 is that the stream plays for a bit longer before dying. 32KB is still the only setting that seems to work consistently.

Here's another sample for you: https://onedrive.live.com/?id=7CAC01C8D93B566E%21254&cid=7CAC01C8D93B566E

Dan / Gib.

[djp952]

Thanks for the sample, will queue it up tonight. Based on this feedback and your test I think we may have reached the limit of what can be done in the PVR and it's time to go looking into Kodi again. Ugh.

The stream lasting a little longer at 4K makes sense, the change allows the PVR to return more than 4K if it has it available, with a cap at 32K. The stream is being buffered in the background while Kodi starts things up, so there is usually going to be a handful of extra bits available at that time, but those extra bits run out quickly. I think the change itself is a good one to keep in place, it allows Kodi to receive more data more quickly when the PVR buffer happens to have a surplus.

For now it seems the only way to get this to work for you is to ensure we always send Kodi lots of data, even if we have to wait for it. I don't think adding something to buffer the first 32K and then falling back to smaller chunks would help, but am willing to cook that up if you think it might.

Let me see if I can figure out what Kodi is asking for and why it's unhappy if it doesn't get these very large chunks of data every time.

I think you're stuck with the minimum 32K chunk/read size setting for the time being. Hopefully it won't cause drop outs or buffering on you and keep you fully functional until we can figure out the next step to try or finger to point :)

[djp952]

Just to make sure I'm looking at the correct issue, playback technically "continues", as in the progress bar moves along, but the audio itself ceases to exist? If that is what happens, I was able to duplicate it by making things take longer than they should artificially; this gives me something to work with - thank you!

[djp952]

There is definitely some wacky stuff happening here with that sample, but so far they seem to be seek related. I don't see Milkdrop (or 2) causing it specifically, I can replicate it easily enough by trying to "resume from xx:xx" on Windows or by seeking at all from a Wifi connected Android device. In these cases, Kodi is asking to seek to very near the end of the file, then reading whatever data is left, and aborting completely because the end was reached.

More as I have it.

As a side note, I've remapped a Recorded TV category of 'audio' to 'series' like I did with 'news', so next version you shouldn't see an 'audio' folder anymore. I wish SD would stop changing these things; even in their own application you won't see "news", "special", or "audio" recordings anywhere but "All Recordings".

[gibxxi]

Hi Mike,

the buffer progress bar is still going up and down at about 3/4 after audio is lost, and Milkdrop 2 is still "active", (not frozen). I can't be certain if what's being displayed is as a response to the sttream, because Milkdrop will show itself even with a silent track. All activity freezes once a preset change is forced, which usually happens at a window change, or when playback is stopped. It doesn't freeze at normal transitions between presets in the same window (i.e: Milkdrop auto preset changes).

Hope that makes sense.

Dan / Gib.

[djp952]

Well, I realize it's not much of an answer but everything I'm seeing so far is inside of Kodi. I'm having extreme problems with it trying to seek beyond the end of the stream repeatedly, which ultimately causes it to die. It appears to be related to either an issue with calculating timestamps and subsequently trying very hard to seek into a place that just doesn't exist, or has to do with the lack of a video component on the stream.

As stated, it isn't much, but digging into ffmpeg is extremely difficult.

[gibxxi]

Another thing I've noticed, is that even at 32KB read buffer, it's not possible to do a direct channel change between one radio station and another, as the stream also dies in the same way as before. This and the fact there is one station (TalkSPORT) that broadcasts as 64KB/s, which I can't tune at all, even at 32KB buffer, as the stream dies exactly the same as at the 4KB setting.

Since I run my setups in portable mode (to keep Kodi's clutter off my SSDs), I can start a vanilla install fairly easily (which I also tried - no change). But I can do some testing in this way with debug logging on, where the only errors shown will be either Kodi or the addon. I may give this a go over the weekend, and send you the findings.

I don't run any dodgy addons, but I do have a couple of addons, including the skin I currently use, that throw non-critical errors that will add to the confusion in the log (log spam).

Dan / Gib.

[djp952]

I don't think it's any addons, so far it has something to do with how Kodi/ffmpeg is generating the timestamps for the stream. Unfortunately that math is buried in non-debuggable code, so I'm building Kodi 18.4 from scratch with ffmpeg debugging enabled to dig deeper.

Info as it becomes available. ffmpeg spelunking is not an enjoyable process, but it has borne fruit for us before. This may even be a side-effect of the changes we successfully got into Kodi last time; it's looking suspect.

[gibxxi]

Hi Mike,

Disregard my previous comment about 64K streams. I was unaware at the time that an Acronis backup was in progress on the HTPC. It seems any buffer underrun is enough to cause the issue to appear.

It's my belief that the PVR code in Kodi needs an overhaul. It's optimised for the old style of PVR backend, where the hardware separates analogue "TV" and "Radio" channels into different categories / streams, where TV channels are audio / video streams and Radio are pure audio-only streams. This isn't the case with modern digital TV tuner hardware, where everything is an MPEG2 transport stream. I don't think Kodi's PVR module is optimized to deal with that, any FFMPEG issues aside.

Sorry m8, I've not had time to do the additional testing I had planned to do over the weekend, hopefully I'll be able to do some testing this week. I think there may be a specific "FFMPEG" setting in component-specific debug logging, in Kodi. If so, when I do get around to doing some additional testing, i'll enable that, and send you whatever I find.

[djp952]

No worries. Getting a fully debuggable build in place helped a lot. I'm 75% sure I know what the problem is. I now believe the problem is the stream data itself. 80% of the data is junk (program 7201). Since the primary stream is low bitrate, ffmpeg is actually unable to find the PTS (timestamp) it wants within a reasonable number of tries, so it's giving up. This is why the large chunk size helps, it's getting more of program 1302 each time, so it can latch onto it.

I demuxed out program 7201 and will try that .MPG as-is later tonight. FWIW, the file went from 272MB to 55MB without the junk data in it. It plays fine and seeks fine in non-Kodi players like this, so we'll see.

[gibxxi]

Hi Mike,

OK m8. There is a lot of ADS (Alternate data streams) broadcast alongside the actual program data. OTA EPG data (ironic that the HDHomeRun can't make use of this and instead charges you for a SchedulesDirect Subscription, which is far less complete) and TV Text / Red Button / menu data as per the screenshot I previously thought was "video" data.

at 32K, i did experience a micro-stutter at the beginning of the stream, but it only lasted for a split-second, and never more than once or twice while the buffer was low. Once the buffer catches up / fills, no issues with stuttering or buffer dropouts.

There's no issue with throughput on my Powerline adapters once a successful playback of PVR or any other data over the network from my NASes actually starts. It's just that getting a stream actually started can occasionally take a while if there's network saturation. For example, late last night, when the HDHomeRun was recording two seperate channels to one (or both) of the NASes, I'd tried to queue up a music album to play. It took about 5 minutes for Kodi to become responsive and begin playback, because of network saturation. But once started, no issues thereafter.

I hate the QoS on the things, and have found the least problematic way of getting around it is to set it to one of the 4 modes it offers that i'll never use, in my case VoIP (I have no VoIP hardware), because then it won't block everything else to prioritise one type of traffic. Also, TP-Link hsve a nasty habit of releasing hardware then abandoning it firmware-wise, in preference to just pushing out a new hardware version. I'll never buy anything by them again.

[gibxxi]

If you have something you want me to test prior to me doing my own testing, let me know and i'll give it a go.

[djp952]

I believe I found the cause and a solution for the main problem I'm seeing here. It is related to the fix we made in Kodi the first time around. My first attempt to quick-fix it failed yesterday but will take another crack at it. It seems like it needs to be more invasive than I was hoping.

What I found was that pretty much all seek operations on an audio-only stream fail to an extent. They can recover most of the time, but I can get certain operations to fail 100% of the time. What's happening is that my PR to Kodi added stuff to say "if there is no video, use audio to set things up". Works OK when you are starting a new stream, but seeking requires Kodi to find the timestamp it wants. My PR didn't take that into account properly, so it's trying to find an infinitely large timestamp, which it never finds. Depending on the operation, it either takes way too long to figure that out, or if you happen to be looking for the end of the stream, will fail outright and stop since the PVR keeps saying "I have no more data for you".

The problem I'm having fixing it is that there is all kinds of metadata stored for Video streams that don't apply for audio-only streams so I can't use that to fix things. Since Video always has an Audio stream with it, I can't just hack it for "IF AUDIO", that's always true.

I may be able to get you a new kodi.exe to try (18.4 Leia) if I can hobble something together, but forming it into a PR that Kodi may accept will take time and I assume will get kicked back more than once.

I agree with your thought patterns, audio-only MPEG-TS is just plain problematic for Kodi. It's pretty clear after digging in that the code was written to always expect both a video and an audio stream to use. How this is affecting the visualization is unknown to me right now, since I can't seem to make that fail, but I'll keep going and see where we end up!

FWIW, I'm still not seeing anything wrong with the PVR module at least. It seems to be doing exactly what it should, it's just not being asked to do the right things ....

(Can we rename this problem to "Dan and Mike's White Whale"?)

[gibxxi]

Hi Mike,

Sounds like you are on the mark with your findings so far. Back in the analogue days this issue wouldn't been an issue because radio was only broadcast on FM/AM and tuner cards were designed to handle that, and Kodi's PVR module (which itself spent a long time in development) was initially coded around that kind of setup.

I have no doubt the devs have added modifications to the code to cater for digital TV setups, but I don't think they've probably modernised it as much as is required. Since now analogue TV is no longer a thing, pretty much worldwide, a complete rewrite of the PVR module would probably be the order of the day.

Having said that, much like the music library, which recently got an overhaul, because there was a dev with an interest in doing so, having the PVR module modernised, when a lot of the core functionality is provided by the PVR backend, and in absence of a developer with "interest" in doing the work, I won't hold my breath waiting for it to come to pass.

I feel my setup highlights the problem, because there's not an issue with throughput for me, at least not once a stream ramps up to full speed. It's the high latency of the Powerline adapters that screws everything up. Also, anything that interrupts that throughput, also causes a major issue. Someone with a dedicated, fully-wired network would probably be totally oblivious to the issues I experience on Powerline AV.

For me, Wireless is not a solution, as most of my devices that even have wireless adapters are only Wireless-N 150/300Mbps, and even on 5GHz channels, the transfer rate is pretty poor compared to Powerline AV, even with it's inherent flaws. For me transfer rate is the most important factor, and Powerline does have an advantage in that the sync rate is far more stable than Wireless, and much higher to boot (900Mbps+).

If it comes to pass that the 32K workaround is the only solution for the near-term, then i'll have to live with that. Although far from "ideal" per-se, it does work (with caveats), so I could live with it.

Dan / Gib.

[djp952]

I have what I think is a reasonable fix for the primary issue I've seen here, if you want to try it. The link below contains a kodi.exe, Windows x64, built from the formal Leia-18.4 source:

https://1drv.ms/u/s!AgEGEEVzGNq-jIUAK4gCc4AWiV0Smg?e=lQ7hRv

This corrects the problems with Kodi being unable to find the timestamps for operations like "Resume from XX:XX". How it may affect visualization (good/bad) remains to be seen. I do know it doesn't improve the seek speed for audio-only streams, it will still thrash about a bit, but should ultimately lock in.

Let me know, I give this a low probability of solving your concern(s), but I'm going to submit it to Kodi as-is for Matrix and request back-port to Leia.

I will, of course, keep going. The PR for Kodi won't take me that long, just need to build Matrix and test it before submitting.

[djp952]

Proposed solution for first problem has been submitted: https://github.com/xbmc/xbmc/pull/16910.

I fear the only true solution to the continuing audio-only problems will be to bite the bullet and create/use a custom demuxer, but let's hope it doesn't come to that :)

[gibxxi]

Hi Mike,

had some initial success on the Desktop PC last night, but that didn't bear fruit on the HTPC or Laptop. Tonight, the Desktop PC refuses to play ball again. I even tried using a few ethernet couplers to run a cable between the HTPC and the router (Thus bypassing the Powerline adapter), this also had no effect.

Here's a Verbose log file from the failed run tonight on the HTPC.

https://1drv.ms/u/s!Am5WO9nIAax8gblDz_gqtBrcMg_B0A?e=n1ls3Y

Setup: Zuki HDHomeRun 2.4.1, Source built Kodi 18.4.

I had to install the Visual C++ RunTime Redistributable 2015-2019 to get your Kodi build to run on the HTPC/Laptop. It ran 1st time on the Desktop (Missing VCRUNTIME140_1.dll).

EDIT: Details of my (OCD-Annotated) advancedsettings.xml and usernames have been removed / obfuscated. Everything else is untouched.

Dan / Gib.

[gibxxi]

...Next thing to potentially try (with the stock Kodi 18.4) is mess about with the QoS settings of the Powerline adapter(s), and check the settings of the router (Archer VR2800) as I'm pretty sure it has DDoS protection, but it also seems to (occasionally) add local clients to a 5 min ban list if they send too much data. I don't think this is the reason for the latency / throughput dropouts, but will check to rule it out, regardless.

The Desktop / NAS-1 are not connected to the network via the Powerline adapter(s). The both use 802.3ad (LACP) teamed ethernet connections (Desktop - 2, NAS - 4) to a Netgear GS208Tv2 managed switch, which in turn then connects to the router. The powerline adapter at the desk, also connects via the router. Each powerline adapter has 2 ethernet PHY adapter ports. The HTPC and Laptop each connect to one at their respective locations, and the (unmanaged) switches containing the HDHomeRun, Amazon TV box, amplifier, TV and NAS-2 then connect to those switches. This means that the HTPC / Laptop, in theory should get (at least) half the available bandwidth available to the Powerline adapter they are connected to.

https://www.tp-link.com/uk/home-networking/powerline/tl-pa9020p-kit/

Dan / Gib.

[djp952]

The runtime thing is my bad, I forgot that I was using Visual Studio 2019 in native mode. Oops. Glad you were able to resolve it.

I'll check out the log (it's immense! lol), but I'm as usual having a bit of a hard time with Kodi and the PR, not that it helped anyway.

At this point I'm starting to look into just demuxing the streams on my own. That's a pretty big undertaking unless I want to deal with ffmpeg (I don't) and may take a very long time. But, it's seemingly the only way I can take control of this problem. I'm sure something specific for MPEG-TS instead of the generic "DVD" code in Kodi will ultimately turn out to be super fast, too. It's just ... daunting. There are some less complicated open source projects out there that I am researching before going the ffmpeg route.

[gibxxi]

Hi Mike,

OK. I didn't bother sending you another stream, as I doubt it would differ much from those I've already given you. It is possible to start a recording from the OSD menu, even once the audio stream is lost, as Kodi hasn't hung at that point. Like I said earlier, it only hangs on a screen change, or any change that will affect the Milkdrop 2 side of the operation.

On screen, the buffer is still fluctuating, and the time-counter still increasing as per normal. But the instant you either use [Backspace] or the [Backslash] (switch from fullscreen to windowed mode) and/or playback keys, insta-freeze and it's Task-Manager - Kill time.

I'll wait on you from this point.

Dan / Gib.

[djp952]

I'm on that one too, it's Milkdrop that's crashing, but it may be trying to access something from somewhere else, not necessarily a Milkdrop bug. Once I get the PR sorted out I'll get that code and see why that's crashing too. It has something to do with the settings .... I have a nice clue to work from when I get there:

SYMBOL_NAME: visualization_milkdrop!ADDON_SetSetting+8179 MODULE_NAME: visualization_milkdrop IMAGE_NAME: visualization.milkdrop.dll STACK_COMMAND: ~0s ; .ecxr ; kb FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_visualization.milkdrop.dll!ADDON_SetSetting

[gibxxi]

Hi Mike,

I wish I could be of more help in diagnosing the issues here, but sadly it's been about 25 years since I last touched a line of code, and I was never a brilliant coder, even back then.

I'm quite happy to wait it out, and I want you to know I'm not going to put any pressure on you to fix this. I simply hate unsolved mysteries / conundrums.

As it stands, I can "work around" the problem, as we've already proven. So until a more permanent fix comes about, i'm happy to wait it out. I will say though, I don't envy the task ahead of you.

EDIT: Milkdrop 2 doesn't have (and never has had) any configurable settings in Kodi, so maybe that's a bogus error.

;)

Dan / Gib.

[djp952]

I found the overall problem with Milkdrop2, but the root cause is elusive at the moment. What is happening here is that Kodi is releasing a buffer and never reallocating it. I have ideas why but it will take more time to figure out if any of those ideas are correct. That buffer is very frequently released/reallocated so I need to pin down the action that causes the release with no reallocation to occur. As has been usual of late it's not a PVR bug/defect, it's a Kodi thing again. If Kodi asks for more data the PVR will happily hand it over, it just stops asking due to whatever is happening here.

I am so looking forward to another Pull Request against Kodi for this -- NOT. :)

[gibxxi]

Hi Mike,

Milkdrop 2 changes the running preset, essentially "restarting" itself, each time there's a screen / window change of any description. Since I have my current skin set to transition to the "Music Vis screen" as soon as audio playback starts, It will start, load presets, then repeat in very short amount of time due to this transition. I do not know if the presets are loaded again when this ocurrs.

However, at the same time, the same thing happens with other types of audio playback, including internet streaming, and local music playback, but this does not cause the crash. So there must be something in the PVR code in Kodi that's acting differently than other audio playback types.

It's not the skin. It crashes just as much in Estuary, as it does in the Beta skin I'm currently using / testing.

Dan / Gib.

[djp952]

Correct, it's definitely not the skin and it's not even Milkdrop 2 specifically. I can get it to happen if any visualization is enabled. I need to find out what's different about the audio "engine" when there is a visualization present. You can make it happen by just pausing the stream and waiting a few seconds, you don't need to force the visualization to reload. That action in and of itself doesn't appear to be related but it's still on the list.

What I see is that when the condition occurs bad things start to happen inside Kodi which can lead to it crashing since it's just sitting in an infinite loop. Not every time for me, and when you run through a debugger things tend to behave a little different :)

Milkdrop 1 just flat out has a bug in it, I've stopped using it for testing.

Just slow going.

[djp952]

Ha. Found it.

What's happening is that when the stream stalls out or has a sync error, the visualization is being destroyed, but it's not being unregistered from the audio player (ActiveAE). The audio player still thinks it has a visualization and calls into a deleted object and poof. This is why it only happens when you have a visualization enabled. Locating point to fix it, will pass along a kodi.exe when found and if it has the desired effect.

[gibxxi]

Good catch m8!

I look forward to testing whatever you provide. Hopefully we are within touching distance of a solution to this annoying issue.

Will probably have to spread any testing over a few sessions / days if it's an initial success to be sure. Well done in any case!

:)

Dan / Gib.

[djp952]

Don't be too impressed, I can't fix it properly. I have no idea how it "should" behave so I'm going to open an Issue against Kodi instead for this.

But, I have a quick-fix hack in place that should bypass the issue and prevent the lock-up/crash from happening. I'm cleaning out the build folders and will cut a Release build (this time from VS 2017 instead of 2019 - sorry about that!) and share it.

I built a couple visualizations from source and was a bit wrong :) What is really and truly happening with 100% certainty now is that the visualization is entering an infinite loop because it's being asked to process zero bytes of data, and there is no protection against that. They (Kodi) may want to fix the visualization(s) to deal with this, or they may want to quick-fix hack it like I did, or they may decide that there is a more systemic problem they need to solve (like stopping the visualization and restarting it when there is data again).

Anyhoooo ... build will take around an hour to cut and test. I am leaving my currently ill-fated PR for the other problem in there as well. It should be stable enough to use for at least a while.

The only remaining problem I've seen is that seeking on audio-only programs is dreadfully slow. If that's not a problem for you, I'm willing to let it drop. If I ultimately decide to write my own demuxer that problem will go away. I'm sure it has to do with the constant assumption that MPEG-TS always has a video elementary stream.

TTYL, in about an hour.

[djp952]

See what this does for you ...

https://1drv.ms/u/s!AgEGEEVzGNq-jIUAK4gCc4AWiV0Smg?e=A12vr2 (kodi-18.4-release-audiofix2.zip).

It will take me until tomorrow to enter a proper Issue, but I do honestly think that Kodi needs to deal with/fix this on their own. Hopefully this .exe will solve the lionshare of your concerns.

[gibxxi]

Well the initial test was successful. However this was also the case with the previous compile. I'll have another go tonight when I have more time on all 3 systems, and let you know how it goes.

Many thanks for this.

Dan / Gib.

[gibxxi]

Hi Mike,

Done a few test playbacks on all 3 systems, and have had success on all 3 at the default stream chunk size of 4KB. This is only across one session on the HTPC/Laptop, and 2 on the desktop. But the results (thus far) look promising.

I noticed on the Kodi RSS feed a news item for Kodi "18.5", so i'm guessing they've put out an update. So we've probably missed the bus on getting a permanent fix into Leia, as I doubt there will be another release on the 18.x branch from this point. A check of the website will confirm it (Not had chance yet, at the time of writing).

So if you could build me a copy of 18.5 with your fix / hack contained, i'd apprieciate it. I'll keep testing for the rest of the week and post up a definite conclusion sometime over the weekend period (The time of the week I do the most radio listening), and let you know one way or another.

I'm assuming you'll be contacting the Kodi team with regards this issue and your fix? If so, and you wish to use any of the assets I've sent you so far, feel free to do so.

Thanks for your persistence on this issue, I greatly apprieciate it.

:)

Dan / Gib.

[djp952]

Yes on all counts. There is a new .zip for kodi-18.5 in the same OneDrive folder for you:

https://1drv.ms/u/s!AgEGEEVzGNq-jIUAK4gCc4AWiV0Smg?e=A12vr2

I strongly suggest installing Kodi 18.5 first, then replacing the .EXE. You don't necessarily need the .PDB (debug symbols), but it may enable better crash dumps if kodi crashes.

I'm going to take one more crack and finding a reasonable place to suggest a fix for the second issue since it only happens with streams like MPEG-TS, but if not I'll enter the Issue with what information I have for them. I still need to redo the PR for the first problem to their liking.

[djp952]

Here are the relevant links to the generated PR and Issue:

Problem 1 - PR (not accepted, needs work): https://github.com/xbmc/xbmc/pull/16910 Problem 2 - Issue: https://github.com/xbmc/xbmc/issues/16949

[gibxxi]

Yes on all counts. There is a new .zip for kodi-18.5 in the same OneDrive folder for you:

https://1drv.ms/u/s!AgEGEEVzGNq-jIUAK4gCc4AWiV0Smg?e=A12vr2

I strongly suggest installing Kodi 18.5 first, then replacing the .EXE. You don't necessarily need the .PDB (debug symbols), but it may enable better crash dumps if kodi crashes.

I'm going to take one more crack and finding a reasonable place to suggest a fix for the second issue since it only happens with streams like MPEG-TS, but if not I'll enter the Issue with what information I have for them. I still need to redo the PR for the first problem to their liking.

Hi Mike,

Thanks for that. Will do as advised. I'll keep the kodi.pdb file in there if you can make use of the crash logs it may generate, should we encounter any further issues? If it's of no use to you personally, i'll bin it.

I've monitored (followed) https://github.com/xbmc/xbmc/issues/16949, and should recieve an email if the topic gains any traction. If you need me to validate your/our findings by posting in the thread, let me know. Otherwise i'll let it run it's course.

Thanks again.

Dan / Gib.