RSS/Feed episode audio length problem.

[alexolivan]

Hi. I'm finding that, for a given recording and uploading of 5h length, the resulting audio ends up being just a little above 1 and a half hours length. The source record is in fact 5 hour lenght, so there's no problem regarding the source media.

To put into perspective, let me point out that, BEFORE RC0, RDcatch upload events ended up with episodes with a 'declared' length of around 96 mins, matching the resulting audio length, regardless of the source media being 5 hours. After upgrading to RC0, I deleted the RDCatch upload event, and re-created it as a new event (just to ensure a fresh new DB entry for the event is created). I also increased the VM CPU count up to 4 cores, just in case it helps... AFTER RC0, the issue is still there, but now the episode length do reads as if the uploaded audio would be 5h, altough it is not.

In the following screenshot, I show marked in green how now cut length and episode length do match, wereas before, it didn't:

However, the media length keeps up hitting the 96/97 minute dead-end, as easily seen on the server content, where previous episodes and the RC0 one do roughly have the same size... so there's something wrong here:

As a way to check RDCatch and its events being the source of the problem, I tried deleting the new episode from RDCastManager, and use its Post from Cart/Cut facilities to reupload it manually:

No error reported and, apparently, it all went good:

However, to my surprise, the uploaded audio matched the old one, being just 96:26 minutes in length... a refresh on the console at the server, confirmed the file size was what it was:

I re-checked the recording has audio at its full length... and it has:

Now I'm planning to try to reduce the recording length to 4 hours (all the other recordings, being up to 3 hours, do work fine) and see if I'm hitting some kind of limitation or something.

Cheers.

[ElvishArtisan]

Hmm. I suspect that you're running into the 2 GB file length limitation that is inherent in the 'Broadcast Wave File' (BWF) format that is used for audio storage in Rivendell. You might want to try using MPEG Layer II for the initial RDCatch recording rather then one of the PCM formats, as that will allow you to save longer durations.

[alexolivan]

Hi Fred! I think I'm missing the point.

All the settings I recall to have I set to MPEG Layer II (this is the way I was told I should to)... Maybe I'm missing something.

But, what is confusing to me is that, while I understand that a 2 GB file size limit (I wasn't aware of!) would cause an audio media to eventually run out of space, I don't understand why the recorded audio has indeed 5 hours in length (the recording is successful, it's the process to convert the record cut into an mp3 file what seems to be failing somehow) In fact, looking at the actual file in var/snd/ I see it has some 289MB in size (quite a large file for audio, but far from huge):

I wonder whether you're refering to some kind of temp file or cache, PCM format, that's being used while transcoding from MPEG L2 to MPEG L3... But, even in such case... why does it end up in a 96/97 mimutes length, while I have 120 and 180 minutes length records and feed uploads successful? ... I shouldn't be able to have 2h, 3h episodes, nothing beyond 96/97 minutes, but that's not the case. So It's quite a mistery. I've set the recording time to 4 hours and see what happens. Providing it would succeed, it would be very indicative, and further testing should be done... will report.

Cheers!

[alexolivan]

Hi!.

some updates... changed length on same recording. First I went down from 5h to 4h ... same result. just above 96 minutes. Ok, went down to 2h, very conservative... and it worked! For the first time I got this show uploaded for more than 96/97 minutes! I'm rescheduling to 3h. I have some 3h recordings uploaded successfully... I bet it will work!

Cheers.

[alexolivan]

Ok... just as anticipated, it worked.

By just editing the recording event, and changing its duration/length, and only it, I think I can asert that the very minor changes at backend/DB have occured on these tests. The upload event has not changed at all... yet, for a 3h long (or below) source cut, as expected (I have several others that long) it works, (so I have an almost 180 min RSS episode), whereas at 4h long (or the original 5h) the upload ends up alway with a kind of boundary length around 96/97 mins.

I've noticed upload of RSS episodes have been removed on the code, and what appears to be deep refactoring of code related to RDCatch is ongoing on another branch... so, I've freezed updating/upgrading my lab VM. I think that right now, the best idea is to wait until those changes are merged into qt5 branch before keep testing this (my plan would be to create a new set of record and upload events, 3h 30min length, and try to zero-in the 'max upload' value)

Cheers.

[ElvishArtisan]

By just editing the recording event, and changing its duration/length, and only it, I think I can asert that the very minor changes at backend/DB have occured on these tests. The upload event has not changed at all... yet, for a 3h long (or below) source cut, as expected (I have several others that long) it works, (so I have an almost 180 min RSS episode), whereas at 4h long (or the original 5h) the upload ends up alway with a kind of boundary length around 96/97 mins.

This is all a function of 32 bit BWF file format used for audio storage by Rivendell, which maxes out at 2 GB chunk size. Unfortunately, there's no easy way determine where the precise duration limit is as it is a function of both the audio encoding and the accompanying metadata chunks in the file. The "worst case" is using stereo PCM24 @ 48ksample/sec, which will overflow at around 2 hours duration. Perhaps we should just put a "system cap" on file duration at that point?

I've noticed upload of RSS episodes have been removed on the code, and what appears to be deep refactoring of code related to RDCatch is ongoing on another branch... so, I've freezed updating/upgrading my lab VM. I think that right now, the best idea is to wait until those changes are merged into qt5 branch before keep testing this (my plan would be to create a new set of record and upload events, 3h 30min length, and try to zero-in the 'max upload' value)

Mostly finished over the past weekend. One of the things that has come out in scalability testing here is that Qt5 is much more resource intensive than previous versions; to the point where rdcatch(1) was blowing up at startup due to a combinatorial explosion in the monitor code attempting to open point-to-point connections with every other Rivendell instance. That's all been replaced by a multicast-based system that scales linearly. We do still have to finish up the RSS stuff (currently blows up at scale due to the loading of feed images at startup). I'm just going to back out that feature for the time being.

[alexolivan]

Perhaps we should just put a "system cap" on file duration at that point?

Makes sense to me. I think that for the vast majority of use-cases, current capacity is well up to the task.

Cheers!

[ElvishArtisan]

We do still have to finish up the RSS stuff (currently blows up at scale due to the loading of feed images at startup). I'm just going to back out that feature for the time being.

I ended up not having to go this route. Instead, as of 4a3454c the RSS thumbnails have been optimized to load about an order of magnitude faster (the previous code was causing rdcatch(1) to experience start-up delays of more than a minute for sites with lots of RSS feeds enabled). And, as of that same commit, you'll need ImageMagick's 'Magick++' API installed in order for Rivendell to build. See 'INSTALL' for details.

[alexolivan]

I'm closing this one to de-clutter the issues tab on the way to 4.0 release... Cause, being the root cause known, I hope that, in the mean time until knowledge of this kind of limit spreads into Rivendell users community, if someone tries too long RSS Episodes to upload, he/she will find this thread, even when closed, googling around.

Thank you for your patience. Cheers!