ElvishArtisan / rivendell

A full-featured radio automation system targeted for use in professional broadcast and media environments
197 stars 63 forks source link

Cart loops repeatedly in main log instead of advancing to next item in log #921

Open Chester-Graham opened 7 months ago

Chester-Graham commented 7 months ago

Rivendell version 3.6.7 Database 347 on Centos 7 install in accordance with script I have a setup of 7 radio stations, all playing same production log that is refreshed nightly in the main log. I have a second log running in Aux 1 on each station that "pauses the main log" plays the legal id, then "resumes the main log". Yesterday at 10 am a long cart began playing that is 1 hour and 58 minutes long. On 3 of these stations that cart played to the end then started over, playing the same program over and over again until I was alerted to it this morning. The other 4 stations playing the same log had no issues, but they also weren't playing exactly the same cut as they are at different places in the rotation. The 3 stations that had the issue were in 3 different time zones, so the system was not trying to play out the same cut simultaneously on 3 stations. This morning I watched one station play this cut until it finished, the meters and playout timer were working fine in the main log. When it finished it just started over again without advancing to the next item in the log. In reviewing the playouts of that cart, there is one other cut that is nearly as long, and it looks like it has had excessive plays also, but I don't know when they occurred. All of the "client" machines are running in Virtualbox on top of Centos 7, with the server instance running directly in Centos 7. The "client" VM'S are all identically configured and were cloned, changing the hostname and ip address. Any ideas of where to look for resolution?

ElvishArtisan commented 7 months ago

What are the sizes of the audio files for the affected carts? You can get this information by doing:

ls -s /var/snd/<cartnum>_<cutnum>.wav

It sounds like we may run up against an overflow in the WAV file handling code (the version of BWF being used overflows for chunk sizes > 2,147,483,647). A file of 1:58:00 length can definitely be pushing close to that limit (the precise point depends on the format details --i.e. channels, sample rate, etc).

Chester-Graham commented 7 months ago

Encoded mp2 128bit ls -s /var/snd 012432 225168 /var/snd/012432_076.wav 224288 /var/snd/012432_077.wav

-Cart- Cut W/O- -Cart Title-------------- -Description-- -Len- Last Play Plays Start Date End Date -Days of Week- -Daypart----------- 012432 076 O076 Desire of Ages with David Desire of Ages 1:59 11/10/23 26 [none] TFN SuMoTuWeThFrSa [none] 012432 077 O078 Desire of Ages with David Desire of Ages 1:58 11/10/23 67 [none] TFN SuMoTuWeThFrSa [none]

ElvishArtisan commented 7 months ago

Which works out to a tad larger than 128 MB each. Nowhere near the overflow limit. Cross off that theory!

Please generate a Technical Playout Report for one of the logs, showing the looping behavior.

Chester-Graham commented 7 months ago

This is what played WBAJ-231109_Daily_Program_Log.txt

It continued until I manually stopped it the next day, surviving a log reload WBAJ-231110_Daily_Program_Log.txt

This is what was scheduled to play Playoutlog 11-09-2023.txt