Audio and slides are out of sync

[DaleHensley]

In some created videos, the audio for slide x may play during the display of slide y.

Describe the bug When choosing the options for creating a video, some options cause the audio for slide X to play in part of slide y. It appears when merging a slide picture with corresponding audio, some of the picture frames are lost causing the audio to play into the next slide.

To Reproduce Steps to reproduce the behavior:

1. Load a clean version of "002 Lost Coin.bloom" into the device.
2. Execute Story Producer app and "002 Lost Coin" should appear in the "Story Templates" list.
3. Select "002 Lost Coin" and the "002 Lost Coin" story is loaded.
4. Go to the "Accuracy Check" phase and approve all the slides.

   TEST 1

5. Go to the "Finalize" phase and enter a field in "Local Credits" and use the word "Coin" for "Text in Filename". Then select the following options: "Include Background Music" & "Include Pictures". Once this data has been entered, press the "Create Video" button.
6. A status bar appears showing the progress of the creation of the video.
7. When the creation of the video is complete, go to the share phase.
8. Play the file named "002_Coin_FxPx.mp4". Unexpectedly, near the end of the video, the audio does not correspond to the displayed slide. You will also notice that the story audio plays during the local credits and copyright page.

   TEST 2

9. Go to the "Finalize" phase and enter a field in "Local Credits" and use the word "Coin" for "Text in Filename". Then select the following options: "Include Background Music", "Include Pictures" and "Include Story Text". Once this data has been entered, press the "Create Video" button.
10. A status bar appears showing the progress of the creation of the video.
11. When the creation of the video is complete, go to the share phase.
12. Play the file named "002_Coin_FxPxTx.mp4". Unexpectedly, near the end of the video, the audio does not correspond to the displayed slide. You will also notice that the story audio plays during the local credits and copyright page.

    TEST 3

13. Go to the "Finalize" phase and enter a field in "Local Credits" and use the word "Coin" for "Text in Filename". Then select the following options: "Include Background Music", "Include Pictures" and "Enable Picture Motion". Once this data has been entered, press the "Create Video" button.
14. A status bar appears showing the progress of the creation of the video.
15. When the creation of the video is complete, go to the share phase.
16. Play the file named "002_Coin_FxPxMv.mp4". Expectedly, the appropriate audio clips are played with the proper slides You will also notice that the audio does not plays during the "local credits" and "copyright page".

Below are the capture audio files.

Expected behavior The expected behavior is for the audio clips to be played with the proper slide. There should not be story audio during display of the "local credits and "copyright page".

Screenshots If applicable, add screenshots to help explain your problem.

Versions of the software and Android -StoryProducer Version: 3.0.6 -Android Version: Android 12 on a Pixel 5 phone

Additional context Add any other context about the problem here.

---

The sections above are all that's needed for a report. The following sections are filled out and groomed by the development team before work begins...

To Do:

• Do __
• Do __

Acceptance

[steps to verify that the fix works as expected]

Testing Ideas

[Tips for the testers. If you include detailed steps, use the 'Steps to Reproduce' format.] __

Original Report from __

indented quoted text is good but optional

https://user-images.githubusercontent.com/78509270/156290451-717dccde-b9a3-4029-b023-43b06c687e91.mp4

https://user-images.githubusercontent.com/78509270/156290521-908979b8-2d4c-4aa1-ab20-8894a2678066.mp4

https://user-images.githubusercontent.com/78509270/156290626-59eccb6b-d6b3-429b-a170-f7c8ddc2970d.mp4

[johnml1135]

Hello @DaleHensley, here are a few thoughts:

• Hardware decoders are always finicky. I have spent many hours trying to work through them and they kept giving me grief.
• There are software decoders available, but they are much slower

Idea 1: release buffer from true to false?

• From the log: 03-04 14:03:58.946 24753 27366 W CCodecBufferChannel: [c2.android.mp3.decoder#550] The app is calling releaseOutputBuffer() with timestamp or render=true with non-video buffers. Apps should call releaseOutputBuffer() with render=false for those.
• The relevant code is here: https://github.com/sillsdev/StoryProducer/blob/65f9f6550e3ebff0f910e58d63d00054e9bbe693/app/src/main/java/org/sil/storyproducer/tools/media/pipe/PipedMediaCodec.kt#L77
• The documentation for the function is here: https://developer.android.com/reference/android/media/MediaCodec#releaseOutputBuffer(int,%20boolean)
• In looking at the history of the file, it appears that I made a change to "False" here: https://github.com/sillsdev/StoryProducer/blob/65f9f6550e3ebff0f910e58d63d00054e9bbe693/app/src/main/java/org/sil/storyproducer/tools/media/pipe/PipedMediaCodec.kt#L136
• but did not propagate it to the other instance. I don't fully know why. Changing it to false in the first location is worth a shot.

Idea 2: Codec hell Here is where the codecs are chosen: https://github.com/sillsdev/StoryProducer/blob/65f9f6550e3ebff0f910e58d63d00054e9bbe693/app/src/main/java/org/sil/storyproducer/tools/Utilities.kt#L29-L42

Here is an idea based upon a post here [https://stackoverflow.com/questions/35207531/android-mediacodec-releaseoutputbuffer-throws-mediacodec-codecexception-when-dec]

• "The best approach to do although it can feel wrong is to try + catch + retry. "
• Basically, instead of just trying the first codec that is successfully returned, get all the ones returned. If there is an error (you should be able to detect it), try the next codec that has a possibility of working. One of them will likely be the software codec and work well, even if it is very slow.

[johnml1135]

I would recommend trying idea 1 first (changing True to False). The other one will take design and rework.

[DaleHensley]

John:

Thanks for looking into this, it is a huge help !!! I will start on option one and then dive into option 2.

What is curious is that only videos with movement works for Android 12 and any type of video works for Android 9. Plus we get all those errors on Android 12 and not for Android 9. Any speculation as to why?

I have noticed that for the Android 12 video with movement, the processing time is noticeably shorter than the Android 12 video video without movement. This might indicate the difference between hardware and software codec selection.

Were you able to ascertain the source of the two messages that were printed multiple times( FrameEvents: updateAcquireFence: Did not find frame. (2885 times) & MediaCodec: rendring output error -38 (9886 times)? Do you think the MediaCodec is due to the wrong Codec selection?

Thanks again for all your help !!! In Him Dale

On Tue, Mar 8, 2022 at 11:54 AM John Lambert @.***> wrote:

I would recommend trying idea 1 first (changing True to False). The other one will take design and rework.

— Reply to this email directly, view it on GitHub https://github.com/sillsdev/StoryProducer/issues/630#issuecomment-1062146802, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASW7JVRPQ5KQR4NFX2PK5LLU66V7TANCNFSM5PVZY5AA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>