Enhancement/Question

[milap612]

Thank you very much for awesome lib. Need to ask you 2 question of video operation.

1. Is there any way to create fast Movie.
2. Is there any way to create reverse an existing Movie.

[izzytwosheds]

Thank you! Glad you like LiTr! We don't do any manipulation with frame frequency or order, so both of those capabilities are not currently supported. They are very nice things to have, though, we will definitely consider adding them in the future. Thank you for this suggestion!

[milap612]

Thanks for quick reply, I would like to contribute if it is possible to implement, can you please help me with the road map to make it happen @izzytwosheds ?

[izzytwosheds]

I think you can trick LiTr into "speeding up" the video by adjusting video frame presentation time. Try dividing original sample time by "acceleration rate" in MediaExtractorMediaSource.getSampleTime and return that value. Make sure audio track is removed (sample app does that using TrackTransform). This should be a good place to start.

Reversing is a much more difficult problem. I feel we will have to write a custom implementation of MediaSource and Decoder where we will have to serve frames to Renderer in reverse order. The issue here is that we can't seek to any arbitrary position in encoded video file, we can seek only to a key frame and "play" from there, decoding frames one by one until we reach the right frame. So we will need a way to buffer up those intermediate frames and serve them in reverse order. That might require implementing a software renderer which provides access to frame data. That is something on my todo list.

[milap612]

Thanks for your support @izzytwosheds With your help, i have achieved fast/slow video and audio but on static basis. I will modify core library with passing the params into interface and create a PR in near future.

Regarding reversing the video, i have read some blogs and questions of last year and i have found a suitable response on stack-overflow may it will be helpful to implement. https://stackoverflow.com/a/38450847

i want to ask you that, can we manage it simply by changing the presentationTimeUs of each bufferInfo? which we passing while enqueuing "decoder.queueInputFrame(frame);" or while extracting frames "decoder.getOutputFrame(tag);" or while writing it into mixture " mediaMuxer.writeSampleData(targetTrack, frame.buffer, frame.bufferInfo);"

I appreciate your efforts and quick support.

[izzytwosheds]

I don't think we can reverse the video simply by switching presentationTimeUs. This is because of the way video compression works - each video frame depends on previous video frame to be correctly decoded. There are frames (called I-frames or key frames) that are independent, they are used when we seek the video. But the frames that follow I-frames usually contain only differences from the previous frame. So to reverse a video we would have to decode each frame between two I-frames (which would require extractor and decoder working together, as shown in stack overflow example) and reverse those frames. Once frame is decoded it has full information and becomes independent. So we will need a way to decode and store those frames before we can send them to renderer in reverse order.

[milap612]

Hi @izzytwosheds as per your guidance i have made some changes. Here i am attaching a patch for storing frames after decoding and writing it to mixture after all the frames get decoded. In this patch i have done nothing for reversing video, i am just feeding all the frames to the mixture at the end. But the result is kinda scary, whole video gets messy with buffers. Isn't it should working with the correct frame order ? M i missing anything? Writing_frames_into_mixture_inend.patch.zip

For reference, Output video (without audio) transcoded_123.mp4.zip Input video(with audio) 123.mp4.zip

[izzytwosheds]

Yeah, video is full of rendering artifacts, I don't think it is working well. Let me take a better look into this, reversing a video might require some deeper changes in LiTr.

[milap612]

Hey @izzytwosheds , I have tried storing all values of decoder by creating a dummy Decoder before actual decoder starts,

But when i start feeding the stored values to real decoder it always throws "android.media.MediaCodec$CodecException: Error 0xfffffff3". I'm not getting why it gives (MediaCodec.BUFFER_FLAG_END_OF_STREAM) with very first buffer info.

Cross Checked, Actual sequence and reverse sequence(with edited timeStemp). also, debug with sampleSize, BufferId and flags.

For Reference -> private fun feedInputToDecoder() { tempStoreValues = tempAllValues.pop() val inBufferId = tempStoreValues.inputBufferId if (inBufferId >= 0) { if (syncSampleTimes.isNotEmpty() ) { // If we're not yet at the beginning val buffer = tempStoreValues.buffer val sampleSize = tempStoreValues.sampleSize if (sampleSize > 0) { decoder!!.queueInputBuffer( inBufferId, 0, sampleSize, tempStoreValues.timeStamp, tempStoreValues.flag ) } // tempStoreValues = tempAllValues.pop() tempStoreTime = syncSampleTimes.pop() // extractor!!.seekTo(tempStoreValues?.timeStamp!!,) // syncSampleTimes.pop() extractor?.advance() } else { decoder!!.queueInputBuffer(inBufferId, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM) allInputExtracted = true } }else{ tempStoreValues = tempAllValues.pop() } }

Thanks for continues support.

[milap612]

Hey @izzytwosheds , Some how i have achieved reverse forward individual video and individual audio. But when i do that with audio and video both, in the processNextFrame for audio mediaSource.getSampleTrackIndex() always gets 0 value and due to that i'm not able to write audio after complete writing video. Find the patch and please help me out with it we are close to achieve this.

audio_stops_writing_after_video_complete.patch.zip

Can i get your email id ?

[izzytwosheds]

Hey. Sorry, just got back from vacation. Sounds like you are making very good progress! I will try your patch today and get back to you. Audio is very tricky to work with.

My email is izzat@linkedin.com

[izzytwosheds]

I took a look at your patch. One pretty major issue with it is that it will increase size of the video (I think, substantially) because you are setting I frame interval to zero. That allows you to reverse the frames, because now each frame in target video is independent, but that will significantly reduce compression rate.

Regarding audio. Each audio frame has a duration - it contains 1024 audio samples, which need to be reversed, too. That has to be performed on decoded frames inside of audio renderer, and then those frames themselves must be reversed. This might require doing things in two passes - decode and write out audio track, and then read samples from it in reverse order while reversing a video track.

Both of these operations would require introducing more implementations of MediaSource, Decoder and Renderer abstractions. I will help there.

[milap612]

Yes, i have been looking for creating a separate MediaSource for both audio and video to be render, but stuck between it. Is it a good choice to create a separate MediaSource for render audio for now ? if you would guide me to creating a separate instance for audio which only return audio index, then it would be great help to achieve it for now, also it will take a lot changes in to core lib. Regarding easy way to implement it would require more time and i don't want to steal your that much time for my requirements.

[milap612]

Hey @izzytwosheds , I have archived boomerang by adding a new object of MediaSource in TransformationPresenter class "startTransformation()" and for watermark "startVideoOverlayTransformation" functions.

Thanks a lot for your continues support.

Here is the patch -> complete_boomerang_in_middle_ofmovie.patch.zip

Exported Video -> transcoded_transcoded_123.mp4.zip

[izzytwosheds]

Hey Milap,

Nice work, video looks pretty cool. I will looks into generalizing your work and rework it into some combination of different MediaSource/Renderer implementation. I don’t want to make any changes in TransformationJob or transcoders. I will keep you posted.

Izzat

From: milap chokshi notifications@github.com Reply-To: linkedin/LiTr reply@reply.github.com Date: Friday, June 26, 2020 at 6:57 AM To: linkedin/LiTr LiTr@noreply.github.com Cc: Izzat Bahadirov ibahadirov@linkedin.com, Mention mention@noreply.github.com Subject: Re: [linkedin/LiTr] Enhancement/Question (#31)

Hey @izzytwoshedshttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fizzytwosheds&data=02%7C01%7Cibahadirov%40linkedin.com%7C08d1ec258cf04f7ba39e08d819bfb373%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637287658426896865&sdata=ljgdlgV%2FLPRNyyQpAaBVgjFF2N58oCh95fUA0kKzyOM%3D&reserved=0 , I have archived boomerang by adding a new object of MediaSource in TransformationPresenter class "startTransformation()" and for watermark "startVideoOverlayTransformation" functions.

Thanks a lot for your continues support.

Here is the patch -> complete_boomerang_in_middle_ofmovie.patch.ziphttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flinkedin%2FLiTr%2Ffiles%2F4836520%2Fcomplete_boomerang_in_middle_of_movie_.patch.zip&data=02%7C01%7Cibahadirov%40linkedin.com%7C08d1ec258cf04f7ba39e08d819bfb373%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637287658426896865&sdata=pgOuaG4UO0Zn7aS6ihJvekgMiAFIfzEeF75Qei%2BD1Dk%3D&reserved=0

Exported Video -> transcoded_transcoded_123.mp4.ziphttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flinkedin%2FLiTr%2Ffiles%2F4836526%2Ftranscoded_transcoded_123.mp4.zip&data=02%7C01%7Cibahadirov%40linkedin.com%7C08d1ec258cf04f7ba39e08d819bfb373%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637287658426906861&sdata=zoTRAGgmkPW2JCsBZSdlFPVLg8U1iUfHpw7b9mnKDjQ%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Flinkedin%2FLiTr%2Fissues%2F31%23issuecomment-650118703&data=02%7C01%7Cibahadirov%40linkedin.com%7C08d1ec258cf04f7ba39e08d819bfb373%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637287658426916857&sdata=d7l4KqmNUBSEGCvgnmXeDl7H4lXdY83JbLyGrUChSPs%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAGZBFPHA7MW7DEAIMP476DRYR5I3ANCNFSM4N7VNBUA&data=02%7C01%7Cibahadirov%40linkedin.com%7C08d1ec258cf04f7ba39e08d819bfb373%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637287658426916857&sdata=tg9Z1p6TyXNEQ%2Fn4SZuIsldwZkeTIY%2FyLkYhI%2BxP%2Bgk%3D&reserved=0.

[pawaom]

Is this included in the Demo app, where can we test it.

[izzytwosheds]

No, this is not included in Demo app. @milap612 was experimenting with this on his private fork.