saki4510t / ScreenRecordingSample

Simultaneous audio and screen recording sample using MediaCodec/MediaMuxer/VirtualDisplay
Apache License 2.0
189 stars 89 forks source link

How to add pause recording functionality? #1

Closed RazorPT closed 7 years ago

RazorPT commented 9 years ago

Just tried it and it works really good i just would like to know how to make a pause system? I read that to pause a recording for a media encoder you just record the time that you pause and subtract that time to the PTSUs.

So i tried create a new toggle pause button and replicate most of the code for your normal toggle, when i hit pause i register the System.nanotime in a varible and on resume i subtract the new System.nanotime with my variable like this (mPauseTime is set to 0 when recording starts):

} else if (ACTION_PAUSE.equals(action)) {
        mlastNanoSecondsFromSystem = System.nanoTime();
        mIsPaused = true;
        updateStatus();
    } else if (ACTION_RESUME.equals(action)) {
        long pauseTimeInNanos = System.nanoTime() - mlastNanoSecondsFromSystem;
        mPauseTime = mPauseTime + (pauseTimeInNanos / 1000);
        mIsPaused = false;
        sMuxer.setPausePTSUs(mPauseTime);
        updateStatus();
    }

Then on the sMuxer i have:

public void setPausePTSUs (long pausePTSUs) {
    mAudioEncoder.setPausePTSUs(pausePTSUs);
    mVideoEncoder.setPausePTSUs(pausePTSUs);
}

and on the MediaEncoder class i finally have:

 protected long getPTSUs() {
    long result = System.nanoTime() / 1000L;
    // presentationTimeUs should be monotonic
    // otherwise muxer fail to write
    if (result < prevOutputPTSUs) {
        result = (prevOutputPTSUs - result) + result;
        result = result - mPausePTSUs;
    }
    return result;
}

public void setPausePTSUs (long pausePTSUs) {
    mPausePTSUs = pausePTSUs;
}

Result: the Video starts black and after some seconds it starts showing the audio plays as normal from beginning even with video black. Am i close?

Thank you so much for your time

saki4510t commented 9 years ago

Hello,

I just added short period pausing feature.

I think basic way to calculate PTS is correct but it will be better to always subtract offset(mPausePTSUs) from PTS.

protected long getPTSUs() {
    long result = System.nanoTime() / 1000L - mPausePTSUs;
    // presentationTimeUs should be monotonic
    // otherwise muxer fail to write
    if (result < prevOutputPTSUs) {
        result = (prevOutputPTSUs - result) + result;
    }
    return result;
}

I assume the issue is on other part of your implementation. I'm not sure how your actual overall implementation is, you may mistake the way to queue data and handling of queued data. You need to block and skip all new coming data, but you need to keep handling already queued/encoded data until all buffers of MediaCodec encoder become empty.

with best regards, saki

RazorPT commented 9 years ago

Again thank you so much its working :), although something strange is happening on my nexus 5 the normal photos app (default for android movies) reads the movie correctly but VLC for android is having trouble starting with 1 second black screen and pausing the video when it hit pause while recording.

What i found is your code is working but something strange is happening for example i made a video where i start recording and pause it at 00:04 then resume recording at 00:15 and stoped recording at 00:17 the final video says it has 00:17 duration and automatically jumps (using default lollipop player) from 00:04 to 00:15 when it really should shrink in duration from the original 00:17 to 00:07 duration.

In conclusion the recorder is not recording any frames/sounds while on paused but is recording the time that passed by. Something wrong with the timestamps?

Thank you!

saki4510t commented 9 years ago

Hello,

Well I'm not sure which is correct behavior, but most video player behave like Photo app.
I assume there are mainly two reasons; one is related to timestamps as you mentioned and the other is related to key frame(sync frame).


The background of the reason related to timestamps is little bit complex. I'm not sure I can explain well with my poor English...
As I know it is better to use timestamps value when provided queuing into MediaCodec encoder instead of when de-queueing from MediaCodec because no one know how long encoding takes time.
But like this sample project, when video data is directly passed from VirtualDisplay to MediaCodec or from MediaPlayer to MediaCodec etc. via Surface(this is the most efficient way), it is difficult to adjust and provide timestamps when queueing data into MediaCodec. So I adjust timestamps when dequeuing data from MediaCodec.
And as you know, video encoding takes much time than audio encoding and video encoding stops after some delay of several hundreds milliseconds - few seconds than audio encoding even if stop queuing data at the same time. If you don't pause recording, this is not a matter. But this will cause some gap between audio timestamps and video timestamps(video timestamps will delay) when resuming.
As I know there are several solution,

  1. use SurfaceTexture etc. to get video images from VirtualDisplay/MediaPlayer etc. and queue that data with adjusted timestamps by your own code(and don't adjust timestamps offset when de-queuing).
  2. Second way is something tricky and difficult to say in English well, please check my other project TimeLapseRecordingSample

The second reason, H.264 encoded video data include key frame(I-fram/sync-frame) periodically and should start with key frame. In this sample project, the period is set 10 as MediaFormat.KEY_I_FRAME_INTERVAL. But app don't know when use pause / resume and there is a possibility that key-frame become incorrect or lost. I assume this will make some issue on VLC.
This may be solved using PARAMETER_KEY_REQUEST_SYNC_FRAME when queuing data into MediaCodec encoder by your own code(instead of queuing via Surface directly).

with best regards, saki

RazorPT commented 9 years ago

Hi there sorry for the late response i have been bit a busy :)

I think you misunderstood me there isn´t gap between audio and video both are perfectly in sync for what i could see. The only problem is that the time that i paused is added to the final video.

So what happens is that the default player on lollipop automatically jumps from second 4 to second 15. I will explain better what i did:

So 00:00 ---Recording --- 00:04 --- Paused ---- 00:15 --- Resume --- 00:17 STOP

The final video when played will show that is 00:17 long but when player gets to 00:04 it automatically jumps to 00:15 while video and audio being perfectly in sync i just need to remove the paused time from the final video and i think this is what breaking VLC apart.

This seems like a simple timestamp problem like we aren´t subtracting the timestamps correctly?

I will try to study your code and see if something is missing

Thanks again!

RazorPT commented 9 years ago

Ok just found out what the problem was you were miscalculating the timestamps as i expected. I thought of forking your repo and ask for a pull request but since you are using Eclipse and i am using Android Studio i will tell here what to change:

In MediaEncoder.java on line 229 replace: offsetPTSUs = mLastPausedTimeUs - System.nanoTime() / 1000; with offsetPTSUs = (System.nanoTime() / 1000) - mLastPausedTimeUs;

And thats it! No more VLC problems and now the paused time doesn´t show on our final video!

There is only one bug left that i just discovered that when i pause and resume the video will start flickering when changing activities always showing between ScreenRecordingSample Main Activity and what i am doing and this just happens after resuming recording.

I just tested the pause code that i used on https://github.com/yrom/ScreenRecorder (no audio support) and that doesn´t happen after resuming. So its something about your awesome code. Any ideas?

I can send a recording that i made for you to see.

saki4510t commented 9 years ago

Hello,

Thank you for providing information. I will fix the issue about offsetPTSUs.

Video flickering issue... well I have tested on my devices but it did not occur. Could you send me movie by e-mail if possible? It may fix with changing frame rate to 30fps and/or shorted waiting time in mScreenCaptureTask in MediaScreenEncoder.java.

I also found another issue that this sample does not work well on my Nexus9 with Android M. I assume the reason is VirtualDisplay does not pass any screen images to MediaScreenEncoder. When I use Android5.x on Nexus9, this worked well. I already found workaround and will update if I have a time.
saki

RazorPT commented 9 years ago

I will send the video to your email: t_saki@serenegiant.com.

I will try what you suggested to see if the flickering stops, this problem is happening on my nexus 5 running android 5.1.1. I didn´t tried installing Android M yet to see what happens

saki4510t commented 9 years ago

Hello,

I fixed the blinking issue on Nexus5. I'm not sure whether this is the best way.
Unfortunately issue on Nexus9 with Android M does not fix yet.

saki

RazorPT commented 9 years ago

Might be a bug on Android M? Google specifically states "The Android preview system images are not stable releases, and may contain errors and defects that can result in damage to your computer systems, devices, and data"

I will try soon install Android M on my nexus 5 to see how it behaves.

saki4510t commented 9 years ago

Hello,

Android M on Nexus9 can not override by factory images of previous version (like Android 5.1.1 etc.) now and I can not restore to Android 5.1.1 ;-( I'm not sure other users have same issue and whether the issue occur on Nexus5. So please be careful to install Android M.

saki

2015-06-24 18:51 GMT+09:00 RazorPT notifications@github.com:

Might be a bug on Android M? Google specifically states "The Android preview system images are not stable releases, and may contain errors and defects that can result in damage to your computer systems, devices, and data"

I will try soon install Android M on my nexus 5 to see how it behaves.

— Reply to this email directly or view it on GitHub https://github.com/saki4510t/ScreenRecordingSample/issues/1#issuecomment-114809769 .

♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪ ♪ serenegiant ♪ ♪ t_saki@serenegiant.com ♪ ♪ saki4510t@gmail.com ♪ ♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪♪

RazorPT commented 9 years ago

Thats weird the google website http://developer.android.com/preview/download.html states "If you want to uninstall the preview and revert the device to factory specifications, go to developers.google.com/android and download the image you want to flash to for your device. Follow the instructions on that page to flash the image to your device."

So it should work did you received any error while trying to downgrade? Did it freeze?

Did you followed every step like running "fastboot oem unlock"?

Following this steps should work: https://developers.google.com/android/nexus/images

saki4510t commented 9 years ago

Of course I know the site but but it never works in case of downgrading from Android M to lower version(at least on Nexus9(WiFi)). Flashing factory image always fail(even if I use newly downloaded factory images and SDK and adb etc.). Overriding Android M with Android M works well and upgrading/downgrading between other factory images than Android M also work well. I assume when using old SDK(and old adb), it my work well though.

The console log is like this.

...
(bootloader) ability is 1
(bootloader) Device stete is unlock already
OKAY [  0.142s]
finished. total time: 0.142s
MacMini:volantis-lmy47x saki$ ./flash-all.sh
target reported max download size of 518205818 bytes
sending 'bootloader' (2915 KB)...
OKAY [  0.264s]
writing 'bootloader'...
(bootloader) Device State : Unlocked
(bootloader) zip header checking...
(bootloader) shift signature_size for header checking...
(bootloader) zip info parsing...
(bootloader) checking model ID...
(bootloader) start image[hboot] unzipping for pre-update check...
(bootloader) start image[bct] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[mts_preboot_prod] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[mts_prod] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[hboot] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[nvtboot] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[nvtbootwb0] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[tos] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[sp1] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[gp1] unzipping & flashing...
(bootloader) ...... Successful
(bootloader) start image[pt] unzipping & flashing...
(bootloader) ...... Successful
OKAY [  5.416s]
finished. total time: 5.680s
rebooting into bootloader...
OKAY [  0.028s]
finished. total time: 0.028s
target reported max download size of 518205818 bytes
archive does not contain 'boot.sig'
archive does not contain 'recovery.sig'
fastboot(867,0xa03ca1a8) malloc: *** mach_vm_map(size=1896734720) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
failed to allocate 1894839552 bytes
error: update package missing system.img
RazorPT commented 9 years ago

Thats really not normal try searching for other people with same problem or report the problem to Google

Another thing to do is try to install cynogen and then revert back to vanilla 5.1.1 http://wiki.cyanogenmod.org/w/Install_CM_for_flounder

I successfully flashed several phones including another nexus 5 with cynogen might give it a try?

RazorPT commented 9 years ago

Hi again Saki just ran your code on Android M on Nexus 5 and didn't found an issue. So it most be a problem related with the preview for the Nexus 9. I can send the recording i made and you will its perfectly fine.

saki4510t commented 9 years ago

Hi, Thank you for providing feedback. saki

RazorPT commented 9 years ago

Actually after some more extensive tests i noticed that videos produced on Android M can't be correctly reproduced by VLC (Saying error playing video) and this aren't bugs on your code since i tested Telecine Project from Jake Wharton which uses a different approach (uses MediaRecord instead of MediaCodec & MediaMuxer) to record screen and it also gives problems on VLC so its an Android M related issue as you suggested.

I also detected other issues on Android M apis which aren't related with screen recording so it most have some internal apis broken. So i reverted back to android 5.1.1 and all is fixed!

By the way could you explain what was causing the flickering problem after pause? You added a ton of code (glutils package) i am wondering if all that code is needed?

Thank you so much for your time

saki4510t commented 9 years ago

I don't confirm well and I'm not sure actual reason, I assume VirtualDisplay on Android M will not able to write screen images into Surface came from MediaCodec encoder/MediaRecorder and/or can not produce event when it write screen images into Surface. And MediaCodec/MediaRecorder can not accept screen images.
I'm not sure this is normal behavior or issue of Android M, but fortunately SurfaceTexture can accept both screen images and events and possible to write them into Surface. As you may know there are some way to do so and I think using EGL is best/most efficient way.

praveenyadati commented 7 years ago

Hi Saki, I just cloned your project and ran it on my device. Video recording is working fine and i found that pause/resume functionality doesnt seem to be working fine. Can you please look into this

saki4510t commented 7 years ago

Hi,that is because short time pausing/resuming was disabled on MediaMuxerWrapper. I just updated this repository and it will work.

NLLAPPS commented 7 years ago

@saki4510t @RazorP It seems that jump of the time or rather miscalculated time bug still there but only if you pause and resume more than 2 times T

NLLAPPS commented 7 years ago

changing offsetPTSUs = System.nanoTime() / 1000 - mLastPausedTimeUs; to offsetPTSUs = offsetPTSUs+ ((System.nanoTime() / 1000L) - mLastPausedTimeUs);

seems to solve it