Problems when writing data to frame_timestamps.txt on Android device

[luckykk273]

Hello, thanks for providing such amazing and easy-to-use tool for collecting images and IMU data simultaneously. When I try to collect data with pixel 4XL, I encountered two problems:

1. frame_timestamps.txt won't be written if I didn't click the stop button(for instance, click return on smart phone and leave the app temporarily); The frame_timestamps.txt will be written only when VideoEncoderCore.release()​ is called. If I leave the app without clicking the stop button, the CameraCaptureActivity will go into state of onPause(or onDestroy) and the recording state in CameraSurfaceRenderer won't be changed so that VideoEncoderCore.release() won't be called.

2. The number of data in movie_metadata.csv and the frame counts in movie.mp4 are sometimes inconsistent; I would check whether the frame counts in movie.mp4 and the number of data in movie_metadata.csv are the same or not after collecting the data. I notice that the frame counts in movie.mp4 sometimes are less than the number of data in movie_metadata.csv. The higher the frame size is, the more likely it happened. The metadata will be written to movie_metadata.csv when the capture is completed(onCaptureCompleted in CameraCaptureSession.CaptureCallback). When stop button is clicked, it will signal EOF on input stream and stop submitting data to the MediaCodec immediately. But there are still some data in buffer that haven't been drained out.

[JzHuai0108]

Hi @luckykk273, thank you for using marslogger. I appreciate the detailed description of the two issues.

For the first one, I think you have described exactly one scenario the program malfunctions. Indeed, the pause function is kind of experimental. In the following days, I will follow it up and try to patch up the issue. But be cautious that it may take a week since my workload gets me on the brink of burnout everyday :). If you have further suggestions regarding the patch, please let me know. Also, are you installing the app from source (e.g., via android studio) or from the released apk? I would like to know this to decide how to update the version on your end.

For the second one, I have noted this fact earlier. See the wiki page, "For android devices, because the metadata entry may not exactly match frames in the video, timestamps for frames are saved separately into a txt file, frame_timestamps.txt." I think you have described its cause precisely. For timestamps, it is advised to use frame_timestamps.txt. Otherwise, I honestly think the movie_metadata.csv contains enough info for the recorded frames since the camera changes characteristics slowly.

[luckykk273]

Hi @JzHuai0108 , thank you for your detailed answer. I have to say sorry for being late reply.

I installed the app from source via android studio exactly. I think a quick fix for the first problem is to write data to the frame_timestamps.txt immediately in drainEncoder() rather than storing it into mTimeArray temporarily. But I'm not sure if there is any consideration(for efficiency, design, etc.) to use mTimeArray to store.

public void drainEncoder(boolean endOfStream) {
    ...
    // Original:
    mTimeArray.add(new TimePair(mBufferInfo.presentationTimeUs, System.currentTimeMillis()));

    // Modified:
    String delimiter = ",";
    StringBuilder sb = new StringBuilder();
    sb.append(mBufferInfo.presentationTimeUs+ "000");
    sb.append(delimiter + System.currentTimeMillis() + "000000\n");
    mFrameMetadataWriter.write(sb.toString());
    ....
}

Another solution for the first problem is to call changeRecordingState() in CameraSurfaceRenderer.notifyPausing()(or in CameraCaptureActivity.onPause() before CameraSurfaceRenderer.notifyPausing() is called). I think this solution is more suitable but I'm not sure if it has chance that onDrawFrame() won't be called(before onDrawFrame() is called, mSurfaceTexture is release first?).

Thank you for the reference to the wiki page and the advice about the usage of timestamps. I didn't notice before. I will edit the title and keep only the first problem:).

[JzHuai0108]

Thank you for the two workable solutions to remedy the first problem. I will weigh the two solutions and give you feedback within a week. For the moment, I try to provide you with quick thoughts and leave myself some notes without actually tuning the code base.

1. I save the timestamps at the destruction of the VideoEncoderCore possibly for slightly better efficiency. Obviously, this will not make much difference for such petty data.
2. For the second solution, I need to consider the interactions between the SurfaceTexture, the Renderer, and the CaptureActivity.

[luckykk273]

Thank you for giving some direction to my feedback. As you mentioned, the interactions will be the most complicated part to deal with. Furthermore, I have no need to get this fix immediately(just to describe what problem I met before), just feel free to update any time:).