mismatch between the recorded data and frames

[lawpdas]

SPECIFIC ISSUE ENCOUNTERED

I use the recording API to record the custom data (camera pose) into the mp4 file. To read the data conveniently, I save it by using the following code:

public void onDrawFrame(SampleRender render) {
    ...
    frame = session.update();
    ...

    Camera camera = frame.getCamera();
    Pose camera_pos = camera.getPose();
    float[] translation = camera_pos.getTranslation();
    float[] quaternion = camera_pos.getRotationQuaternion();
    String data = String.format(
                      "%.6f,%.6f,%.6f,%.6f,%.6f,%.6f,%.6f\n",
                      translation[0], translation[1], translation[2],
                      quaternion[0], quaternion[1], quaternion[2], quaternion[3]);

    ByteBuffer payload = ByteBuffer.wrap(data.getBytes());
    try {
        frame.recordTrackData(ANCHOR_TRACK_ID, payload);
    } catch (IllegalStateException e) {
        Log.e(TAG, "Error in recording pos data into external data track.", e);
    }
    ...

After that, I use this command ffmpeg -i arcore.mp4 -map 0:7 -codec copy -f data arcore.txt and a python script to read the data. However, the video has 590 frames, and the data saved in the data track has 587 lines. There is a mismatch between the data and the video frame. I also find that some lines at the beginning are '0.000000,0.000000,0.000000,0.000000,0.000000,-0.707107,0.707107'. I don't know why these lines are incorrect. In fact, I want to record some data and frames for a machine learning project, but I don't know how to save and read them in the correct way. Could you help me with this problem?

VERSIONS USED

• Android Studio: 2020.3.1 patch4
• ARCore SDK for Android: 1.29.0
• Device manufacturer, model, and O/S: Redmi note 7, android 10
• Google Play Services for AR (ARCore): 1.29.213210223 On Windows, use: adb shell pm dump com.google.ar.core | findstr /i "packages: versionName" On macOS, use: adb shell pm dump com.google.ar.core | egrep -i versionName\|packages:
• Output of adb shell getprop ro.build.fingerprint:

[GEllickson-Hover]

@lawpdas I'm wondering if you have any code example of how you're manually parsing the recorded mp4 data with python? I'm hoping to do something similar and record my own AR or sensor data into the mp4, so it's associated with a frame and then analyze it outside of ARCore, but I'm having trouble finding examples of how to do this. In particular, like how best to de-serialize the binary data added from frame.recordTrackData() outside of Android as I'm not sure how the data is written to the mp4 tracks. Thank you.