Landmark returning lagging by one frame in MediaPipe

[Tendaliu]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

windows

Mobile device if the issue happens on mobile device

No response

Browser and version if the issue happens on browser

No response

Programming Language and version

c++

MediaPipe version

0.10.14

Bazel version

6.1.1

Solution

Holistic tracking

Android Studio, NDK, SDK versions (if issue is related to building in Android environment)

No response

Xcode & Tulsi version (if issue is related to building for iOS)

No response

Describe the actual behavior

When using Opencv to render landmarks on video frames, the landmarks appear to be lagging by one frame.

Describe the expected behaviour

Landmarks should be returned for the current frame, matching the motion of the face/body in real-time.

Standalone code/steps you may have used to try to get what you need

std::cout << "Running Mediapipe_RunMPPGraph_Direct" << std::endl;
    try {
        cv::Mat camera_frame(cv::Size(image_width, image_height), CV_8UC3, (uchar*)image_data);
        cv::Mat camera_frame_RGB;
        cv::cvtColor(camera_frame, camera_frame_RGB, cv::COLOR_BGR2RGB);

        auto input_frame = absl::make_unique<mediapipe::ImageFrame>(
            mediapipe::ImageFormat::SRGB, camera_frame_RGB.cols, camera_frame_RGB.rows,
            mediapipe::ImageFrame::kDefaultAlignmentBoundary);
        cv::Mat input_frame_mat = mediapipe::formats::MatView(input_frame.get());
        camera_frame_RGB.copyTo(input_frame_mat);

        size_t frame_timestamp_us = (double)cv::getTickCount() / (double)cv::getTickFrequency() * 1e6;

        MP_RETURN_IF_ERROR(m_Graph.AddPacketToInputStream(
            m_Video_InputStreamName, mediapipe::Adopt(input_frame.release())
            .At(mediapipe::Timestamp(frame_timestamp_us))));

        // Clear the landmarks initially

        if (m_pFaceLandmarksPoller != nullptr) {
            mediapipe::Packet faceLandmarksPacket;
            if (m_pFaceLandmarksPoller->QueueSize() > 0) {
                if (m_pFaceLandmarksPoller->Next(&faceLandmarksPacket)) {
                    face_landmarks_ = faceLandmarksPacket.Get<mediapipe::NormalizedLandmarkList>();
                } else {
                    std::cerr << "Failed to get next packet from poller." << std::endl;
                }
            } else {
                face_landmarks_.Clear();
                std::cerr << "No packets in queue." << std::endl;
            }
        } else {
            std::cerr << "Face landmarks poller is nullptr." << std::endl;
        }

Other info / Complete Logs

No response

[Tendaliu]

And when using MediaPipe to process video frames, the first frame always returns "No packets in queue."

[kuaashish]

Hi @Tendaliu,

Sorry for the delay in responding. Could you please let us know if this issue is still ongoing or if it has been resolved on your end?

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No