When I run this package on my video,have some error

[JH95-ai]

Error is QOpenGLContext::swapBuffers() called with non-exposed window,behavior is undefined QOpenGLContext::swapBuffers() called with non-exposed window,behavior is undefined and failed to update video frame information

[kaufManu]

Hi @NeekHua,

These error messages are sometimes printed at startup, but they don't usually have any noticeable negative side-effect. Could you provide a bit more information what is the problem exactly that you are experiencing? What does failed to update video frame information mean exactly? Are you running any of the examples that are not working? If you could provide sample code and may be some screenshots if required that would help us a lot to debug your problem.

Thanks!

[JH95-ai]

@kaufManu Hi,thanks your reply. I want to display SMPL model mesh results online.and I added this module after the pose parameter.

def main(){
   --------pretreatment code---------
    for frame in frames:
         pose,shape=inference(frame)
         seq=SMPLSequence(smpl_layer,poses_body=pose,betas=shape)
         v=Viewer()
         v.scene.add(seq)
         v.run()   
}

and then run it ,I find this module only show first frame SMPL parameters results, and don't update results.What should I do? Looking forward to your reply

[kaufManu]

I see - the interactive viewer was not designed to be used like this, rather it is meant to be used as in the following example:

poses, shapes = [], []
for frame in frames:
   pose, shape = inference(frame)
   poses.append(pose)
   shapes.append(shape)
seq = SMPLSequence(smpl_layer, pose_body=np.concatenate(poses, axis=0), betas=np.concatenate(shapes, axis=0)
v = Viewer()
v.scene.add(seq)
v.run()

If you would like to render results within a loop, it would be better to use the HeadlessRenderer, which you can use to save images like so:

from aitviewer.headless import HeadlessRenderer
r = HeadlessRenderer()
for frame in frames:
   pose, shape = inference(frame)
   seq = SMPLSequence(smpl_layer, pose_body=np.concatenate(poses, axis=0), betas=np.concatenate(shapes, axis=0)
   r.scene.add(seq)
   r.save_frame(...)
   r.scene.remove(seq)

This should work but it is still not the most efficient thing to do if you just want a video of the final result - it's more efficient to first build the entire sequence and then visualize the whole thing (either with the interactive viewer or the headless renderer). Does that help you? Feel free to also take a look at the examples here: https://github.com/eth-ait/aitviewer/tree/main/examples

[JH95-ai]

@kaufManu Thanks your reply , This two method is Ok.But I hope show 3d result per frame,such as opencv imshow per frame 2d images, open3d or unity show per frame 3d mesh results,I don't know aitviewer package whether support .

[kaufManu]

Unfortunately, we currently do not support this use case out-of-the-box. The reason is that when you call viewer.run() on an interactive viewer instance, this is a "blocking" call, i.e. you won't get control back until you close the viewer window. We are however looking into this and hope that we can support this better in the future.

Having said that, I think it is already possible to get the functionality that you are looking for using the headless renderer. With the headless renderer, you can get the current frame as an RGB image and display it in an OpenCV window.

If you know the entire SMPL sequence beforehand, this code should do it (adapted from the headless rendering example).

import cv2
import os
import numpy as np

from aitviewer.configuration import CONFIG as C
from aitviewer.renderables.smpl import SMPLSequence
from aitviewer.headless import HeadlessRenderer

if __name__ == '__main__':
    # Load an AMASS sequence.
    smpl_seq = SMPLSequence.from_amass(
        npz_data_path=os.path.join(C.datasets.amass, "ACCAD/Female1Running_c3d/C2 - Run to stand_poses.npz"),
        fps_out=60.0, name="AMASS Running", show_joint_angles=True)
    smpl_seq.color = smpl_seq.color[:3] + (0.75,)  # Make the sequence a bit transparent.

    # Create the headless renderer and add the sequence.
    v = HeadlessRenderer()
    v.scene.add(smpl_seq)
    v.lock_to_node(smpl_seq, (2, 2, 2), smooth_sigma=5.0)

    for f in range(v.scene.n_frames):
        v.scene.current_frame_id = f
        img = v.get_frame()
        img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
        cv2.imshow('Frame', img)
        cv2.waitKey(333)

If you don't know the entire sequence beforehand, you can still do it. It's not much more complicated, you just have to add a single-frame SMPLSequence to the scene and then updated it from the outside in your for loop:

import cv2
import os
import numpy as np
import torch

from aitviewer.configuration import CONFIG as C
from aitviewer.models.smpl import SMPLLayer
from aitviewer.renderables.smpl import SMPLSequence
from aitviewer.headless import HeadlessRenderer

if __name__ == '__main__':
    # Load some data to simulate that we only have one frame available at a time.
    npz_data_path = os.path.join(C.datasets.amass, "ACCAD/Female1Running_c3d/C2 - Run to stand_poses.npz")

    body_data = np.load(npz_data_path)
    smpl_layer = SMPLLayer(model_type='smplh', gender=body_data['gender'].item(), device=C.device)

    poses_root = body_data['poses'][:, :3]
    poses_body = body_data['poses'][:, 3:3+smpl_layer.bm.NUM_BODY_JOINTS*3]
    betas = body_data['betas'][np.newaxis]
    trans = body_data['trans']

    # Create a SMPLSequence with just one frame.
    smpl_seq = SMPLSequence(smpl_layer=smpl_layer,
                            poses_root=poses_root[0:1],
                            poses_body=poses_body[0:1],
                            betas=betas,
                            trans=trans[0:1],
                            z_up=True)

    # Create the headless renderer and add the sequence.
    v = HeadlessRenderer()
    v.scene.add(smpl_seq)

    for f in range(poses_root.shape[0]):
        # Load the next available frame data and update the SMPL sequence.
        smpl_seq.poses_body = torch.from_numpy(poses_body[f:f+1]).float().to(C.device)
        smpl_seq.poses_root = torch.from_numpy(poses_root[f:f+1]).float().to(C.device)
        smpl_seq.trans = torch.from_numpy(trans[f:f+1]).float().to(C.device)
        smpl_seq.redraw()

        # Get the frame and display it in an OpenCV window.
        img = v.get_frame()
        img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
        cv2.imshow('Frame', img)
        cv2.waitKey(333)

[JH95-ai]

@kaufManu Thanks your detail reply,I get it