Camera and zoom control

[andupotorac]

Hey there. This tool has an amazing potential to be used for OpenPose in diffusers. I'm wondering if the code allows controlling the camera angle, and zoom. So you could interface with the prompt to generate the movement, but also determine which angle and what zoom could be used when seeing the output.

[zhaowtVincent]

+1

[h-y1heng]

Hi, what you might want to achieve is to visualize the generated 3D motions from multiple perspectives and different zooms? This issue has nothing to do with our generation pipeline, but with 3D visualization. We save the generated motion results as relative xyz-coordinates (Y-up) in .npy file, which you can further transform to fit any world coordinate system, so you can see the output by rendering the 3D skeletons. Further, by adjusting the camera position, rotation and focal length during the rendering process, the visualization results of different zooms from multiple perspectives can be realized.

We recommend using Blender to do this, which we've successfully tried to modify the camera angle and zoom. Our easy way is to clone the mld repo, set up Blender and render the output directly, without generating a mesh if you don't ask for it. The code allows controlling the camera angle and zoom lies in motion-latent-diffusion-main/mld/render/blender/camera.py：

BTW, we're eager for future work to further enhance our methods. Feel free to share our work with others and encourage collaboration.

[andupotorac]

Thanks! Another question - how would your 3d model movements be passed to blender? As frames or?

[h-y1heng]

Thanks! Another question - how would your 3d model movements be passed to blender? As frames or?

Yes, because the generated output is an npy file of size (nframes, njoints, 3), when you run blender with a python script in mld，the skeleton joints will be processed frame by frame for rendering.

[andupotorac]

Got it, thanks a lot!