Interaction Teach

[XinandYu]

Hi, I notice you have fixed some codes. So I downloaded the latest code and found something wrong.

Could you please have a look at this problem?

[athn-nik]

Yes, sure. It seems that you haven't followed the instructions. Please read the README and download the smpl_models. Unfortunately, I cannot pack them in the repo or release them/make them automatically downloadable.

[XinandYu]

Sorry. But according to the instrction, there is only smpl_male. But it seems there is no update in the dataset processing. I can ru n the code on Dec but can't run after I download the new code this month again. Or did I make a mistake?

[athn-nik]

True using it now, I have pushed an untested fix.

[XinandYu]

Still the same error, is SMPL_NEUTRAL modal necessary in the interaction step?

[athn-nik]

Can you replace neutral with male in this file when the SMPL layer is called, line 77? If this work I can push the fix or you can make a pull request and I will accept it

[HappyPiepie]

I have a error when i get the result , the mesh out of playground, what can i do ?

[athn-nik]

From the beginning? I doubt so, I think I center the camera. I have no time these days to fix this but I can help you. Go to this file and call this method of MeshViewer inside the for loops that renders each image as mv.set_cam_trans(....trans=current_trans+body.mean()) that way it will move according to the mean of vertices.Access the intiial camera translation as mv.camera_node.translation. Hopefully, in that way it will follow the body.

In general, you have to fix the camera of pyrender to account for following the body.

Please please 🙏🏽 let me know about the fix and I cna incorporate it.

[HappyPiepie]

Thanks your reply,.The image occured in between, I use you provided model "teach", after i fix the error,i will tell you as fast as.

[athn-nik]

Let me know if it works. The high level bug is that I don't move the camera when the motion is happening so if you try long sequences with changes in translation eg 10 actions it is normal to happen. The fix I proposed moves the camera between actions or frames to follow the center of the body

[HappyPiepie]

my commond is python interact_teach.py folder=/home/beibeigh/Reserach/Projects/Projects_context/teach/path/to/experiment/teach output=/home/beibeigh/Reserach/Projects/Projects_context/teach/path/to/experiment/teach/test texts='["jump with left foot", "walk forward","sit down"]' durs='[2,2,2]', the mesh still go out

[athn-nik]

There should be something wrong with the camera or your update doesn't work. The key is to change the camera so that follows the mesh . The camera is pyrender camera inside meshviewer, I won't be able to push a fix until Tuesday Wednesday if you can wait. Otherwise this is the "bug". I suggest try tweaking the camera and see the result. If that doesn't work and urgently need it to work you can save generated I'm obj and visualize them with some code or interface it is easier for you to tweak

[ywdong]

Can you replace neutral with male in this file when the SMPL layer is called, line 77? If this work I can push the fix or you can make a pull request and I will accept it

I have change the code，but I got the below error. Any help~ Global seed set to 1234 [24/03/23 15:23:50][main][INFO] - Loading model [24/03/23 15:23:51][torch.distributed.nn.jit.instantiator][INFO] - Created a temporary directory at /tmp/tmpkc3t5dup [24/03/23 15:23:51][torch.distributed.nn.jit.instantiator][INFO] - Writing /tmp/tmpkc3t5dup/_remote_module_non_scriptable.py [24/03/23 15:23:52][main][INFO] - Model 'teach' loaded [24/03/23 15:23:54][main][INFO] - Model weights restored Global seed set to 0 Error executing job with overrides: ['folder=experiment/teach', 'output=output/teach', 'texts=[jump with left foot, walk forwards, sit down]', 'durs=[1.0, 1.0, 1.0]'] Traceback (most recent call last): File "/group/20000/weidongyang/pose-estimation/teach/interact_teach.py", line 108, in _interact() File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/main.py", line 48, in decorated_main _run_hydra( File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra run_and_report( File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report raise ex File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report return func() File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/_internal/utils.py", line 378, in lambda: hydra.run( File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/internal/hydra.py", line 111, in run = ret.return_value File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value raise self._return_value File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job ret.return_value = task_function(task_cfg) File "/group/20000/weidongyang/pose-estimation/teach/interact_teach.py", line 35, in _interact return interact(cfg) File "/group/20000/weidongyang/pose-estimation/teach/interactteach.py", line 103, in interact vid = visualize_meshes(motion) File "/group/20000/weidongyang/pose-estimation/teach/teach/render/mesh_viz.py", line 93, in visualize_meshes mv = MeshViewer(width=im_width, height=im_height, File "/group/20000/weidongyang/pose-estimation/teach/teach/utils/mesh_utils.py", line 104, in init self.viewer = pyrender.OffscreenRenderer(self.figsize) File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/pyrender/offscreen.py", line 31, in init self._create() File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/pyrender/offscreen.py", line 149, in _create self._platform.init_context() File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/pyrender/platforms/egl.py", line 177, in init_context assert eglInitialize(self._egl_display, major, minor) File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/OpenGL/platform/baseplatform.py", line 402, in call return self( args, **named ) File "/data/miniconda3/envs/teach/lib/python3.9/site-packages/OpenGL/error.py", line 228, in glCheckError raise GLError( OpenGL.error.GLError: GLError( err = 12296, baseOperation = eglInitialize, cArguments = ( <OpenGL._opaque.EGLDisplay_pointer object at 0x7fcfebdab9c0>, c_long(0), c_long(0), ), result = 0 )

[athn-nik]

Pff, opengl errors are always tricky, hmm, try these:

• set os.environ['PYOPENGL_PLATFORM'] = 'egl'
• setos.environ['DISPLAY'] = ':0.0' and os.environ['PYOPENGL_PLATFORM'] = 'egl'
• set os.environ['PYOPENGL_PLATFORM'] = 'osmesa'

Try each one separately and rerun your script and let me know. But this error is not on my side unfortunately 😞

[ywdong]

set os.environ['PYOPENGL_PLATFORM'] = 'osmesa' is ok for me~