I need driving characters in Unreal Engine

[SlimeVRX]

Hello @vivi90

I read your comment

Link: https://github.com/Arthur151/ROMP/issues/172#issuecomment-1214918118

I need driving characters in Unreal Engine

Can you support me?

Thank you very much!

Best regards,

Tin Nguyen Thanh

[SlimeVRX]

Hi @vivi90

I want to tell you about the research I'm working on. Research on posture estimation from 6 sensors. After releasing the V1 version based on the TransPose article, now I have released the V2 version based on the PIP article, with many improvements, and smoother motion.

This is my latest test in Unity

Link 1: https://drive.google.com/file/d/1ZTnN37yazS60ipiwx6R2VwaiZ3wDMCw9/view Link 2: https://drive.google.com/file/d/19UHiG1n5cqElEofuXWPYcXClD5GKmPqG/view

Data is processed in Python and then sent through Unity using Socket library The position of 24 skeletons is transferred to the Unity-chan model

Currently, I want to create a plug-in to send skeleton data from Python to Metahuman, can you help me? I am familiar with Python language I'm not familiar with Unreal, I started learning how to use Unreal I can send Python and Unity code for your reference.

Please help me!

Best regards,

Tin Nguyen Thanh

[vivi90]

@SlimeVRX See: https://codeberg.org/vivi90/python-vmcp/issues/2#issuecomment-660073

[SlimeVRX]

Hi @vivi90 I transfer data from Python to Unreal, here is the video. Thanks a lot Video: https://drive.google.com/file/d/1Z1MuacdF-hRbXLZxwUz5q2VpDhdEbHsJ/view Models in order are Metahuman, SMPL, FBX

[vivi90]

@SlimeVRX

Video: https://drive.google.com/file/d/1Z1MuacdF-hRbXLZxwUz5q2VpDhdEbHsJ/view

Nice. This motions seems to be fluent, but the overlay says 2 FPS?

PS: Sorry for editing your comment by accident xD

[SlimeVRX]

Hi @vivi90

I want to thank you very much. The hardest part of the job is sending Python test data to Vseeface using the VMC Protocol. I used your code.

Do you know any way to send data directly from Python to Unreal? (do not use VseeFace)

Best regards,

Tin Nguyen Thanh

[vivi90]

@SlimeVRX

Do you know any way to send data directly from Python to Unreal? (do not use VseeFace)

See: https://codeberg.org/vivi90/python-vmcp/issues/2#issuecomment-663820

[SlimeVRX]

Sorry @vivi90
I just now read your response on codeberg.org. I will reply on that page.

[SlimeVRX]

Hi @vivi90

VMCP worked, see more on codeberg.org

[SlimeVRX]

Hi @vivi90

I was able to pass the SMPL skeleton data into Unreal. I want to continue hand data transmission. Can you help me develop?

I created an issue here: https://codeberg.org/vivi90/python-vmcp/issues/4

[zhanghongyong123456]

Hi @vivi90

I want to thank you very much. The hardest part of the job is sending Python test data to Vseeface using the VMC Protocol. I used your code.

Do you know any way to send data directly from Python to Unreal? (do not use VseeFace)

Best regards,

Tin Nguyen Thanh

I tried to use vmcps project and VMC4UE plug-in to drive a character, but this problem occurred. What should I do? I think it might be Bone Position Synchronize, but I could not find this option in UE. What should I do and look forward to your reply

[SlimeVRX]

Yes, I also got the same error. The workaround is that you need to retarget motion to another character, the retargeted character will not fail

[zhanghongyong123456]

Yes, I also got the same error. The workaround is that you need to retarget motion to another character, the retargeted character will not fail

I don't understand what you mean. Could you explain how to solve it，Thank you very much for your reply， for use VMC4UE i using unity get VMC4UEVRMMapping ， like this video https://www.youtube.com/watch?v=2h2wkowg0oA

So you said to change the role, I don't quite understand how to operate, can you give me the implementation details or pictures

[SlimeVRX]

Yes, You do right.

You need to retarget from the current character to another character. (current characte is error, another character is not error)

You can see retargeting from character to character here

[zhanghongyong123456]

Yes, You do right.

You need to retarget from the current character to another character. (current characte is error, another character is not error)

You can see retargeting from character to character here

The video I refer(https://www.youtube.com/watch?v=2h2wkowg0oA) to also works like this, and the result is wrong. i use this projct(https://codeberg.org/vivi90/vmcps not use other project(https://codeberg.org/vivi90/python-vmcp)) get data

[zhanghongyong123456]

I use VSeeFace get date it‘s ok，when use vmcps get data have this error,(You need to retarget from the current character to another character) i think i have do it Select the correct mesh for Mapping matching here

[vivi90]

I use VSeeFace get date it‘s ok，when use vmcps get data have this error,(You need to retarget from the current character to another character) i think i have do it Select the correct mesh for Mapping matching here

Thanks for your solution 👍 May you please also add an english screenshot? Since your Unreal UI is in chinese and i want to add some notes to the readme to help also other users.

[zhanghongyong123456]

1. if bev can be used to get data (instead of romp)

Should work with some small modifications. You are free to edit the code.

2. and can multiple drivers be used (more than one person drives)

If you mean tracking of multiple persons: Yes, romp does support this. But the VMC protocol not. To do this, you require multiple VMC protocol channels. My vmcp package supports this. The vmcps example script supports this also with some small modifications. But it depends on the implementation of your used Unity/Unreal plugin. Not tested yet.

[SlimeVRX]

2. can multiple drivers be used (more than one person drives).

I am also interested in this, is it possible? multi-input for multi-character

[zhanghongyong123456]

2. can multiple drivers be used (more than one person drives).

I am also interested in this, is it possible? multi-input for multi-character

HI , Can you tell me about your implementation process in detail? I am still stuck on this question

[vivi90]

@zhanghongyong123456 Sorry for editing your comment 🥺 Was an mistake, while writing on my mobile phone.

@zhanghongyong123456 @SlimeVRX

1. if bev can be used to get data (instead of romp)

Should work with some small modifications. You are free to edit the code.

2. and can multiple drivers be used (more than one person drives)

If you mean tracking of multiple persons: Yes, romp does support this. But the VMC protocol not. To do this, you require multiple VMC protocol channels. My vmcp package supports this. The vmcps example script supports this also with some small modifications. But it depends on the implementation of your used Unity/Unreal plugin. Not tested yet.

[zhanghongyong123456]

@zhanghongyong123456 Sorry for editing your comment 🥺 Was an mistake, while writing on my mobile phone.

@zhanghongyong123456 @SlimeVRX

1. if bev can be used to get data (instead of romp)

Should work with some small modifications. You are free to edit the code.

2. and can multiple drivers be used (more than one person drives)

If you mean tracking of multiple persons: Yes, romp does support this. But the VMC protocol not. To do this, you require multiple VMC protocol channels. My vmcp package supports this. The vmcps example script supports this also with some small modifications. But it depends on the implementation of your used Unity/Unreal plugin. Not tested yet.

OK thank you ,When I implement the UE project, I will write a UE related process for your readme file

[zhanghongyong123456]

Yes, You do right.

You need to retarget from the current character to another character. (current characte is error, another character is not error)

You can see retargeting from character to character here

i use yours video, VSeeFace can use,but use vmcp have error,Look forward to your help, thank you

[SlimeVRX]

You need to watch the next video tutorial 02 -> 03. Here

Follow the instructions of this video, retarget motion from Vroid model to Metahuman, the Vroid model will fail, but the Metahuman model will not fail.

[zhanghongyong123456]

You need to watch the next video tutorial 02 -> 03. Here

Follow the instructions of this video, retarget motion from Vroid model to Metahuman, the Vroid model will fail, but the Metahuman model will not fail.

Instead of matahuman, how do you get VRMMapping，

[SlimeVRX]

Your VMC4UE is working correctly, please skip it!

Continue with Retarget, Retarget motion from Vroid to Metahuman (or another character)

Metahuman (or another character) will work

[zhanghongyong123456]

Your VMC4UE is working correctly, please skip it!

Continue with Retarget, Retarget motion from Vroid to Metahuman (or another character)

Metahuman (or another character) will work The replacement of matahuman is still wrong, I think it is the problem of data mismatch, but I do not know how to modify it

[SlimeVRX]

Are you sure to follow video tutorial 03? Retarget motion from Vroid to Metahuman

[zhanghongyong123456]

Are you sure to follow video tutorial 03? Retarget motion from Vroid to Metahuman

It is according to this operation, I use VSeeFace is working, switch to vmcp there is an error,

[zhanghongyong123456]

Are you sure to follow video tutorial 03? Retarget motion from Vroid to Metahuman

It is according to this operation, I use VSeeFace is working, switch to vmcp there is an error, Can you help me remotely? I've been stuck here for a long time

[SlimeVRX]

Can you show me your skeleton retarget 2 mapping file? I want to see if retarget has any errors?

[SlimeVRX]

Like image

[zhanghongyong123456]

Can you show me your skeleton retarget 2 mapping file? I want to see if retarget has any errors?

can you download todesk to Control my computer ， My ID 643391818 password: 008763

[SlimeVRX]

ok

[zhanghongyong123456]

ok

thank you very much

[SlimeVRX]

Please check again anydesk, I don't find your computer

[zhanghongyong123456]

Please check again anydesk, I don't find your computer

小四邀请您进行远程控制 ToDesk设备代码:643 391 818 临时密码:008763 点击链接直接进行远程控制： https://wechat.todesk.com/invite-page?id=5rLAe4CEL1Y5DZlaNfjD-

[SlimeVRX]

Sorry, I don't connect

[SlimeVRX]

Sorry, I confused todesk and anydesk

[zhanghongyong123456]

Sorry, I don't connect

Maybe it's a regional issue, and my colleagues can connect

[SlimeVRX]

Sorry, I don't have number phone China to verify ToDesk

[zhanghongyong123456]

Sorry, I don't have number phone China to verify ToDesk

Do you have any other ways to help me，can you give me your test project

[SlimeVRX]

Like image

Please check again

[zhanghongyong123456]

Like image

Please check again

it‘s ok， can you upload a UE test project on your github

[SlimeVRX]

You can connect with my friend in China, I hope he can help you

[zhanghongyong123456]

You can connect with my friend in China, I hope he can help you

OK,you have QQ ? Maybe remotely via QQ

[SlimeVRX]

I have WeChat

[zhanghongyong123456]

I use VSeeFace get date it‘s ok，when use vmcps get data have this error,(You need to retarget from the current character to another character) i think i have do it Select the correct mesh for Mapping matching here

Thanks for your solution 👍 May you please also add an english screenshot? Since your Unreal UI is in chinese and i want to add some notes to the readme to help also other users.

Implement UE-driven MetaHuman step process, here, you can see what needs to be modified https://codeberg.org/vivi90/vmcps/issues/3

[SlimeVRX]

Nice work!

[zhanghongyong123456]

Nice work! thank you very much for your help， But there are a few problems: 1) fps is very low, how to improve it, 2) how to replace the romp model with the bev model; 3. How to deal with multiple people

[SlimeVRX]

1. I don't check FPS. My test video
2. Output romp is SMPL, you can replace any model SMPL output
3. Support multi people: Please ask @vivi90

   If you mean tracking of multiple persons:
   Yes, romp does support this.
   But the VMC protocol not.
   To do this, you require multiple VMC protocol channels.
   My vmcp package supports this.
   The vmcps example script supports this also with some small modifications.
   But it depends on the implementation of your used Unity/Unreal plugin.
   Not tested yet.