VPD pose import-export? (& BVH to VMD)

[Hogarth-MMD]

Hello @powroupi! How are you doing? I congratulate and thank you for all of your excellent work on MMD tools. Can you please add support for the import and export of .vpd (Vocaloid Pose Data) files? MMD tools already has support for import and export of .vmd files, so this should not be too difficult a task should it?

[stefnotch]

@powroupi This might help you (Chances are that you are already aware of its existence): https://github.com/ousttrue/pymeshio They already implemented the .vpd format.

[powroupi]

@stefnotch Thanks for the link, that helps. :smile:

@Hogarth-MMD Thank you. And yes, .vpd is not difficult to implement since it is a simple ASCII text file. :smile: I may implement it in the future when I have time. (I'm a little lazy currently, sorry) :blush:

[Hogarth-MMD]

Does anyone have an anti-laziness python script for @powroupi?

[powroupi]

Does anyone have an anti-laziness python script...

I definitely need it. :laughing:

Well, I will try to see if I can release it in 1 or 2 weeks. :)

[Hogarth-MMD]

LOL. I am a big fan of @powroupi. :-)

[Hogarth-MMD]

Hello @powroupi! This is an idea which I have for helping people to make their own original animations more quickly. A feature can be added to MMD tools which can dissect an animation into individual poses, and do a mass export of one .vpd pose for each keyframe of an animation, each .vpd pose being accompanied by a rendered preview image of the pose. By this method, people can create their own libraries of poses, and then use these poses to make new animations more quickly. 😆

[powroupi]

I think it is better to use blender's "Pose Library", and it is no good if we export too many unwanted pose file (.vpd). Blender provided many handy tools to make animations, so we should learn to use it. :smile:

I think the VPD importer/exporter will import/export armature's pose library from/to pose files (.vpd). :)

[Hogarth-MMD]

I just did an experiment. I loaded the "Glide" vmd animation into Blender using MMD tools. I made a little python script to count the number of keyframes in an animation. 2853 keyframes in Glide.

I am trying to puzzle solve a method to accelerate the making of an animation. But now I can see that doing a mass export of all 2853 frames of Glide to .vpd would not be particularly helpful. That would be too many poses. So now I am feeling disappointed that my latest idea has hit a snag. :-( If anyone has any ideas. please post them. Manually saving poses into a Blender pose library is a reasonable idea, but it would be slow and would not be an ideal solution either.

[stefnotch]

How about "important" poses. (A small subset of all of those poses.) Not sure about how you would pick them though... (randomly? Every 100th pose? When the difference is high enough?)

You could also try having poses/animations for individual limbs. e.g. A wave pose/animation (1 arm) or a walk animation, (legs) a stand pose (full body)

[ptrthomas]

Brainstorming here. I have been considering 2 ideas but don't have the time (possibly the skills also).

One is to manually identify a set of common patterns or dance "loops" that can be extracted out of existing VMDs. Normalize the speed. Then blend them together, something like this: https://docs.unrealengine.com/latest/INT/Engine/Animation/Blendspaces/

The other (crazy) idea I have is to feed a bunch of frame-synced videos corresponding to the VMD-s to train some machine learning tool and then in theory it would be possible to generate the VMD given a new video !

Somewhat un-related, but this may be additional inspiration for animation enthusiasts: https://www.youtube.com/watch?v=Ul0Gilv5wvY

[stefnotch]

"feed a bunch of frame-synced videos corresponding to the VMD-s to train some machine learning tool and then in theory it would be possible to generate the VMD given a new video !" <--What do you mean with that? Videos as in random videos from YouTube?

[ptrthomas]

@stefnotch take a look at this: https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation :)

I would start with MMD motion trace videos like this one: https://www.youtube.com/watch?v=DutSiCI6A4w (fixed camera, facing straight / forward)

If that works, I would "graduate" to training different camera angles. The ultimate would be a video with multiple camera angles being handled. But I agree random videos would be a stretch.

If it wasn't clear, the first phase of machine learning assumes you have a set of videos WITH the corresponding VMDs. Once the "training" is over is when the fun starts.

[powroupi]

Okay, I guess I have completed this feature, but I'm not good in making documents, sorry for my english. :worried: If you found any issue, please let me know. Thank you. :smile:

[Hogarth-MMD]

Thanks @powroupi, I will test this new feature and I will tell you what I think. I also want to announce it to other people on my deviantart account. Also thank you to @stefnotch and @ptrthomas. This was very interesting information about neural net animation and armature animation from motion tracking. I apologize to everyone for being so slow to reply. Unfortunately, I have been very busy and distracted. :-(

[Hogarth-MMD]

Hello @powroupi ! This is excellent work. I am really amazed at how talented you are, and how generous you are in giving your work to the world for free.

I found a few flaws with the vmd import/export:

1. With MMD tools, the armature is hidden by default. This is not a problem for me, but this is a problem for someone who is less expert with Blender.

2. A widget appears whose title is "Apply Pose Library Pose". This is an excellent feature. Blender does not allow scrolling through poses so easily. But later this widget disappears for some reason.

3. In MMD, a pose for arms and a pose for legs can be added to a model one after the other. But this does not happen when a pose is imported into Blender. Instead the entire pose is overwritten for all bones.

4. No preview image of a pose is exported. Of course this is not a bug, but this would be a very nice additional feature.

[ptrthomas]

@Hogarth-MMD I'm hijacking this thread a little - from your DeviantArt page I figure you have been experimenting with importing BVH (motion cap) files to MMD. Have you figured out a way to consistently do this in Blender (I'd like the final animation to be pre-viewable and final within blender). I'm also planning to write a script to re-target animations to MMD. May be worth comparing notes. I also haven't used pose-libraries and wonder if they are relevant. We seem to have the same goal of being able to create brand new MMD animations by stitching together bits scavenged from various sources.

[powroupi]

Thank you, @Hogarth-MMD. :smile: I really enjoy using this addon, so I'd be happy to share my changes to you all. :)

Reply to your suggestions:

1. Maybe it is better to show Mesh and Armature by default when importing a MMD model. :)

2. It is a blender's feature. In Pose Mode, you can go to menu [Pose]->[Pose Library]->[Browse Poses..] (or Ctrl+L) to scroll through poses easily. It will apply the pose to selected bones, or apply to all bones if none selected.

3. When using pose library, you might need to select the bones before applying a pose. It is different from MMD. :)

4. You might be interested:

   • add-on-pose-lib-preview-makes-pose-switching-easy-and-fast
   • https://github.com/jasperges/pose_thumbnails

[Hogarth-MMD]

Hello @ptrthomas . If you read through my comments in these 2 journal entries, you can see what I have figured out as far as converting bvh to vmd in concerned:

http://hogarth-mmd.deviantart.com/journal/2548-free-downloadable-BVH-files-631005592

http://hogarth-mmd.deviantart.com/journal/MMD-tools-can-now-export-VMD-files-644644129

Additionally, there is a Blender add-on called Motion Capture Tools, which can be used to re-target an animation from one armature to another armature.

https://wiki.blender.org/index.php/Extensions:2.6/Py/Scripts/Animation/Motion_Capture_Tools

https://git.blender.org/gitweb/gitweb.cgi/blender-addons.git/tree/HEAD:/mocap

(Click on "snapshot" to download the add-on.)

Several youtube tutorials, first one here: https://www.youtube.com/watch?v=_VhbSrau-0c

[nagadomi]

@ptrthomas @Hogarth-MMD

Live Animation supports importing BVH and exporting VMD. It can easily convert from BVH to VMD. But it loses some bone action. My current method is:

1. Import the BVH bone
2. Change MMD model's rest pose to T-pose (also, make the MMD bone's local axis match the BVH bone's local axis)
3. Bind BVH bone to MMD bone with bone constraint

That is a little hard work. It would be great if mmd_tools could import BVH directly.

[ptrthomas]

@nagadomi - hey thanks, I was also evolving a similar workflow !

in addition to the T-pose fix I was actually doing Pose-->Re-apply as Rest Pose and then re-applying the armature modifier - or will things work without doing that ?

one improvement to step 2 (for the bone axis fix) possibly is to use the Armature-->Bone Roll-->Recalculate Roll-->View Axis for both the source and target rigs. Is this what you were doing - or can you check if this works for you ?

I think all the above steps could possibly be made into a python script.

My initial experiments suggest that the Blender NLA editor works really well to "cut and paste" animations into a new one !

[nagadomi]

@ptrthomas I use corrective shape key addon to create T-pose shape key after manually setting T-pose and apply. My experimental blend file is here: http://www.mediafire.com/file/kwk9wj6eb8hv7q0/bvh_test_clean.zip (the blend file includes standard miku model and bvh from axis neuron, I will delete this file in a few days.)

[powroupi]

My method may not be simple, just for your reference:

1. Import the BVH bone (change rotation conversion to Quaternion, and proper scale to fit MMD model size)
2. Pose the BVH model to fit MMD model's rest pose, and fill corresponding Japanese names of MMD bone property (mmd_bone.name_j). (in general, only need to rotate some bones and move root bone, ignore bone roll)
3. Export VMD file of BVH model (enable "Treat Current Pose as Rest Pose")
4. Import VMD file to MMD model (you may need to disable IK constraints), and check the result.

This may take some time to experience, I don't know how to explain that, sorry for my english. :cry: Once you satisfied with the result, the Step 2. can be made into a python script for other BVH files which have the same bone structure, and exported VMD should work for most MMD models. :smile:

[ptrthomas]

Thanks everyone for a great discussion !

@nagadomi - I got the file, thanks. Here is my approach for the T-pose fix (without any add-on)

• after setting the T-pose, switch to the mesh and Object mode
• in the Modifiers tab (spanner icon), click "Copy", a duplicate of the "mmd_bone_order_override" will be created called "Armature"
• click "Apply as Shape Key" for the second / duplicate modifier
• go to the Data / Shape Keys tab (triangle icon) and there will be an "Armature" shape key as the last one in the list. Set the value to 1

So I think you don't need the corrective shape key addon. Wow so do you have an Axis Neuron ? I also have a suggestion where I got good results (on TDA models) by binding the IK to the "Foot" and "Toe" bones instead of using rotations on the leg, knee, ankle etc.

@powroupi this is a useful reference, thanks !! One advantage I see if you bind an MMD rig to the BVH (which is what @nagadomi and myself are doing) is that now you can keep importing other BVH motions and use the NLA editor to switch motions.

I have a question - how do you "ignore bone roll" ?

[nagadomi]

powroupi's method is that I do not need to edit the MMD model side, it's an advantage, I think. I usually use multi armature modifier, so it cannot create correct shape keys from a single armature modifier.

[ptrthomas]

@nagadomi good point. but when trying to use multiple BVH files, I think it is also an advantage to not have to export every single BVH to VMD. I can create a complex animation in Blender, and then bake the final animation (constraints to keyframes) and render to VMD (or FBX in my case).

@powroupi do provide some tips on Bone-Roll ! I was also not converting the BVH to Quaternion - what are the advantages ?

[Hogarth-MMD]

The Blender bvh importer has a bug. If you try to import a bvh file which has Japanese bone names, it won't import into Blender, and you get an error message. This simple fix worked for me:

Open this file (from the Blender program folder) in a text editor, such as Notepad++: scripts\addons\io_anim_bvh\import_bvh.py

Change line 97 from: file = open(file_path, 'rU')

to: file = open(file_path, 'rU', encoding = 'utf-8')

[powroupi]

Okay, here is detailed information of my method:

1. Import BVH file, change import option:

   • Scale: proper scale to fit MMD model size. I usually set to 0.024 (to fit MMD model's default import scale 0.2).
   • Rotation: change rotation conversion from "Euler (Native)" to "Quaternion" (VMD exporter only support "Quaternion" type)
   • Other options are optional, you can change them as you like. I usually enable "Scale FPS" to fit scene's frame rate.

2. Pose the BVH model to fit MMD model's rest pose.

   • In general, only need to rotate some bones and move root bone, bone roll is not important. And do not scale bone, it does not work as expected when exporting VMD.
   • It is not necessary to fit perfectly, I sometimes only rotate the upper arms along global Y-axis.
   • You may add the pose to pose library for later use (in case that you don't like the result, you will need to adjust the pose).

3. Fill corresponding Japanese names of MMD bone property (mmd_bone.name_j).

   • Copy mmd_bone.name_j from MMD model's bone to BVH model's bone one by one, depends on how you want to map the bones.

4. Export VMD file of BVH model, change export option:

   • enable "Treat Current Pose as Rest Pose"

5. Import VMD file to MMD model, optionally change import option:

   • I usually set Margin: to 0

6. Disable IK constraints of MMD model, and check the result.

   • If the result does not look good, go to step 2.

A simple tutorial for you to follow up:

1. download "Zip file for BVH directories 86-94" of "PRIMARY RELEASE: Daz-friendly hip-corrected BVH files release."

2. I'll take folder 94 for example, here is sample 94_sample_pose_and_script.zip (pose "94.vpd" & script "94.py") for step 2 above. The pose will explain how to fit the rest pose.

3. Open blender, change scene's frame rate from 24fps to 30fps, and follw my method above:

   1. Import BVH "94_01.bvh", change import option:
      • Scale:: change from 1.0 to 0.024
      • Rotation:: change from "Euler (Native)" to "Quaternion"
      • optionally enable "Scale FPS" & "Update Scene Duration"
   2. Import VPD "94.vpd" to BVH model, change import option Bone Mapper from "PMX" to "Blender"
   3. In blender's text editor, open text file "94.py", hit run script button to fill corresponding Japanese names of MMD bone property.
   4. Export VMD file of BVH model, enable export option "Treat Current Pose as Rest Pose"
   5. Import MMD default model "初音ミクVer2.pmd" or standard TDA Miku Append model, use default import scale 0.2, then import VMD (change Margin: from 5 to 0) to that MMD model.
   6. Disable IK constraints of MMD model, and check the result.

[Hogarth-MMD]

I have figured out a few more things about how to convert CMU bvh files to vmd:

1. The "hip-uncorrected" bvh files are more MMD-compatible than the "hip-corrected" bvh files. "Hip-corrected" means that the resting pose (Blender edit-mode pose) of the hip bone will be located at the world origin(0,0,0). These BVH files have been "corrected" to make them compatible with DAZ characters whose rest pose hip bone location is at the world origin(0,0,0). MMD models do not have their center bone located at the world origin, however, so the "hip-uncorrected" BVH files are more compatible with MMD models.

2. The CMU bvh files have a frame rate of 120 FPS (120 frames per second). VMD animations normally have a frame rate of 30 FPS. To correct the frame rate, you need to manually set the frame rate of your Blender scene to 30 FPS. When you import the BVH file, you need to enable the Scale FPS option to change the frame rate from 120 FPS to 30 FPS.

3. In the CMU BVH animations, the character's rest pose (edit mode pose) is a T pose with the arms outstretched at a 90 degree angle from the body. MMD models have the arms rotated downward at a 45 degree angle in an A pose (or collar bones rotated down 5 degrees and shoulder bones rotated down 40 degrees). To convert the animation from being T-pose-compatible to being A-pose-compatible, before you export the BVH animation to VMD, go to the first frame of the animation and rotate the character's left arm 45 degrees, and rotate the character's right arm -45 degrees. When you export the animation to VMD, enable this export option: "Treat Current Pose as Rest Pose".

[Hogarth-MMD]

If I am really, really lucky, correcting bone roll, is the LAST issue that I need to understand in converting BVH to VMD. I don't clearly understand what bone roll is. In Blender 2.78,, in edit mode, there is a Clear Bone Roll menu item. After clearing bone roll on all bones, I don't really feel sure if I have corrected the animation or if I have messed it up. Does anyone have any clues or suggestions about this aspect?

[ptrthomas]

BTW - in the future if anyone searches for MMD, BVH etc, I think this page is going to be the first search result :stuck_out_tongue_winking_eye:

@Hogarth-MMD I mentioned this above, did you get a chance to try this (see below) - it seemed to fix the bone-roll problem for me. Yes, clearly setting bone-roll to 0 on everything is not a solution.

Do this for both the source and target rigs, I did this only for the shoulder bones onwards - but I have a feeling it may work for the entire rig. In my case I did go through the process of setting the MMD model into T-pose before retargeting.

Armature-->Bone Roll-->Recalculate Roll-->View Axis

[Hogarth-MMD]

I think that "bone roll" has something to do with how the X and Z local axes of a bone align with the X and Z world axes(?).

@ptrthomas Thanks, I'll try this suggestion: Armature-->Bone Roll-->Recalculate Roll-->View Axis .

[Hogarth-MMD]

Here is a Blender manual page that talks about bone roll: https://docs.blender.org/manual/ja/dev/rigging/armatures/bones/editing/transform.html

[Hogarth-MMD]

OK, here is what I have figured out about bone roll:

1. I did this experiment: I imported the CMU 08_09.bvh file into Blender. The result is a very normal-looking walk animation.

2. I cleared the bone roll on all bones. The result is a Frankenstein-looking walk animation with the character's arms stretched out in front of him.

3. I then re-imported the same bvh animation and I tried recalculating the animation according to every available option. In every case the result was a weird messed-up walking animation.

Based on that experiment, it looks like the bone roll should not be re-calculated or cleared.

If someone wants to better understand these bone roll modifications, you can enable viewing of the axis of bones and use the octahedron display of bones to see visually which direction the bones are facing.

In MMD, as far as I can tell, the x,y and z axis of a bone is always aligned with the world x,y and z axis. You can see this in MMD, if you select one of a character's bones and hover your mouse over one of the rotation widgets. Red, green and blue axis lines will appear on the bone.

[Hogarth-MMD]

Here is a download link to my new Blender python bones-renaming script (CMU_BVH_to_MMD.py) to rename bone names from CMU bvh to MMD Japanese bone names:

http://sta.sh/01acutxl5g0i

Please tell me if you find any flaws in it which need to be corrected. It is a standalone script, not an add-on, so you need to run it from the Blender text editor. I think that you need to enable javascript in your web browser to download it.

[Hogarth-MMD]

How tall is Miku???!!!
A CMU bvh file is imported with a height of 153 Blender units. That would be a reasonable height for a character if the units are centimeters. On average, an MMD model is about 20 units tall. In MMD, a grid square is 5X5. I don't think that Miku is supposed to be 20 meters tall, so what units are being used with MMD?

(Several people have asked me to make a BVH to VMD tutorial and I'm making a valiant effort at it.-) )

[Hogarth-MMD]

Hatsune Miku's official height is 158 cm (about 62 inches). So 1 MMD unit is 7.854 centimeters or about 3.082 inches. I am ignorant about Japanese units of measurement. Japanese people are logical people. So I am curious to know what is this mysterious Japanese unit of measurement?

[nagadomi]

in japan, mm, cm, m, km are commonly used as units of length. 10mm = 1cm, 100cm = 1m, 1000m = 1km. it is easy to calculate unit transformation. also, 158cm is often used as the height of a common female character because 158cm is the average height of a 17 year old female in japan.

[Hogarth-MMD]

Hello @powroupi! Thank you for all your excellent help and excellent programming (and tolerance of wandering off topic by people. 😆 ) ! Can you please add an option to the VMD export panel to automatically add an MMD-compatible A-pose as the rest pose? I am trying to write a tutorial for BVH to VMD and this seems to be the step which is most likely to be a problem for someone who is not experienced with using Blender. This step is complicating my tutorial-writing. And this wastes time for everyone who wants to convert BVH to VMD.

[Hogarth-MMD]

I have a new idea about how to implement BVH to VMD. Instead of sending my bone-renaming script to people as a stand-alone script, it can be added as an option to Blender's BVH importer. Automatic scaling to 30 FPS can be another option. All of the default BVH import options can be exactly what they need to be for BVH to VMD. BVH importer also bug fixed to allow import of Japanese bone names. VMD exporter has A-pose option. In this case we have extremely simple instructions and user-friendliness. "Import BVH. Don't change import options. Export VMD. Don't change export options."

[powroupi]

Hmm..., I think I may not want to add any option in order to convert BVH :expressionless:, because it is not the only way to do that. And it depends on how the rest pose of a BVH looks like and the bone name mapping.... As far as I know, some BVH files have different scale(size) and there is not only one standard rest pose, so we can't just convert it without changing any options.

As I said, you could make a script to convert specific BVH files, but extra adjustments are always needed to convert other BVH files which have different bone structure or scale. Understanding the steps would be better than just one button click, I think. :sunglasses:

[Hogarth-MMD]

Hello @powroupi! Okay, I understand that there are different bone mappings and scales of bvh files, but at the beginning, we can add conversion options for the CMU BVH files, and then gradually add other compatibility settings for other collections of BVH files, and someone can choose these from a menu in the import options panel. In every journey there is always a first step. The important thing is to make a beginning. I can tell you that many people are wanting to have this BVH to VMD conversion.

And maybe with some clever programming an A-pose can be obtained which does not depend on the specific rest pose of a BVH file.

[Hogarth-MMD]

Does anyone know of any other interesting collections of BVH files? I already know about the CMU collection of 2548 free BVH files. If we can start looking at some other interesting collections of BVH files, maybe we can see that some fun and exciting animation possibilities will be waiting for us when we have completed the work of programming BVH to VMD.

[ptrthomas]

I didn't try downloading this yet: http://www.youtube.com/watch?v=s1jREX_31Ig

[Hogarth-MMD]

Thank you @ptrthomas , Here is the direct download link to the 500 free Truebones BVH files (642 MB). http://www.truebones.com/500Free.rar

[Hogarth-MMD]

Blender has module called mathutils. mathutils.Vector has all the functions which are needed to calculate the angles of the edit mode collar bones and shoulder bones. Making an A pose which does not depend on the rest pose should be a fairly simple programming task, I think. The MMD standard Japanese bone names would be adequate, I think. How are you doing @powroupi ? All the time people are saying to you: "Please do this. Please do that." :-) Anyways when you have enough time, maybe you can do this. Otherwise, maybe someone else will do it.

[ptrthomas]

Quick question @Hogarth-MMD - I have been working with the TrueBones BVH-s and have not even looked at the CMU ones. Did you get a chance to see how different they are ?

[powroupi]

@Hogarth-MMD I think making the steps into a script will not give you a good performance on converting BVH files. I guess you need a converter, and here is a converter http://atupdate.web.fc2.com/ (Japanese) that you might be interested. :smile:

Or maybe you can change the computer time to 2020 or more, and do a future search, then you may find a new BVH-VMD converter. :kissing_heart:

[Hogarth-MMD]

Hi @ptrthomas . The only way to answer your question is for you to look at the files yourself, I think. Both collections of bvh files include a listing of the files that tell you what types of motions they are. The Truebones bvh files have more professional cleanup done on them, I think.

[Hogarth-MMD]

I am using MMD so much, and I love MMD so much, I would really like to be able to fluently read all of the Japanese bone names and Japanese morph names, both transliteration/pronunciation and translation. Can anyone offer any help with that? This would be helpful for programming also.

[ptrthomas]

Have you tried importing a model with the translate to english enabled ?

Take a look at this file if it helps (contributed by yours truly): https://github.com/powroupi/blender_mmd_tools/blob/dev_test/mmd_tools/translations.py