search

makehumancommunity / mpfb2

MPFB2 is a free and open source human generator for Blender
http://static.makehumancommunity.org/mpfb.html
Other
315 stars 37 forks source link

Ability to include weights and maybe even extension rigs with proxies and clothing. #64

Closed angavrilov closed 1 year ago

angavrilov commented 1 year ago

I think it may make sense to allow proxies and clothes (since they are basically the same thing underneath) to include some rigging information.

Tweaked Weights

Instead of interpolating weights from the base mesh, a proxy/clothing could optionally include a weight file with manually polished weighting. I think the file probably should be specific to a certain built-in rig, using a similar naming convention to the base mesh, e.g. weights.default.json for the default non-rigify rig, or weights.rigify.human.json for the default rigify rig.

I'm not sure if any attempt should be made to support a 'generic' weight file, or if that is best to be handled via interpolation. After all, the goal here is manual tweaking of weights. Maybe the explicit files should even be applied on top of interpolated weights, so you can only include changed values there?..

Extension Rigs

Allow including an extra rig to e.g. handle freely moving detail parts of clothing or hair etc not covered by the main rig, together with weights for it.

The extra rig probably should be kept separate from the main rig, so it's necessary to think how to connect it to the main rig (with constraints, I imagine), and also an appropriate double armature constraint setup.

Development

I'm thinking about trying to implement this and want to discuss design and implementation approaches. E.g. the rig and weight import/export tools are aimed purely at main rig and base mesh, and various find rig by mesh and vice versa utilities won't understand anything.

angavrilov commented 1 year ago

So I made a branch implementing this idea: https://github.com/angavrilov/mpfb2/commits/asset-subrigs

I also made a sub-rig for the standard ponytail asset as a test: ponytail01.rig.zip: the json files should be unpacked next to the existing mhclo file. Mind you, I didn't spend a lot of effort on weights so it goes kinda wobbly when you bend it - probably needs more weight smoothing passes.

joepal1976 commented 1 year ago

Thanks, this looks very interesting.

Actually makehuman has some partial support for this via MHW ("mh weights") files earlier, but since it wasn't used a whole lot I haven't gotten around to porting it to MPFB.

I think it's a very good idea to add the functionality though, be it MHW or not.

angavrilov commented 1 year ago

@joepal1976 So I wonder, is the scheme I used of looking up files based on the mhclo file name without opening it appropriate, or should I add entries to mhclo referencing the files like was done with that mhw thing?

The following file names are currently supported in the branch, as associations for NAME.mhclo:

Weights are applied on top of interpolated weights in the order listed, with full group replacement, i.e. any group present in one of the files completely replaces the whole group contents, instead of overwriting just individual vertex weights.

Re MHW it seems to be essentially the same as weight json files used by MPFB, just with a different extension?

angavrilov commented 1 year ago

@joepal1976 I changed the weight file names for assets to use the MHW extension, and also hard-coded a rule for converting weights with toes to no-toes on the fly: while MPFB doesn't have the full generic weight conversion system of MH, it can at least do this trivial difference to reduce the number of required weight files.

I also looked at mhskel files, but they are radically different and can't be used with MPFB. So what do you think about introducing a 'mpfbskel' file or something, that basically combines current rig.json and rigify_layers.json into one?

angavrilov commented 1 year ago

Over the weekend I went on and implemented a mpfbskel file format that combines rig.json and rigify_layers.json into one structure. It's completely incompatible with mhskel, hence the different file name.

I also implemented a new UI panel for conveniently changing bone strategy settings through the UI instead of manually editing the json: https://github.com/angavrilov/mpfb2/commit/1bb673b223a3f0c611e47e1e179eac3b682e8f56

@joepal1976 I'm thinking about making a documentation page describing how to make the rig for the ponytail target from scratch, with pictures and all. Where should this page be put - somewhere in static-website, or in this repository; and where exactly there?

joepal1976 commented 1 year ago

The mhskel format is an unfortunate remnant of older times. It is an archaic bastardization of BVH, and not something I'm keen on having to maintain in mpfb. Thus the current json format. So being incompatible with makehuman's mhskel isn't really a problem.

The docs/tutorial page should end up on the static website. Since it is mpfb specific, it should be under the mpfb topic. Then either as a subpage of docs, or maybe we should have a new subpage tutorials. In either way, it's not hard to move if we need to change the topic hierarchy. So I suggest starting under mpfb/docs for now.

joepal1976 commented 1 year ago

Regarding the new UI, it sounds like good idea. I plan on catching up on code next week. I'll take a closer look at it then.

angavrilov commented 1 year ago

Regarding the new UI, it sounds like good idea. I plan on catching up on code next week. I'll take a closer look at it then.

I intend to make that doc some time this week+weekend, and it would have an explanation and pictures.

I also converted the rigify rig definitions to directly include the new style face, because currently generating a rigify rig breaks Refit To Base Mesh, because the metarig now contains face bones that are not listed in the definition, and thus can't be repositioned correctly.

angavrilov commented 1 year ago

@joepal1976 So, I made a tutorial for rigging an asset: https://github.com/angavrilov/makehuman-static-website/blob/master/content/MPFB/Docs/rigging_mesh_assets.md

I also added a reference guide style description of the new UI panel to this document: https://github.com/angavrilov/makehuman-static-website/blob/master/content/MPFB/Developer/rigging_and_posing.md

I think this is more or less ready, but I want to hear your opinion on the way weights are loaded, from the point of view of making assets compatible with both MH and MPFB. E.g., should MPFB use the "vertexboneweights" setting from mhclo; should the files for MPFB be referenced via new similar settings in mhclo, or is using just filenames like currently implemented ok, etc.

joepal1976 commented 1 year ago

I have now read through both documents, and they look like very useful additions to the docs on the website.

Keeping things backwards compatible with MH is, of course, a good thing. But only if it can be done without being a great hindrance to the development. For example, I don't think that the vertexboneweights setting is used a whole lot as it is, and maybe things could be implemented in a more efficient way?

I have no readily formed answer on exactly what that solution would be, but my gut feeling tells me that sooner or later we'll reach the limits of MHCLO. Besides, MHCLO files are more or less immutable. What happens if you want two different alternative rigs and/or weights setups? Possibly it'd make more sense to store separate JSON structures with the reference pointing the other way: this is secondary weights setup / rig made for an asset with this uuid.

For now, though, I think that the current approach with files you load explicitly is the best available option.

Backporting added functionality to MH is out of the question, especially when it comes to rigging. The MH codebase is so burdened with legacy that every simple addition will require a huge effort.

I think the procedure you describe is perfectly fine for now, but in the longer run I think a few things should be added as new features to MPFB.

None of this blocks the tutorial you've written, they're just thoughts for features to add in the future. So my suggestion now is that you make a PR out of your changes and I'll merge them to the website.

Random thoughts on related features I think should be added at some point:

Move the rig/weight stuff out of the developer panel

One thing that strikes me, is that I think several of the steps you describe should be possible to automate. Or at least assist significantly. And that they are desirable for general users, not only developers.

Maybe it'd make sense to start moving the rig functionality out of the developer panel and instead form a more user-friendly interface with relevant functionality under "create assets".

This could be combined with various utility button for performing some of the steps described in the tutorial.

Create, store and reload alternative weights in a structured manner

It would be beneficial to have a structured way to manage weights for various combinations of rigs and assets. Preferably this should be possible to manage from within the UI. Yes, it is in practice possible to do much of this via the developer panel, but it would benefit from having an asset-library like UI where you get a drop-down with possible alternatives matching the selected rig.

The weights creation stuff could have a "store in library" button as an alternative to save to file, and character presets could be extended to persist which weights have been selected for a character.

This would make it more accessible to keep different weights for different types of characters, and also for getting quick access to weights files that are more efficient than the current interpolation strategy.

Create, store and reload asset specific secondary custom rigs in a structured manner

This is more or less what you have already described. But as said, I think many of the steps described could be assisted with scripted automation.

The secondary rigs should be possible to store in an asset-library style way too so that they are easily selectable on the next character having the same asset loaded. And possible to persist in character presets.

angavrilov commented 1 year ago

I don't quite understand what you think could be automated, besides maybe saving to the correct file names next to the mhclo. The rig and weights are loaded fully automatically with the mhclo, all technical messing with files is for the asset creator. Bone fitting strategies can still be assigned automatically, the tutorial describes the manual tweaking process and UI in detail, because, well, describing complicated things is the point of a tutorial. ;)

Maybe you mean a button to automatically create a single bone sub-rig for the selected asset mesh and initialize it for development in one click?

Basically, now mesh assets can include a rig and weights in addition to the mesh. I think character presets should handle it via referencing the asset itself?

I don't really see any point in alternative rigs for assets, because can't you just make multiple versions of the asset?

Edit: But if you come up with a use case, it may be possible to extend it based on the same idea as materials maybe? I.e. any mpfbskel files in the same directory as the mhclo are considered compatible with the mhclo (with the one sharing the basename considered default), and weight files are loaded based on the mpfbskel name, kind of thing?

Regarding MH compatibility, I mean specifically making an asset that would work both in MH (which doesn't understand sub-rigs), and MPFB which does. MH uses one weight file that covers all skeletons, and is referred via vertexboneweights, while MPFB loads multiple specific file names (but will check vertexboneweights too). Currently you can make a legacy MH asset with one weight file and vertexboneweights set, and MPFB would try to make something of it. You can also make a purely MPFB asset (with files, but no vertexboneweights setting), and MH would ignore all weights. But it is difficult to make an asset that provides weights both for MH without rig and MPFB with rig, because mainly the use for ASSETNAME.mhw is different: in MPFB that would be the place where you put the weights for the sub-rig, while main rig specific customizations go into other files.

One idea could be that maybe MPFB should not check vertexboneweights after all: currently for standard assets the file name seems to be ASSET.mhw anyway. So you can do something like that for a dual compatible asset:

If you want to use the same weight file for MH and MPFB, you can just call it ASSET.mhw and also reference it via vertexboneweighs, exactly like the current standard assets do by pure coincidence.

Re PR I'm primarily waiting for your OK re if this way MHW files are loaded based on the MHCLO name and main rig ID is ok re the above paragraph, or if there is a better way to improve MH compatibility (basically what is described at the very end of the tutorial).

joepal1976 commented 1 year ago

I don't quite understand what you think could be automated, besides maybe saving to the correct file names next to the mhclo. The rig and weights are loaded fully automatically with the mhclo, all technical messing with files is for the asset creator. Bone fitting strategies can still be assigned automatically, the tutorial describes the manual tweaking process and UI in detail, because, well, describing complicated things is the point of a tutorial. ;)

Maybe you mean a button to automatically create a single bone sub-rig for the selected asset mesh and initialize it for development in one click?

Yes, I was talking about the part of the tutorial where you start setting up a project for a secondary rig. A rough mockup UI would include options like:

I don't really see any point in alternative rigs for assets, because can't you just make multiple versions of the asset?

Edit: But if you come up with a use case, it may be possible to extend it based on the same idea as materials maybe?

Yes, this is more like the materials case. It might be like you say: there's likely only one rig for an asset. However, we'll also have to consider that the assets reside in the asset repository (or, as it were, in asset packs) and are usually created by another author than the one wanting to add a rig. Patching the asset in the repo would be cumbersome, since you can't touch someone else's files.

However, as you also say, there is the parallel case with alternative materials: You can add another file to place alongside the asset. This does not require patching the original asset.

One idea could be that maybe MPFB should not check vertexboneweights after all: currently for standard assets the file name seems to be ASSET.mhw anyway. So you can do something like that for a dual compatible asset:

  • vertexboneweights points to ASSET.legacy.mhw, will be loaded by MH
  • ASSET.mhw, ASSET.RIG.mhw contain weights for MPFB, will be ignored by MH.

If you want to use the same weight file for MH and MPFB, you can just call it ASSET.mhw and also reference it via vertexboneweighs, exactly like the current standard assets do by pure coincidence.

This sounds like a feasible strategy.

Re PR I'm primarily waiting for your OK re if this way MHW files are loaded based on the MHCLO name and main rig ID is ok re the above paragraph, or if there is a better way to improve MH compatibility (basically what is described at the very end of the tutorial).

I think the strategy you propose above sounds like a good start. Let's explore that, and in the unlikely case we run into some reason why it wouldn't work, we'll think up another strategy.