Cleaner explanation & example of loading our own datasets?

[jdbodyfelt]

First off, thank you for open sourcing this great project - you AE-GAN idea is beautiful! The Mixamo examples work, but when it comes time for my own datasets, I can't get them to train. As an outline of my process:

1. We have a motion capture suit that our actors use. These all have the same skeletal structure, so I place these into an "A" group ("A" for actor...).
2. For every actor bvh, our animators take the bvh, do in-house IK and retargeting in MotionBuilder, and produce a new much more dense skeleton that gets dumped to bvh, so this is cross-structure. It's the exact same skeleton for each actor's file, so it's a character of its own, let me call it "avatar".
3. I have a script that goes through and syncs the data from a file in A to a file in B, so that I have a rate-synced pair A-B.
4. I have a custom skeleton rig for both my mocap side, and my avatar side. This is incorporated into bvh_parser. Now, in datasets/__init__.py I have defined my characters as [[actor1, actor2, actor3], [avatar]].

When I train off this, I get the following output error:

Traceback (most recent call last):
  File "train.py", line 56, in <module>
    main()
  File "train.py", line 29, in main
    model = create_model(args, characters, dataset)
  File "retargeting/models/__init__.py", line 5, in create_model
    return models.architecture.GAN_model(args, character_names, dataset)
  File "retargeting/models/architecture.py", line 25, in __init__
    model = IntegratedModel(args, dataset.joint_topologies[i], None, self.device, character_names[i])
  File "retargeting/models/integrated.py", line 45, in __init__
    self.auto_encoder = AE(args, topology=self.edges).to(device)
  File "retargeting/models/enc_and_dec.py", line 129, in __init__
    self.enc = Encoder(args, topology)
  File "retargeting/models/enc_and_dec.py", line 30, in __init__
    neighbor_list = find_neighbor(self.topologies[i], args.skeleton_dist)
  File "retargeting/models/skeleton.py", line 379, in find_neighbor
    global_part_neighbor = neighbor_list[0].copy()
IndexError: list index out of range

I noticed that Issue #30 has a mention near this line. What is going on?! How do I load my datasets?!

[jdbodyfelt]

The list index out of range comes from the fact that args.num_layers=2 and len(self.topologies)=1 within the Encoder class.. .

[PeizhuoLi]

Hi, the problem looks different from the one in Issue #30.

len(self.topologies) = 1 means the BVH_file class has trouble in recognizing the joints for retargeting. I would suggest you to check if the corps_name, the ee_name of your character is correctly set and BVH_file.skeleton_type is also correctly set. You can check the four comments in retargeting/datasets/bvh_parser.py for more details.

Hope this can help.

[jdbodyfelt]

@PeizhuoLi, thank you for your rapid response! Yes, I have these set, but it might be that my end-effectors are 13 rather than 5 (we are interested in fingertips for our mocap). Didn't think this would be an issue, as these are the only datasets I'm loading. I noticed that skeleton.type=0 (the Mixamo base) sets a new_root, is this related to the problem? (PS: Mixamo example works awesome, BTW...)

[PeizhuoLi]

The set_new_root is for making the three-junction joint as the root of character. Since the root joint will never be pooled, it will lead to a different primal skeleton and the latent space will have a different dimension if the three-junction joint i not root. In your case I wouldn't think it will cause a problem in finding neighbors.

However, it might be the point that you have more than 5 end-effectors. We've only tested our code with no-finger character so there could be some point that we missed.

If you can try to use debug mode and provide more information about related variables it will be helpful.

[graham-eisele]

I am also getting the same error.