process animal data

[Grem-Lin]

Hi! In animal.py file, at https://github.com/c-he/NeMF/blob/146a1eade5dd7eb77db8380c7f03adf99bfb09a2/src/datasets/animal.py#L39-L42 Q1 why do : data_translation = anim.positions[:, 0] / 100.0 ? It seems that it rescaled root translation with a factor of 100, but why do this? And why we don't scale the whole anim.positions?

Q2 why we insert identity quaternion for rotation ?

Thanks!

[c-he]

A1: L39 scales root translation to change its units from centimeters to meters. Since other data are unnecessary for our processing pipeline, we just leave them.

A2: Since the visualization scripts we use cannot render skeletons correctly if their end joints have extra length, we modify their topology by inserting addtional bones with identity rotation to hack the rendering. That's why we need to insert identity quaternions.

[Grem-Lin]

Thanks for your quick reply! I have two follow-up questions. Q1. If the values of dog motion data are in centimeters, why do you only scale anim.positions, but not anim.offset, which is the input of fk object in L21? I read through fk.py and didn't find any rescaling for offset. I think anim.offset is necessary for the local joint position (in L 59), and if the values are in different units, it may cause any issues?

Q2. I see why inserting identity quaternion. It seems that the new rotation is also used to generate the data for network training, not just for visualization. For example, rot6d is the input of fk(). I don't get why we input the "new rotation with more bones" into fk with setting of "old offset and parent". It seems to be that bone numbers are inconsistent.

Thanks.

[c-he]

In https://github.com/c-he/NeMF/blob/146a1eade5dd7eb77db8380c7f03adf99bfb09a2/src/datasets/animal.py#L21 you can see the skeleton used for FK is read from args.animal.parents and args.animal.offsets. These two variables are defined by https://github.com/c-he/NeMF/blob/146a1eade5dd7eb77db8380c7f03adf99bfb09a2/configs/dog_skel.yaml, in which we've already:

1. scale the units of bones
2. update the topology to include extra bones

[Grem-Lin]

I see. Thanks for clarifying! One more question, I directly rescale the dog motion in bvh file, and visualize it in Blender (I just import bvh file rather than using the visualization code). It seems to me that the dog doesn't use the back left/right ends to walk, it uses some joints to walk, and the tail is bent. May I ask why?

Here is the code for rescaling:

anim, _, ftime = BVH.load(bvh_fname)
anim.positions[:, 0] = anim.positions[:, 0] / 100.0 
anim.offsets = anim.offsets / 100.0
BVH.save(work_dir + bvh_fname.split('/')[-1][:-4]+'_scale.bvh', anim, None, ftime)

Here are the original dog and rescaled dog plot in Blender for reference. Both dogs walk towards right:

The bvh files are also attached. original: https://drive.google.com/file/d/1wnyG03LouFNGqmCCCUqktCxSfEAJvV-V/view?usp=share_link rescaled: https://drive.google.com/file/d/1Jj3AVYCBkOwcGJq4MDAGs10E24FZsTXX/view?usp=sharing Thanks!

[c-he]

I encountered similar problems in this project. This should be some visualization issue with Blender. If you try to use the visualization code, it should be fine.

[Grem-Lin]

Thanks! I met with an issue with python when using the visualization code. I followed its Prerequisites about python, but I still got this error. Any ideas of what's going wrong here?

./blender -P blender_rendering/load_bvh.py 
Read prefs: /home/sadie/.config/blender/3.4/config/userpref.blend
Python path configuration:
  PYTHONHOME = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python'
  PYTHONPATH = (not set)
  program name = '/home/sadie/Downloads/blender-3.4.1-linux-x64/blender'
  isolated = 0
  environment = 1
  user site = 0
  import site = 1
  sys._base_executable = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python/bin/python'
  sys.base_prefix = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python'
  sys.base_exec_prefix = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python'
  sys.platlibdir = 'lib'
  sys.executable = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python/bin/python'
  sys.prefix = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python'
  sys.exec_prefix = '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python'
  sys.path = [
    '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python/lib/python310.zip',
    '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python/lib/python3.10',
    '/home/sadie/Downloads/blender-3.4.1-linux-x64/3.4/python/lib/python3.10/lib-dynload',
  ]
Internal error initializing Python!
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f075a769240 (most recent call first):
  <no Python frame>

[c-he]

I didn't encounter the problem before. Actually I didn't follow their prerequisites but just used the built-in Python environment from Blender. The only problem was that the built-in environment didn't have scipy, so I installed it and then everything worked.

[Grem-Lin]

Hi I have two questions related to animal data processing. Q1 I am wondering how to remove some sequences on uneven terrain manually. Do you remove it by watching each motion sequence? Q2 What if we didn't remove the sequences on uneven terrain? Will it cause any issues?

Thanks!! (Too many questions from me..

[c-he]

Hi,

1. Yes we removed those sequences by manually watching them in blender.
2. The main reason for removing those sequences is that when predicting global translations, we would like the predicted root heights to be within a roughly bounded range. If the character doesn't move on an even terrain, its root height would have a much larger variation, making them more difficult to learn by the network.