search

DeepMotionEditing / deep-motion-editing

An end-to-end library for editing and rendering motion of 3D characters with deep learning [SIGGRAPH 2020]
BSD 2-Clause "Simplified" License
1.56k stars 256 forks source link

How evaluate output quality #117

Open fire opened 3 years ago

fire commented 3 years ago

Hi,

Is there a method where I can compare error versus something that exists?

There's evaluate python script but instructions to use wasn't clear.

I was having problems checking intra_a -> intra_b, intra_b -> intra_a, cross_a ->cross_b, cross_a -> cross_b gives the correct result and not invalid or motionless outputs.

PeizhuoLi commented 3 years ago

One common used evaluation is to measure the L2 distance of joint location (normalized by height) between the result and ground truth. You can try to modify the code starting from here: https://github.com/DeepMotionEditing/deep-motion-editing/blob/36f97fe32cc874b2818c42904e7a117886b7740c/retargeting/get_error.py#L45

jdbodyfelt commented 3 years ago

Within test.py (L35-66) there is some great intra/cross error reporting that might be of interest. That could probably be generalised and classed out to what you want to do, @fire.

The team did a good job with tensorboard incorporation, and have learning curves presented across various metrics for a training.py run. However, are similar LCs presented during test.py run? It would be of great use to have training-validation split "built-in" to the algo, similar to tensorflow.data.Dataset.load(<file>, split=['train', 'test']), and also put into the tensorboard output.

I understand the errors via the publication, but from LCs, might be useful to have a few screenshot images of "what to look for" in your own training (@PeizhuoLi, I'll send you a PM...)