search

DeepMotionEditing / deep-motion-editing

An end-to-end library for editing and rendering motion of 3D characters with deep learning [SIGGRAPH 2020]
BSD 2-Clause "Simplified" License
1.58k stars 256 forks source link

style transfer #150

Closed Hellodan-77 closed 3 years ago

Hellodan-77 commented 3 years ago

Hello! I have now generated bfa.npz and xia.npz files through gen_dataset.sh. For video data, I use my own video actions to extract joint points and output json files. I would like to ask the following questions:

  1. What should I do next with the video json file extracted with openpose? Do I need to generate an npz file like the bfa dataset? What needs to be modified? Where is the specific code? How does it work?
  2. I saw that the next step after running gen_dataset.sh that you introduced on the github code homepage is to run train.py, then for the two processed data of video and BVH files, if I want to train, which ones need to be modified What about the relevant code files?
Hellodan-77 commented 3 years ago

There is another question. Are these data such as bfa, xia, and video trained separately?

HalfSummer11 commented 3 years ago
  1. There's no need to generate npz files. The folder containing the json files can be directly used as the input argument for our model. Please refer to style_transfer/demo.sh for details (also please refer to the file organization in our demo json input).
  2. You can directly run train.py without modifying any data - that part is done in gen_dataset.sh. Also, note that video data is not used during our training - it's only used during test time. The default training will use xia dataset. If you want to train on the bfa dataset, config.py needs some further modifications. You may find this relevant issue helpful.
  3. Yes, xia and bfa contains two different sets of styles, so we only support training solely on xia or solely on bfa. Video data is never used during training.
Hellodan-77 commented 3 years ago

Thank you very much for your reply! If it is what you said, I have the following questions:

  1. There is a part of your code to calculate mean and std. What does this mean? Because I use my own video data, can I just modify it in gen_dataset.sh? Is there any other code that needs to be modified?
  2. What do the several yml format files in the global_info folder mean?
  3. There is a file named test2d.npz in your code file. How did you get this file? What role does it play in the whole process?
HalfSummer11 commented 3 years ago

Regarding 1 and 3 (test2d.npz contains std and mean), I think we have already discussed them in this issue and specifically we mentioned test2d.npz. By default, we use mean and std in test2d.npz to normalize your video data at test time. They may not work well for your own data - if the skeletons in your video deviate much from the skeletons in the treadmill video. You can generate your own mean and std if you have a collection of video data, using code referred to in that previous issue.

  1. They contain information about the xia/bfa dataset as well as the CMU skeleton we use (just as their names suggest).
Hellodan-77 commented 3 years ago

My own video data is exactly the same as your treadmill video using openpose to extract the 2D joint points, including the bone point format in the generated json file is the same as yours, under such conditions, I still need Modify the mean and std you mentioned?

HalfSummer11 commented 3 years ago

In that case, I think you can use the default mean and std values.