search

swatsw / isg_official

Code for paper "Integrated Speech and Gesture Synthesis"(ICMI 2021)
Other
6 stars 2 forks source link

invalid start byte #1

Closed yyysj0103 closed 1 year ago

yyysj0103 commented 1 year ago

Hello, I am very interested in your project and have been trying to replicate it recently. Excuse me, when I ran to step 5, this error occurred because there are non UTF-8 encoded models in all models? Looking forward to your answer! Thanks a lot!

1111
swatsw commented 1 year ago

I have not encountered this error before. But as you can see in docker/Dockerfile, the version of python package Unidecode==1.0.22 is required.

yyysj0103 commented 1 year ago

Thank you! I have already solved this problem! I have another question, if I want to achieve better results as soon as possible, can I use the Tacotron 2 model for the fifth step of training? Looking forward to your reply very much~

yyysj0103 commented 1 year ago

And now that I have obtained permission for the dataset, do you still have the pre-trained model from that time? Looking forward to your reply~

swatsw commented 1 year ago

It is important to go through step 4 as we found it to be hard to learn speech and gesture in the same training phase. The amount of time/iterations for speech-only training (step 4) depends on the batch size and other hyperparameters. With that said, you can try to train CT-Tacotron2-ISG directly from LJSpeech Tacotron2 weights, but it may actually take more time than if speech-only training (step 4) is completed frist.

yyysj0103 commented 1 year ago

Can the pre-trained model be provided?Ive already obtained permission for the dataset.Thanks! :)

swatsw commented 1 year ago

Sorry we do not provide pre-trained models for now. We are more than happy to help if you run into issues running any part of the code.

yyysj0103 commented 1 year ago

Thanks a lot for your patient responding！I am already able to test the program, but due to limitations in my deep learning training equipment, my current reproduction effect is not ideal, so I am asking if I can have a pre trained model. My training process has just reached 12500 ltrations in step four, and it may take a lot of time to complete the ideal training in step four. Thank you again!

swatsw commented 1 year ago

Hi, I understand, but unfortunately we are unable to provide trained weights at this time. We completed this work with limited computing resource (2 11GB GPUs). We found that training more iterations with a batch size that fit this much memory achieves good results in our experiments. It is however a downside of this particular model where the sequential modeling uses RNN, that training could take a long time.

yyysj0103 commented 1 year ago

Thank you for your patient reply！