search

ARIA-VALUSPA / AVP

This is the ARIA-VALUSPA Platform, or AVP for short. Use this platform to build your own Virtual Humans with audio-visual input and output, language models for English, French, and German, emotional understanding, and many more. This work was funded by European Union Horizon 2020 research and innovation programme, grant agreement No 645378.
https://aria-agent.eu
32 stars 16 forks source link

Lipsync not working #11

Open merijnman opened 6 years ago

merijnman commented 6 years ago

In the public AVP, Greta appears to have no lipsync to speech. Any ideas what to do to solve it? :) Slightly related / the greta LogWindow does not show any logging...

mvalstar commented 6 years ago

Hmmm. Shashank, do we use AVP 3.0? If so, do you see this problem as well?

dshashank1 commented 6 years ago

I am using the Mirage branch from ARIA (Private) repository. So can't comment on this one. For me there seems to be no problem with lipsync.

mvalstar commented 6 years ago

Right, then we've got something to look at. Assigning Peter and Paul, as Paul's been playing with Greta, and there's nobody in Catherine's team anymore.

Barachia commented 6 years ago

Idem if you're using the ARIA-GRETA (private) repository.

EDIT: I meant, the GRETA from the private ARIA-GRETA repo is working correctly

bpotard commented 6 years ago

I wonder whether this could be due to the language of the voice being used? I believe in the private ARIA-GRETA repo we use the en-GB voice Sarah, while for the public one we use the en-US voice "Alice" (confusingly placed in a en-GB folder, which is a mistake). These en-GB and en-US variants use different phonesets, so I can see two possible reasons for the lipsync not working with Alice:

The best way to check would be to use the private ARIA-GRETA repo with the voice of AVP (alice). The folder where the voice is located is wrong and confusing... and it could possibly be the reason the lipsync is not working.

Although, without the source code for Greta, this is hard to tell really...

mvalstar commented 6 years ago

I believe Catherine is open-sourcing Greta, so that would help. I'll ask her.

pelachaud commented 6 years ago

Dear all,

We are open-sourcing Greta. I have nobody helping me with that, so things are not progressing. We are aiming at an LGPL licence.

Regarding Cereproc voices; we have:

GB: sarah, william

US: hannah; adam

FR: laurent, suzanne

Soumia tried the different voices and it works for her. Could the errors come from: either the american voice is not added in the cereproc directory of Greta or the licence has not been updated?

Soumia is preparing the files source code. We still need to add the reference to the LPGL licence. She will contact you when the files are ready. May be she should be added to the github mailing list?

Kind regards, Catherine

Le 28/03/2018 à 22:11, Michel Valstar a écrit :

I believe Catherine is open-sourcing Greta, so that would help. I'll ask her.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ARIA-VALUSPA/AVP/issues/11#issuecomment-377019852, or mute the thread https://github.com/notifications/unsubscribe-auth/ALNZY_TFZMUhW6eQanV9CVWH3eHzdjQZks5ti-5UgaJpZM4S9Dsr.

dermosamo commented 6 years ago

I think the problem comes from new voice "arctic" added in cereproc and which represents the voice of Alice in this project. Do we have this problem only with this new voice or also with the other voices?

pelachaud commented 6 years ago

Hi Catherine,

Thanks for that. It would be great if Soumia could let me know her GitHub username, so I can make sure she's added.

Best,

Michel

Dr. Michel Valstar Associate Professor in Computer Science http://www.cs.nott.ac.uk/~pszmv

Mixed Reality/Computer Vision Lab School of Computer Science University of Nottingham

tel.:+44 (0) 115 823 2287<tel:0115%20823%202287> fax: +44 (0) 115 951 4254<tel:0115%20951%204254>

==========================================================

New AU Survey out now: http://ieeexplore.ieee.org/document/7990582/

State-of-the-art face tracker: http://ieeexplore.ieee.org/abstract/document/7961726/ http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8017515 http://ieeexplore.ieee.org/document/8017515/

==========================================================

<tel:0115%20951%204254>

From: Catherine PELACHAUD catherine.pelachaud@upmc.fr Sent: 29 March 2018 10:39:09 To: reply@reply.github.com Cc: Dermouche Soumia; Michel Valstar; 'Blaise Potard'; Jelte van Waterschoot; m.bruijnes@utwente.nl Subject: Re: [ARIA-VALUSPA/AVP] Lipsync not working (#11)

Dear all,

We are open-sourcing Greta. I have nobody helping me with that, so things are not progressing. We are aiming at an LGPL licence.

Regarding Cereproc voices; we have:

GB: sarah, william

US: hannah; adam

FR: laurent, suzanne

Soumia tried the different voices and it works for her. Could the errors come from: either the american voice is not added in the cereproc directory of Greta or the licence has not been updated?

Soumia is preparing the files source code. We still need to add the reference to the LPGL licence. She will contact you when the files are ready. May be she should be added to the github mailing list?

Kind regards, Catherine

Le 28/03/2018 à 22:11, Michel Valstar a écrit :

I believe Catherine is open-sourcing Greta, so that would help. I'll ask her.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ARIA-VALUSPA/AVP/issues/11#issuecomment-377019852, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALNZY_TFZMUhW6eQanV9CVWH3eHzdjQZks5ti-5UgaJpZM4S9Dsr.

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment.

Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.

Barachia commented 6 years ago

Hi all,

I just tested the arctic voice in the private ARIA-GRETA repository, and it doesn't work, I get this error:

ERROR : CereprocTTS: phoneme conversion failed, language [-] is not supported.

In this case I put the voice in ..\ARIA-Greta\bin\Common\Data\Cereproc\en-us-alice, together with the license file. As Catherine said, the other voices work in the private repository (Jess, Sarah), but these will only be vaild for two more days (license ends at the end of March) and of course only for private use.

@bpotard Is there an option we can use the Sarah voice for some longer time?

Best, Jelte

bpotard commented 6 years ago

It is actually possible that this is due to an issue with the voice file itself. It has been in the repository for several years but it is true that we have noticed a few months back that you could not switch to it because the header did not have the name of the voice; it is very possible it does not have the language either!

I think the updated voice we sent a few months back does not have that issue; I'll try to update it myself.

@yelte: Sure! Just follow the instructions on this page: https://www.cereproc.com/en/products/academic That should give you a free, permanent license. For academic use only :-)

pelachaud commented 6 years ago

Hi Michel,

Soumia is working to prepare the source code of Greta to be added to github. She still has to add a header into each source file with the licence information. It may take some times.

She will send you her github login info.

We are releasing the core of Greta, but anything related to attitude model nor interruption as Soumia/Angelo still need to publish about it.

Best,

Catherine

Le 29/03/2018 à 15:39, Michel Valstar a écrit :

Hi Catherine,

Thanks for that. It would be great if Soumia could let me know her GitHub username, so I can make sure she's added.

Best,

Michel

Dr. Michel Valstar Associate Professor in Computer Science http://www.cs.nott.ac.uk/~ http://www.cs.nott.ac.uk/%7Epszmv

Mixed Reality/Computer Vision Lab School of Computer Science University of Nottingham

tel.:+44 (0) 115 823 2287 <tel:0115%20823%202287> fax: +44 (0) 115 951 4254 <tel:0115%20951%204254>

==========================================================

New AU Survey out now: http://ieeexplore.ieee.org/document/7990582/

State-of-the-art face tracker: http://ieeexplore.ieee.org/document/8017515/

==========================================================

From: Catherine PELACHAUD catherine.pelachaud@upmc.fr Sent: 29 March 2018 10:39:09 To: reply@reply.github.com Cc: Dermouche Soumia; Michel Valstar; 'Blaise Potard'; Jelte van Waterschoot; m.bruijnes@utwente.nl Subject: Re: [ARIA-VALUSPA/AVP] Lipsync not working (#11)

Dear all,

We are open-sourcing Greta. I have nobody helping me with that, so things are not progressing. We are aiming at an LGPL licence.

Regarding Cereproc voices; we have:

GB: sarah, william

US: hannah; adam

FR: laurent, suzanne

Soumia tried the different voices and it works for her. Could the errors come from: either the american voice is not added in the cereproc directory of Greta or the licence has not been updated?

Soumia is preparing the files source code. We still need to add the reference to the LPGL licence. She will contact you when the files are ready. May be she should be added to the github mailing list?

Kind regards, Catherine

Le 28/03/2018 à 22:11, Michel Valstar a écrit :

I believe Catherine is open-sourcing Greta, so that would help. I'll ask her.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ARIA-VALUSPA/AVP/issues/11#issuecomment-377019852, or mute the thread https://github.com/notifications/unsubscribe-auth/ALNZY_TFZMUhW6eQanV9CVWH3eHzdjQZks5ti-5UgaJpZM4S9Dsr.

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment.

Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.

mvalstar commented 6 years ago

Thanks Blaise - if you have the ability to change the voice in the repo, please do so.

bpotard commented 6 years ago

Hi Michel, I have added in git a new build of the arctic / alice voice, although I cannot guarantee it solves the issue! It would be great if someone could try it out.

Barachia commented 6 years ago

Hi Blaise, I've tried the other voice file, but this failed in the agent (the error log of GRETA doesn't show clearly why). I put the old arctic voice back for now.