Open uezo opened 2 years ago
Would you update this for Unity 2021? I'm waiting for it. 😃
Hi @zhuchenyang , Sorry for make you waiting. But this disconnecting issue is not clearly documented, maybe an implicit spec or bug. I will share you a work around for this later, but I don't know it have some side effects or not.
Hello, I just make it run on both 2020 and 2021.
In line 11: for (var i = 0; i < WEBAudio.audioInstances.length; i++)
=> for (var i = 1; i <= WEBAudio.audioInstanceIdCounter; i++)
.
Maybe the reason is "length" is not supported in chronium based browers.
Hello, I just make it run on both 2020 and 2021. In line 11:
for (var i = 0; i < WEBAudio.audioInstances.length; i++)
=>for (var i = 1; i <= WEBAudio.audioInstanceIdCounter; i++)
. Maybe the reason is "length" is not supported in chronium based browers.
what version of 2020 and 2021 did you get this to work on? I see it connecting in the webgl code but it doesn't do anything to the blendshapes
Ok further check I get a string of all 0's coming back despite connecting ....
So it turns out this is the issue @zhuchenyang is right about the query how ever the browser is preventing the audio context from starting up for chrome. So this sorta fixes it but I think I have to find a way to keep that context resumed.
public void Update() {
if (Input.GetKeyDown(KeyCode.Return)) {
#if UNITY_WEBGL && !UNITY_EDITOR
ulipSyncScript = GetComponent<uLipSyncScript>();
if (ulipSyncScript != null)
{
InitWebGLuLipSync(gameObject.name, "SetAudioSampleData");
}
Debug.Log("Started Audio Context");
#endif
}
}
private void SetAudioSampleData(string inputString)
{
// if (!ulipSyncScript.audioSourceProxy)
// {
Debug.Log(inputString);
var samplingData = inputString.Split(',').Select(s => Convert.ToSingle(s)).ToArray();
//Debug.Log($"{samplingData}");
ulipSyncScript.OnDataReceived(samplingData, 1);
//}
}```
The audio context was not allowed to start is the message I get from chrome
Unity Editor Version 2022.1.24 / 2021.3.13 LTS. Hello there! by using these editor versioni, the lip syinc works bad. By logging the inputString value i get all almost zero values in webgl build, In other hand printing in the editor the inputString log is right. There is any solution?
Unity Editor Version 2022.1.24 / 2021.3.13 LTS. Hello there! by using these editor versioni, the lip syinc works bad. By logging the inputString value i get all almost zero values in webgl build, In other hand printing in the editor the inputString log is right. There is any solution?
the values are correct if you are getting them from the webgl version I get 0's all the time
My workaround.
mergeInto(LibraryManager.library, {
InitWebGLuLipSync: function(targetObjectNamePtr, targetMethodNamePtr) {
const targetObjectName = UTF8ToString(targetObjectNamePtr);
const targetMethodName = UTF8ToString(targetMethodNamePtr);
const outputHookNode = WEBAudio.audioContext.createScriptProcessor();
outputHookNode.onaudioprocess = function (stream) {
SendMessage(targetObjectName, targetMethodName, event.inputBuffer.getChannelData(0).join(','));
};
const connectAudioNodes = function (audioInstance) {
if (audioInstance != null && audioInstance.hasOwnProperty("gain")) {
// connect gain -> outputHookNode
audioInstance.gain.connect(outputHookNode);
// connect outputHookNode -> dest (dummy: no output data will go to dest)
outputHookNode.connect(WEBAudio.audioContext.destination);
console.log("Connected audio nodes successfully");
return true;
} else {
return false;
}
};
const jobId = setInterval(function() {
for (var key in WEBAudio.audioInstances) {
if (connectAudioNodes(WEBAudio.audioInstances[key])) {
// Continuously reconnect gain -> outputHookNode (they will be disconnected silently, I don't know why...)
setInterval(function() {
WEBAudio.audioInstances[key].gain.connect(outputHookNode);
}, 200);
clearInterval(jobId);
break;
}
}
}, 200);
},
});
@uezo have you considered using the new api ? maybe this is deprecated and they broke it by accident
Hi @ArEnSc , "new api" means WEBAudio.audioInstanceIdCounter
?
Actually I found WEBAudio.audioInstanceIdCounter
returns number at runtime, but I couldn't find the official document about that explains the spec of this api.
And, we have to make out the way to access each audio instance in the for
loop.
if (WEBAudio.audioInstances[i] != null && WEBAudio.audioInstances[i].hasOwnProperty("gain")) {
Do you have any ideas?
@uezo >
sadly my knowledge of JS is limited by the new api looks like ScriptProcessorNode
My workaround.
mergeInto(LibraryManager.library, { InitWebGLuLipSync: function(targetObjectNamePtr, targetMethodNamePtr) { const targetObjectName = UTF8ToString(targetObjectNamePtr); const targetMethodName = UTF8ToString(targetMethodNamePtr); const outputHookNode = WEBAudio.audioContext.createScriptProcessor(); outputHookNode.onaudioprocess = function (stream) { SendMessage(targetObjectName, targetMethodName, event.inputBuffer.getChannelData(0).join(',')); }; const connectAudioNodes = function (audioInstance) { if (audioInstance != null && audioInstance.hasOwnProperty("gain")) { // connect gain -> outputHookNode audioInstance.gain.connect(outputHookNode); // connect outputHookNode -> dest (dummy: no output data will go to dest) outputHookNode.connect(WEBAudio.audioContext.destination); console.log("Connected audio nodes successfully"); return true; } else { return false; } }; const jobId = setInterval(function() { for (var key in WEBAudio.audioInstances) { if (connectAudioNodes(WEBAudio.audioInstances[key])) { // Continuously reconnect gain -> outputHookNode (they will be disconnected silently, I don't know why...) setInterval(function() { WEBAudio.audioInstances[key].gain.connect(outputHookNode); }, 200); clearInterval(jobId); break; } } }, 200); }, });
i tried this but doesnt works in webgl build for me, character just open and closes mouth
@ArEnSc I don't have considered new APIs such as AudioWorklets
and AudioWorkletNode
for now. Just workaround. But I understand ScriptProcessorNode
is deprecated and I have to replace them in the (near) future. Thank you.
@Gabri94x
character just open and closes mouth
Isn't it lip sync? Does it work on native platform, not WebGL? If not, I guess uLipSync is not configured correctly.
in the editor everything works fine, avatar talking has a very nice and smooth mouth shaping, but when i build to the webgl platform, its just open the mouth when it talks, and close it when it stops to talk
Hello, have you found any solution to run this on newer unity/broser versions? I'm using 2021.3.21 (LTS) and sadly the lipsync on web doesn't work anymore.
Any progress on making the lipsync work on webgl builds?
@Solidsoldier12 This works. https://github.com/uezo/uLipSyncWebGL/issues/1#issuecomment-1362846674
@Solidsoldier12 This works. https://github.com/uezo/uLipSyncWebGL/issues/1#issuecomment-1362846674
I tried it but still doesn't seem to work man🥲 Gonna do some more deep dive and experimenting with it, and gonna let you know! I think the problem is that the for loop isn't getting executed The one in setInterval.
@Solidsoldier12 oh...
Checking what is set to WEBAudio.audioInstances
will help you.
@Solidsoldier12 oh... Checking what is set to
WEBAudio.audioInstances
will help you.
Will do and let you know👍
The WEBAudio.audioInstances has nothing in it when i console.log it, it just prints {}. I think that's why the for loop cannot find a key in it, can you suggest a fix or a workaround, your above workaround isn't working. Unity version: 2021.
Could it be a browser problem? I tested with Chrome and visemes was not working, when I tested with Microsoft Edge it worked.
Still does not work in Unity 2022. I don't know how to implement that workaround
Hi @Morgan-6Freedom, Copy the code of that workaround and replace whole code of uLipSyncWebGL.jslib with it.
Thank you. I will try that. I change my Unity Version to 2020.3.45f and it's also broken :< I will keep you in touch with the workaround !
I used baked lipsync for my project (and it worked). Did not had the chance to test the workaround sorry.
im using unity 2021.3.21 and the workaround is working. but when I add another game object with audio source (I was thinking to use it as background music), then the avatar does lip-sync not only for the avatar's audio but also the background music. it seems inputString
in SetAudioSampleData
is mixed audio data having avatar's audio and background music. in the jslib, how can I separate them?
I only want the avatar to lip-sync with the avatar audio.
My workaround.
mergeInto(LibraryManager.library, { InitWebGLuLipSync: function(targetObjectNamePtr, targetMethodNamePtr) { const targetObjectName = UTF8ToString(targetObjectNamePtr); const targetMethodName = UTF8ToString(targetMethodNamePtr); const outputHookNode = WEBAudio.audioContext.createScriptProcessor(); outputHookNode.onaudioprocess = function (stream) { SendMessage(targetObjectName, targetMethodName, event.inputBuffer.getChannelData(0).join(',')); }; const connectAudioNodes = function (audioInstance) { if (audioInstance != null && audioInstance.hasOwnProperty("gain")) { // connect gain -> outputHookNode audioInstance.gain.connect(outputHookNode); // connect outputHookNode -> dest (dummy: no output data will go to dest) outputHookNode.connect(WEBAudio.audioContext.destination); console.log("Connected audio nodes successfully"); return true; } else { return false; } }; const jobId = setInterval(function() { for (var key in WEBAudio.audioInstances) { if (connectAudioNodes(WEBAudio.audioInstances[key])) { // Continuously reconnect gain -> outputHookNode (they will be disconnected silently, I don't know why...) setInterval(function() { WEBAudio.audioInstances[key].gain.connect(outputHookNode); }, 200); clearInterval(jobId); break; } } }, 200); }, });
I have tried in Unity 2021 and 2022, but neither works. https://yuuuuu.net/ChatYokAI_9/
Hi, I'm trying this on Unity 2022.2.18 with the workaround you mentioned but I'm having the same issue as @Gabri94x , where my avatar is keeping just the mouth open still when talking and closed when not. As him, in the editor it works fine so I believe it is configured properly. There could be some kind of differences in value scales that maybe push all blendshapes to the limit cap ?
@DBigagli, could you please verify the scale difference as you suggested? Additionally, if possible, could you test this issue on different browsers and machines to ascertain whether it is environment-dependent?
I've just released v0.3 that supports Unity 2021. https://github.com/uezo/uLipSyncWebGL/releases/tag/v0.3
It works on WebGL built with Unity 2020 but it doesn't with 2021.
I found that the type of
WEBAudio.audioInstances
is changed from array to object (hash), and this change causes this bug inuLipSyncWebGL.jslib
. https://github.com/uezo/uLipSyncWebGL/blob/main/Plugins/uLipSyncWebGL.jslib#L11And, I also found that reconnecting gain node to outputHookNode periodically(e.g. every 200ms) is needed to keep lip syncing continuously. I don't know why...🤔