search

Unity-Technologies / com.unity.webrtc

WebRTC package for Unity
Other
751 stars 188 forks source link

[REQUEST] WebGL platform support #470

Open karasusan opened 3 years ago

karasusan commented 3 years ago

Precondition

WebGL support has been advanced by external developers in Unity community. Very thanks @nvanofwegen and @gtk2k https://github.com/The-Barn-Games/com.unity.webrtc/tree/feature/webgl-support

WebGL support is a long journey and we have many issues with merging it to the main branch. So I made the feature branch that receiving contributions from developers. https://github.com/Unity-Technologies/com.unity.webrtc/tree/experimental/webgl-platform

TODO

karasusan commented 3 years ago

This is the first PR https://github.com/Unity-Technologies/com.unity.webrtc/pull/478

karasusan commented 2 years ago

Set up CI process of the WebGL platform in the internal CI platform ((@karasusan)

I tried this topic last week, however, I found this is needed more time than I expected. So we test the package for WebGL platform manually.

karasusan commented 2 years ago

memo: WRS-158

karasusan commented 2 years ago

The pull request(#554) above was merged into this feature branch on our repository. https://github.com/Unity-Technologies/com.unity.webrtc/tree/experimental/webgl-platform

Please feel free to try WebGL build using this branch. And post issues on this thread if you find.

karasusan commented 2 years ago

The issue below is posted in Unity forum https://forum.unity.com/threads/webgl-platform-support.1147331/#post-7884949

Thaina commented 2 years ago

Is it possible to add support for microphone in browser too? I have seen MediaStreamAddUserMedia function but I don't see there are any sample for it

karasusan commented 2 years ago

@Thaina I guess the MediaStreamAddUserMedia API is not supported for microphone. Do you have any ideas? @nvanofwegen

Thaina commented 2 years ago

As far as I can see MediaStreamAddUserMedia was called navigator.mediaDevices.getUserMedia which seem totally about microphone so I wish it did work and I am started to try using it now. But because there was no sample I still try to figure out how to start with it

https://github.com/Unity-Technologies/com.unity.webrtc/blob/a39e89c2b18f37c093977ce3106f1305fd03992a/Runtime/Plugins/WebGL/MediaStream.jslib#L28-L37

Thaina commented 2 years ago

I has play with api and it seem I could open microphone and register stream and track. Only these function was not implemented

https://github.com/Unity-Technologies/com.unity.webrtc/blob/a39e89c2b18f37c093977ce3106f1305fd03992a/Runtime/Plugins/WebGL/Context.jslib#L213-L221

nvanofwegen commented 2 years ago

There is a WebGL scene in the Samples, which uses the UserMedia API of the browser. https://github.com/Unity-Technologies/com.unity.webrtc/blob/a39e89c2b18f37c093977ce3106f1305fd03992a/Samples~/WebGL/WebGLSample.cs#L104-L106

This basically create a VideoStream in Unity and attaches the MediaStream received from the UserMedia API on the javascript side. This stream is send to the receiver.

Please note that this has little to nothing to do with the Audio implementation recently added in 2.4.0-exp.5:

The methods you quoted (context register) were added to 'pass' the tests, but are not implemented.

Beside those limitations, you should be able to send your webcam/microphone using this package. It's just plain javascript functionality and cannot be used with Unity audio components.

Hope this helps :)

Thaina commented 2 years ago

I have been using this constraints to get only the microphone

var receiveStream = new MediaStream();
receiveStream.AddUserMedia(new (){ audio = true,video = false });

And this could open the microphone for me. Currently now I have clone this repo and add these (and only these) code to experimental/webgl-platform branch

  // Runtime\Plugins\WebGL\Context.jslib:213
  ContextRegisterAudioReceiveCallback: function (contextPtr, trackPtr, AudioTrackOnReceive){
    if (!uwcom_existsCheck(contextPtr, 'ContextRegisterAudioReceiveCallback', 'context')) return;
    if (!uwcom_existsCheck(trackPtr, 'ContextRegisterAudioReceiveCallback', 'track')) return;
    var ctx = UWManaged[contextPtr];
    /** @type {MediaStreamTrack} */
    var audioStreamTrack = UWManaged[trackPtr];

    const audioContext = new AudioContext();
    const microphone = audioContext.createMediaStreamSource(new window.MediaStream([audioStreamTrack]));
    const analyserNode = audioContext.createAnalyser();

    microphone.connect( analyserNode );

    var bufferLength = analyserNode.frequencyBinCount;
    var dataArray = new Float32Array(bufferLength);
    function live() {
      requestAnimationFrame(live);

      analyserNode.getFloatFrequencyData(dataArray);

      var settings = audioStreamTrack.getSettings();
      var ptr = uwcom_arrayToReturnPtr(dataArray, Float32Array);
      Module.dynCall_viiiiii(AudioTrackOnReceive, trackPtr, ptr + 4, bufferLength,settings.sampleRate , settings.channelCount, settings.sampleSize);
    }

    live();
  },
    // Runtime\Scripts\AudioStreamTrack.cs:257
        [AOT.MonoPInvokeCallback(typeof(DelegateAudioReceive))]
        static void OnAudioReceive(
            IntPtr ptrTrack, float[] audioData, int size, int sampleRate, int numOfChannels, int numOfFrames)
        {
            WebRTC.Sync(ptrTrack, () =>
            {
                if (WebRTC.Table[ptrTrack] is AudioStreamTrack track)
                {
                    foreach(var i in Enumerable.Range(0,audioData.Length))
                        audioData[i] = Mathf.Clamp01(Mathf.Pow(10,audioData[i] / 20));

                    track.OnAudioReceivedInternal(audioData, sampleRate, numOfChannels, numOfFrames);
                }
            });
        }

I now could send audioData which is an array of float in decibel from the microphone into unity. However now it still cannot create AudioClip properly and eventually throw this below exception and stop the loop

Invoking error handler due to RuntimeError: null function or function signature mismatch ``` Invoking error handler due to RuntimeError: null function or function signature mismatch at wasm://wasm/06cfd8fe:wasm-function[17810]:0x5ea515 at wasm://wasm/06cfd8fe:wasm-function[34739]:0xc0a3af at wasm://wasm/06cfd8fe:wasm-function[74214]:0x15080ce at wasm://wasm/06cfd8fe:wasm-function[60521]:0x12a96c3 at wasm://wasm/06cfd8fe:wasm-function[42938]:0xdf8063 at invoke_vii (blob:https://192.168.1.128:55555/81b1912c-55c2-487c-98f9-9160405d93db:4542:444127) at wasm://wasm/06cfd8fe:wasm-function[74154]:0x1506237 at wasm://wasm/06cfd8fe:wasm-function[55110]:0x11ceb11 at wasm://wasm/06cfd8fe:wasm-function[10861]:0x364af9 at wasm://wasm/06cfd8fe:wasm-function[42936]:0xdf804b at invoke_iiii (blob:https://192.168.1.128:55555/81b1912c-55c2-487c-98f9-9160405d93db:4542:442916) at wasm://wasm/06cfd8fe:wasm-function[2049]:0xad693 at wasm://wasm/06cfd8fe:wasm-function[39934]:0xd4daa5 at wasm://wasm/06cfd8fe:wasm-function[25154]:0x9d9ea4 at wasm://wasm/06cfd8fe:wasm-function[10973]:0x368b37 at wasm://wasm/06cfd8fe:wasm-function[15065]:0x546d83 at wasm://wasm/06cfd8fe:wasm-function[1991]:0xab79e at wasm://wasm/06cfd8fe:wasm-function[19648]:0x668b85 at wasm://wasm/06cfd8fe:wasm-function[20323]:0x69e9ca at wasm://wasm/06cfd8fe:wasm-function[3331]:0xfa9ed at wasm://wasm/06cfd8fe:wasm-function[42937]:0xdf8057 at invoke_iii (blob:https://192.168.1.128:55555/81b1912c-55c2-487c-98f9-9160405d93db:4542:442748) at wasm://wasm/06cfd8fe:wasm-function[1764]:0xa1659 at wasm://wasm/06cfd8fe:wasm-function[20557]:0x6c631c at wasm://wasm/06cfd8fe:wasm-function[636]:0x682f8 at wasm://wasm/06cfd8fe:wasm-function[635]:0x682e2 at wasm://wasm/06cfd8fe:wasm-function[34401]:0xc0136d at wasm://wasm/06cfd8fe:wasm-function[9310]:0x2c112a at wasm://wasm/06cfd8fe:wasm-function[34608]:0xc07a99 at wasm://wasm/06cfd8fe:wasm-function[34609]:0xc07af0 at wasm://wasm/06cfd8fe:wasm-function[74041]:0x1502ed3 at wasm://wasm/06cfd8fe:wasm-function[42938]:0xdf8063 at invoke_vii (blob:https://192.168.1.128:55555/81b1912c-55c2-487c-98f9-9160405d93db:4542:444127) at wasm://wasm/06cfd8fe:wasm-function[74042]:0x1502fbf at wasm://wasm/06cfd8fe:wasm-function[55110]:0x11ceb11 at wasm://wasm/06cfd8fe:wasm-function[10861]:0x364af9 at wasm://wasm/06cfd8fe:wasm-function[42936]:0xdf804b at invoke_iiii (blob:https://192.168.1.128:55555/81b1912c-55c2-487c-98f9-9160405d93db:4542:442916) ```
Thaina commented 1 year ago

Is this feature was deprecated?

karasusan commented 1 year ago

@Thaina Sorry for waiting this but we haven't start to make it.

Thaina commented 1 year ago

@karasusan Are there anything blocking? I would like to have some prospect and vision of this feature so maybe I could start making pull request

karasusan commented 1 year ago

@Thaina This branch is old and isn't following the latest update. But you can check the idea. https://github.com/Unity-Technologies/com.unity.webrtc/tree/experimental/webgl-platform

Thaina commented 1 year ago

I have already check that branch but it still stay at the same place

https://github.com/Unity-Technologies/com.unity.webrtc/blob/a39e89c2b18f37c093977ce3106f1305fd03992a/Runtime/Plugins/WebGL/Context.jslib#L213-L221

Should there be anything else I need to concern before start implement this?

vertexfox commented 1 year ago

Hi!

I'm using the new experimental branch (https://github.com/Unity-Technologies/com.unity.webrtc/commits/thaina/experimental/webgl-platform) and building works, except I'm getting the following error when launching the game: image

Is this a bug or is there any config step I'm missing?

Thaina commented 1 year ago

No, I am very sorry but that branch is not really work yet. I just fork to try to implement it but never finishing. Only finish the migration to newer version

vertexfox commented 1 year ago

No, I am very sorry but that branch is not really work yet. I just fork to try to implement it but never finishing. Only finish the migration to newer version

Oh, okay. Thanks for letting me know 🫡