search

devopvoid / webrtc-java

WebRTC for desktop platforms running Java
Apache License 2.0
248 stars 60 forks source link

Custom video sink #127

Open kalgecin opened 8 months ago

kalgecin commented 8 months ago 
val peerConnectionFactory = PeerConnectionFactory(audioDeviceModule)

val customVideoSource = CustomVideoSource()
val customTrack = peerConnectionFactory.createVideoTrack("custom-src", customVideoSource)
connection = peerConnectionFactory.createPeerConnection(config, peerObserver)
connection.addTrack(customTrack, listOf(customTrack.id))

val data = //generate yuv frame byte array 
val nativeI420Buffer = NativeI420Buffer.allocate(width, height)
nativeI420Buffer.dataY.put(0, data, 0, ySize)
nativeI420Buffer.dataU.put(0, data, ySize, uvSize)
nativeI420Buffer.dataV.put(0, data, ySize + uvSize, uvSize)
val videoFrame = VideoFrame(nativeI420Buffer, 0, System.currentTimeMillis() * 1000000)

customVideoSource.OnFrameCaptured(videoFrame)
videoFrame.release()
voqaldev commented 5 months ago

Thanks for this, @kalgecin. I can confirm this works for custom video sources.

luokeith commented 1 week ago 
val peerConnectionFactory = PeerConnectionFactory(audioDeviceModule)

val customVideoSource = CustomVideoSource()
val customTrack = peerConnectionFactory.createVideoTrack("custom-src", customVideoSource)
connection = peerConnectionFactory.createPeerConnection(config, peerObserver)
connection.addTrack(customTrack, listOf(customTrack.id))

val data = //generate yuv frame byte array 
val nativeI420Buffer = NativeI420Buffer.allocate(width, height)
nativeI420Buffer.dataY.put(0, data, 0, ySize)
nativeI420Buffer.dataU.put(0, data, ySize, uvSize)
nativeI420Buffer.dataV.put(0, data, ySize + uvSize, uvSize)
val videoFrame = VideoFrame(nativeI420Buffer, 0, System.currentTimeMillis() * 1000000)

customVideoSource.OnFrameCaptured(videoFrame)
videoFrame.release()

Thank you very much for the code, can you provide a demo of the java syntax? I didn't understand the meaning of this piece of code. Thank you so much! “ nativeI420Buffer.dataY.put(0, data, 0, ySize) nativeI420Buffer.dataU.put(0, data, ySize, uvSize) nativeI420Buffer.dataV.put(0, data, ySize + uvSize, uvSize) “

kalgecin commented 1 week ago

I don't do much java as of late, so can't make a fully java example, but it should not differ from the example I provided (except for val and absence of new keyword)

nativeI420Buffer.dataY.put(0, data, 0, ySize)
nativeI420Buffer.dataU.put(0, data, ySize, uvSize)
nativeI420Buffer.dataV.put(0, data, ySize + uvSize, uvSize)

Here, I have a byte array buffer in which [0 to ysize] is the Y frame, then the next uvSize bytes are the U frame and the rest is the V frame. And i'm copying the values with the ranges specified into the nativeI420Buffer (which has the frames as separate buffers). So essentially i'm splitting my one array into the 3 byte buffers

https://download.java.net/java/early_access/panama/docs/api/java.base/java/nio/ByteBuffer.html#put(int,byte%5B%5D,int,int)

luokeith commented 1 week ago

I don't do much java as of late, so can't make a fully java example, but it should not differ from the example I provided (except for val and absence of new keyword)

nativeI420Buffer.dataY.put(0, data, 0, ySize)
nativeI420Buffer.dataU.put(0, data, ySize, uvSize)
nativeI420Buffer.dataV.put(0, data, ySize + uvSize, uvSize)

Here, I have a byte array buffer in which [0 to ysize] is the Y frame, then the next uvSize bytes are the U frame and the rest is the V frame. And i'm copying the values with the ranges specified into the nativeI420Buffer (which has the frames as separate buffers). So essentially i'm splitting my one array into the 3 byte buffers

https://download.java.net/java/early_access/panama/docs/api/java.base/java/nio/ByteBuffer.html#put(int,byte%5B%5D,int,int)

I got it, thank you so much! @kalgecin

luokeith commented 1 week ago

Hello @kalgecin , I'd like to ask you a question. I'm encountering a difficult-to-understand issue. The same piece of code can successfully execute 'mvn install' on macOS, but on the Windows 10 x64 platform, running 'mvn install' results in a compilation failure. The error log is as follows:

Creating library C:/Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/Release/webrtc-java.lib and object C:/Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/Release/webrtc-java.exp VideoTrackDeviceSource.obj : error LNK2001: unresolved external symbol "public: class std::basic_string<char,struct std::char_traits,class std::allocator > cdecl webrtc::webrtc_sequence_checker_internal::SequenceCheckerImpl::ExpectationToString(void)const " (?ExpectationToString@SequenceCheckerImpl@webrtc_sequence_checker_internal@webrtc@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj] JNI_CustomVideoSource.obj : error LNK2001: unresolved external symbol "public: class std::basic_string<char,struct std::char_traits,class std::allocator > __cdecl webrtc::webrtc_sequence_checker_internal::SequenceCheckerImpl::ExpectationToString(void)const " (?ExpectationToString@SequenceCheckerImpl@webrtc_sequence_checker_internal@webrtc@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj] JNI_VideoDesktopSource.obj : error LNK2001: unresolved external symbol "public: class std::basic_string<char,struct std::char_traits,class std::allocator > cdecl webrtc::webrtc_sequence_checker_internal::SequenceCheckerImpl::ExpectationToString(void)const " (?ExpectationToString@SequenceCheckerImpl@webrtc_sequence_checker_internal@webrtc@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj] JNI_VideoDeviceSource.obj : error LNK2001: unresolved external symbol "public: class std::basic_string<char,struct std::char_traits,class std::allocator > cdecl webrtc::webrtc_sequence_checker_internal::SequenceCheckerImpl::ExpectationToString(void)const " (?ExpectationToString@SequenceCheckerImpl@webrtc_sequence_checker_internal@webrtc@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj] VideoTrackDesktopSource.obj : error LNK2001: unresolved external symbol "public: class std::basic_string<char,struct std::char_traits,class std::allocator > cdecl webrtc::webrtc_sequence_checker_internal::SequenceCheckerImpl::ExpectationToString(void)const " (?ExpectationToString@SequenceCheckerImpl@webrtc_sequence_checker_internal@webrtc@@QEBA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@XZ) [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj] webrtc.lib(wgc_capture_source.obj) : error LNK2019: unresolved external symbol __imp_DwmGetWindowAttribute referenced in function "public: virtual struct ABI::Windows::Graphics::SizeInt32 __cdecl webrtc::WgcWindowSource::GetSize(void)" (?GetSize@WgcWindowSource@webrtc@@UEAA?AUSizeInt32@Graphics@Windows@ABI@@XZ) [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj] C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/Release/webrtc-java.dll : fatal error LNK1120: 2 unresolved externals [C:\Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64/webrtc-java.vcxproj]

Additional Information：

-- Building for: Visual Studio 16 2019 -- Selecting Windows SDK version 10.0.20348.0 to target Windows 10.0.22631. -- The C compiler identification is MSVC 19.29.30154.0 -- The CXX compiler identification is MSVC 19.29.30154.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30133/bin/Hostx64/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- WebRTC checkout path: C:/Users/18500/webrtc -- WebRTC checkout branch: branch-heads/5572 -- WebRTC target CPU: x64 -- WebRTC build type: Release -- WebRTC install path: C:/Users/18500/webrtc/build -- WebRTC: Compiled version found 'C:/Users/18500/webrtc/src/out/x64/obj/webrtc.lib' -- Configuring done -- Generating done -- Build files have been written to: C:/Dev_Env/Project/webrtc-java/webrtc-jni/target/windows-x86_64