[REQUEST]: Android x86_64 support

ghost commented 2 years ago

Is your feature request related to a problem?

I'm trying to use this plugin for an x86_64 Android device but the x86_64 binaries are not present in: https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/Runtime/Plugins/Android/libwebrtc.aar so it fails on device due to a missing binary.

I'm just wondering if you have an obvious path forward for this? I'm happy to create a pull request if that helps. Or happy to help engineer another solution.

Describe the solution you'd like

Ideally, the Android x86_64 binaries would also live in https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/Runtime/Plugins/Android/libwebrtc.aar

Ideally they would also be updated in step with the regular updates for all binaries as the code changes.

Describe alternatives you've considered

I built the binary myself and put it into libwebrtc.aar and everything was fine. I considered making a pull request for this, but I'm not sure if this is really the correct approach because I suspect there is a more automated approach to updating the binaries. And so that pull request would be overwritten by any subsequent rebuild of the binaries. Meaning that the scripts used to generate the binaries would also need to be updated.

Additional context

No response

ghost commented 2 years ago

FWIW: This is the link to my fork: https://github.com/aaron-stafford/com.unity.webrtc

I needed to make a number of changes to get the Android x86_64 build process to work. I might be missing something obvious, but that should be a consideration in this request too.

karasusan commented 2 years ago

@aaron-stafford We are welcome to your contribution! We need to fix this build script for building aar binary. https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/BuildScripts~/build_plugin_android.sh

Of course we also have to make libwebrtc.a for Android x86_64.

ghost commented 2 years ago

Thanks @karasusan , I can potentially help to provide a solution for that build script if no one else is working on it. Is there any particular environment that the script needs to run in? Or is it expected that it would work for all of the environments specified here: https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/Plugin~/README.md ? I was a little unclear if the environments specified in that last link are purely for building the webrtc native code, or if they are also intended to be used to build the plugin. Or are they purely for building the plugin? Right now my solution works on Ubuntu 20.04 for building both the webrtc native code and the plugin but I haven't tried on anything else.

karasusan commented 2 years ago

@aaron-stafford The shell script I posted are purely for building the plugin. libwebrtc build script is here. https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/BuildScripts~/build_libwebrtc_android.sh

Unfortunatelly, we cannot provide the build environment publicly because the build machines are internal resources. But we can say that we use Ubuntu20.04 to build the native plugin for Android ARM64.

karasusan commented 2 years ago

memo: WRS-16

PlinChen commented 2 years ago

You can get android native libraries here: M92_webrtc-android_x64_arm64_armv7a.zip Then， you can build your own aar with build_plugin_android.sh

ghost commented 2 years ago

@karasusan , in the contributing page (https://github.com/Unity-Technologies/com.unity.webrtc/blob/main/CONTRIBUTING.md), its states:

By making a pull request, you confirm you agree to the terms and conditions of the Apache 2.0 Contributor License Agreement (whether Individual or Corporate) and have signed and submitted such Contributor License Agreement to the Apache 2.0 Software Foundation

Is that right? Do I need to submit a contributor license agreement to Apache 2.0 Software Foundation? Seems odd. Just want to clarify.

karasusan commented 1 year ago

@aaron-stafford When you make the PR on the repository, You can see the page to sign the agreement on GitHub. You only need to push the button on the aggeement page.

karasusan commented 1 year ago

@aaron-stafford Any questions?

ghost commented 1 year ago

@karasusan , thanks. No more question on the legal side of things for now. This seems fine.

@PlinChen , it is my understanding that build_plugin_android.sh needs modification to build the x86_64 version of the plugin. Are you suggesting that you were able to use it unchanged to generate the plugin? I don't see how that is possible, I'd love to understand how you managed it.

lumin-leaper commented 1 year ago

I think I might be able to provide a solution for this. From what I can tell, build_libwebrtc_android.sh just needs to be modified to have its target_cpu changed to include both "arm64" and "x64". This would put the binaries for both architectures into the same webrtc-android.zip file.

If this file were to be generated and upload by what ever build system you have, it could then be referenced by the build_plugin_android.sh script.

build_plugin_android.sh would need a bit more work as it appears to only build for a single architecture presently, so it would need to be modified to build for both arm64-v8a and x86_64. This would generate result in a new libwebrtc.aar file that would contain all the binary code for both architectures.

This all seems fairly trivial, if I provide the changes for this, would a solution like that be acceptable? The main issue that I see is that the URL that the build_plugin_android.sh references could not be known to me at the time I generate a pull request. I guess one solution might be to provide a PR just for the build_libwertc_android.sh initially, and once that is working and a URL to the binary is generated, I could provide a PR to the build_plugin_android.sh?

Please let me know what you think. I'd love to help out on this if I can.

karasusan commented 1 year ago

@lumin-leaper

As you mentioned, to solve this issue, we need to modify [build_libwebrtc_android.sh]() to create a static library for x86_64. this script is ran on our in-house CI system to build the library. The library will be uploaded to GitHub Release.

If we want to create a static library, it is sufficient to modify build_libwebrtc_android.sh, but to build the Unity native plugin, we need to tell the linker the location of the static library. I think this will require an update to FindWebRTC.cmake. Additionally, you will need to specify the Android architecture in build_plugin_android.sh.

lumin-leaper commented 1 year ago

@karasusan , thanks for the tip regarding FindWebRTC.cmake, a small change to this file indeed makes it easier for build_plugin_android.sh to find the library:

diff --git a/Plugin~/cmake/FindWebRTC.cmake b/Plugin~/cmake/FindWebRTC.cmake
index 55fe5b3..b812c2e 100644
--- a/Plugin~/cmake/FindWebRTC.cmake
+++ b/Plugin~/cmake/FindWebRTC.cmake
@@ -26,6 +26,9 @@ if(Windows OR Linux)
   set(WEBRTC_LIBRARY_DIR ${WEBRTC_LIBRARY_DIR}/${SYSTEM_PROCESSOR})
 endif()
 if(Android)
+  if(CMAKE_ANDROID_ARCH_ABI STREQUAL "x86_64")
+    set(CMAKE_ANDROID_ARCH "x64")
+  endif()
   set(WEBRTC_LIBRARY_DIR ${WEBRTC_LIBRARY_DIR}/${CMAKE_ANDROID_ARCH})
 endif()

This makes the change to build_plugin_android.sh a bit simpler:

diff --git a/BuildScripts~/build_plugin_android.sh b/BuildScripts~/build_plugin_android.sh
index fb0cff8..4f23d85 100755
--- a/BuildScripts~/build_plugin_android.sh
+++ b/BuildScripts~/build_plugin_android.sh
@@ -3,7 +3,6 @@
 export LIBWEBRTC_DOWNLOAD_URL=https://github.com/Unity-Technologies/com.unity.webrtc/releases/download/M107/webrtc-android.zip
 export SOLUTION_DIR=$(pwd)/Plugin~
 export PLUGIN_DIR=$(pwd)/Runtime/Plugins/Android
-export ARCH_ABI=arm64-v8a

 # Download LibWebRTC 
 curl -L $LIBWEBRTC_DOWNLOAD_URL > webrtc.zip
@@ -11,25 +10,29 @@ unzip -d $SOLUTION_DIR/webrtc webrtc.zip

 # Build UnityRenderStreaming Plugin 
 cd "$SOLUTION_DIR"
-cmake . \
-  -B build \
-  -D CMAKE_SYSTEM_NAME=Android \
-  -D CMAKE_ANDROID_API_MIN=24 \
-  -D CMAKE_ANDROID_API=24 \
-  -D CMAKE_ANDROID_ARCH_ABI=$ARCH_ABI \
-  -D CMAKE_ANDROID_NDK=$ANDROID_NDK \
-  -D CMAKE_BUILD_TYPE=Release \
-  -D CMAKE_ANDROID_STL_TYPE=c++_static
+for ARCH_ABI in "arm64-v8a" "x86_64"
+do
+  cmake . \
+    -B build \
+    -D CMAKE_SYSTEM_NAME=Android \
+    -D CMAKE_ANDROID_API_MIN=24 \
+    -D CMAKE_ANDROID_API=24 \
+    -D CMAKE_ANDROID_ARCH_ABI=$ARCH_ABI \
+    -D CMAKE_ANDROID_NDK=$ANDROID_NDK \
+    -D CMAKE_BUILD_TYPE=Release \
+    -D CMAKE_ANDROID_STL_TYPE=c++_static

-cmake \
-  --build build \
-  --target WebRTCPlugin
+  cmake \
+    --build build \
+    --target WebRTCPlugin

-# libwebrtc.so move into libwebrtc.aar
-cp -f $SOLUTION_DIR/webrtc/lib/libwebrtc.aar $PLUGIN_DIR
-pushd $PLUGIN_DIR
-mkdir -p jni/$ARCH_ABI
-mv libwebrtc.so jni/$ARCH_ABI
-zip -g libwebrtc.aar jni/$ARCH_ABI/libwebrtc.so
-rm -r jni
-popd
\ No newline at end of file
+  # libwebrtc.so move into libwebrtc.aar
+  cp -f $SOLUTION_DIR/webrtc/lib/libwebrtc.aar $PLUGIN_DIR
+  pushd $PLUGIN_DIR
+  mkdir -p jni/$ARCH_ABI
+  mv libwebrtc.so jni/$ARCH_ABI
+  zip -g libwebrtc.aar jni/$ARCH_ABI/libwebrtc.so
+  rm -r jni
+  popd
+  rm -rf build
+done
\ No newline at end of file

The diff makes it look more complicated than it is, the majority of the code is just put into a for loop to iterate over both architectures. Notably, I also remove the build folder between builds. This may be unnecessary, but I did it out of an abundance of caution. Also, hopefully obviously, this wouldn't be the final patch because LIBWEBRTC_DOWNLOAD_URL still references the original url. As mentioned previously, a new version of libwebrtc would need to be generated and uploaded before the correct url could be specified here.

To build the native library, the changes to build_libwebrtc_android.sh are trivial as thought:

diff --git a/BuildScripts~/build_libwebrtc_android.sh b/BuildScripts~/build_libwebrtc_android.sh
index 05b76f2..d571042 100755
--- a/BuildScripts~/build_libwebrtc_android.sh
+++ b/BuildScripts~/build_libwebrtc_android.sh
@@ -41,7 +41,7 @@ popd

 mkdir -p "$ARTIFACTS_DIR/lib"

-for target_cpu in "arm64"
+for target_cpu in "arm64" "x64"
 do
   mkdir -p "$ARTIFACTS_DIR/lib/${target_cpu}"

@@ -83,7 +83,7 @@ do
   "$PYTHON3_BIN" tools_webrtc/android/build_aar.py \
     --build-dir $OUTPUT_DIR \
     --output $OUTPUT_DIR/libwebrtc.aar \
-    --arch arm64-v8a \
+    --arch arm64-v8a x86_64 \
     --extra-gn-args "is_debug=${is_debug} \
       is_java_debug=${is_debug} \
       rtc_use_h264=false \

Should I put these changes to build_libwebrtc_android.sh into a Pull Request so that the library can be generated? Once that is done I could provide the final changes to build_plugin_android.sh.

karasusan commented 1 year ago

@lumin-leaper Sure, please make the pull request for the patch of build_libwebrtc_android.sh. If so, we will update the library on the GitHub.

JacopoMarullo commented 1 year ago

Hello, I am also interested in using Unity WebRTC for Android x86_64 (specifcally for Magic Leap2). Is there any activity in progress on this side? @lumin-leaper did you finally submit your PR?

Thanks a lot Jacopo

lumin-leaper commented 1 year ago

@JacopoMarullo , I'm just waiting on the green light from my company's legal team. The company as a whole is going through a process to re-evaluate all open source project contributions. It's taken a bit longer than I thought it might, but I think we are close to having it all sorted out. Thanks for poking the thread. It gives me the opportunity to escalate it again.

JacopoMarullo commented 1 year ago

Thanks a lot for the clarification! Please, keep me update if you there are some news about it.

Thanks again Jacopo

karasusan commented 1 year ago

@lumin-leaper @JacopoMarullo Hi, if we test the architecture Android X86, which device do you prefer to test for? I haven't tested that arch in our environment.

lumin-leaper commented 1 year ago

@karasusan , I'm targeting the Magic Leap 2 device. From @JacopoMarullo comments, it looks like we are targeting the same device.

JacopoMarullo commented 1 year ago

Hi guys, yes I'm targeting the Magic Leap 2 device

crucial-sailor commented 1 year ago

@lumin-leaper do you have any updates on this? My team would also be interested in using the library on an x64 Android device.

lumin-leaper commented 1 year ago

I have branch ready to create the pull request from: https://github.com/lumin-leaper/com.unity.webrtc/tree/issue-802-part-1

However, it looks like the tip of mainline is not working the same for me compared to when I first built this code several months ago. I now need the following additional change for build_libwebrtc_android.sh to run correctly:

diff --git a/BuildScripts~/build_libwebrtc_android.sh b/BuildScripts~/build_libwebrtc_android.sh
index 05b76f2..b74d6c0 100755
--- a/BuildScripts~/build_libwebrtc_android.sh
+++ b/BuildScripts~/build_libwebrtc_android.sh
@@ -2,7 +2,10 @@

 if [ ! -e "$(pwd)/depot_tools" ]
 then
-  git clone --depth 1 https://chromium.googlesource.com/chromium/tools/depot_tools.git
+  git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
+  pushd depot_tools
+  git reset --hard 96b57b9b59091007162adfa4035a6383e9757815
+  popd
 fi

 export COMMAND_DIR=$(cd $(dirname $0); pwd)
@@ -11,6 +14,7 @@ export WEBRTC_VERSION=5304
 export OUTPUT_DIR="$(pwd)/out"
 export ARTIFACTS_DIR="$(pwd)/artifacts"
 export PYTHON3_BIN="$(pwd)/depot_tools/python-bin/python3"
+export DEPOT_TOOLS_UPDATE=0

 if [ ! -e "$(pwd)/src" ]
 then

To me it appears as though recent changes in depot_tools is not compatible with the tip of this repo. The change above simply pegs depot_tools to a known working version.

So I'm a bit unsure about how to proceed at the moment. @karasusan , are you aware of this issue? Should I raise a bug for it? Or is it not actually a problem in your build environment? Should I submit my PR any way?

kannan-xiao4 commented 1 year ago

@lumin-leaper The dependency of depot tools is now described in the webrtc library, but at present it is not possible to follow it. So, libwebrtc upgrade is required and we proceed that.

lumin-leaper commented 1 year ago

@kannan-xiao4, just to clarify, you are saying that work will be done to get libwebrtc building with the head of depot_tools prior to considering a pull request from me?

kannan-xiao4 commented 1 year ago

@lumin-leaper I think the changes you're showing will work correctly on the current branch. However, I think that if we upgrade from M107 to M112 libwebrtc, it will be unnecessary code. We have confirmed that upgrading to M112 will fix the depot_tools issue.

lumin-leaper commented 1 year ago

Thanks @kannan-xiao4 I see that libwebrtc builds cleanly now with modification. I've submitted the first pull request as discussed in this thread: https://github.com/Unity-Technologies/com.unity.webrtc/pull/932 I look forward to the feedback.

crucial-sailor commented 1 year ago

@lumin-leaper Is there anything else you did to make the generated .aar file work on the MagicLeap 2?

After copying the required project folders from the repo (Runtime (which contains the newly generated .aar), Editor and Tests folder, as well as the package.json), importing the folder in Unity and starting the project on the MagicLeap, the app stays blank / seems to be broken. The same behaviour could not be observed when the "original" webrtc package is used instead (even though that one does not actually run on the MagicLeap 2 of course).

For us, the build process went fine, and the .aar file also seems to contain all necessary files when compared to the one in the webrtc_android.zip file, which can be downloaded in Releases. Could it be that some import settings are wrong, or the build is broken, even though it does not report to be so?

lumin-leaper commented 1 year ago

@oculavis-cvar , I'd need to know more. Where are you getting your webrtc_android.zip and .aar files from? Are these all generated yourself from the information above? Or are you attempting to use files generated from the pull request?

With respect to the pull request I made yesterday, it is only the first part. It allows libwebrtc to be built. But there is a second pull request that will be needed to actually generate the plugin (.aar file) correctly.

If you generated it yourself from the above information, you need to be a bit careful with the generation of the plugin (which is why this is broken up into two pull requests). The code to generate the plugin that I posted earlier in this thread, if used literally, will result in a slightly broken .aar file. The build_plugin_android.sh file will need to look a bit more like this: https://github.com/lumin-leaper/com.unity.webrtc/blob/issue-802-part-2-build-test/BuildScripts~/build_plugin_android.sh (that is, note the commenting out of the curl command). Also in my original post above, technically there is an error (note the position of this line of code: cp -f $SOLUTION_DIR/webrtc/lib/libwebrtc.aar $PLUGIN_DIR). It needs to be moved out of the loop otherwise it'll overwrite previous loops. The correct implementation is also in that same link. That link is just for testing locally though. Due to commenting out the curl command it can't be part of the pull request.

If you are still seeing issues, please explain further how you are generating the files for the plugin.

crucial-sailor commented 1 year ago

@lumin-leaper Currently, we only build the x64 version of the plugin. We started with your proposed changes to FindWebRTC.cmake and build_libwebrtc_android.sh, but only building for x64 Android. The build worked fine, we did not see any error messages, and the resulting webrtc-android.zip looked fine too. We extracted its contents manually to the folder required by build_plugin_android.sh, then started the build script, again just for x86_64 as a target.

The resulting output is written to Runtime/Plugins/Android/libwebrtc.aar. Given the structure of the Unity plugin, we then copied all folders relevant to import into Unity in a "com.unity.webrtc" folder, including the full Runtime, Documentation~, Editor, Samples~ and Tests folder, and the package.json file, all from the repository. Then, we imported this folder in Unity. Finally, we built for the MagicLeap. There were no build errors, and also no code errors in the Editor.

Still, the MagicLeap app is broken as soon as the plugin is loaded. Without it, it works fine. With it, the app starts and stays blank. No loading screen. No Unity logo. Nothing. It does not even crash, or produce any log in the console or anything useful in logcat.

That's why we decided to post here, because we are a bit at a loss. Below I'll post the only file changes we've made in the repository (they're not very clean, but should work for testing locally). The only potential problem I could come up with is that we can't just take the files from the repository to put together the plugin to create for Unity.

diff --git a/BuildScripts~/build_libwebrtc_android.sh b/BuildScripts~/build_libwebrtc_android.sh
index 36d6e43..b00bc61 100755
--- a/BuildScripts~/build_libwebrtc_android.sh
+++ b/BuildScripts~/build_libwebrtc_android.sh
@@ -50,7 +50,7 @@ popd
 mkdir -p "$ARTIFACTS_DIR/lib"

-for target_cpu in "arm64"
+for target_cpu in "x64"
 do
   mkdir -p "$ARTIFACTS_DIR/lib/${target_cpu}"

@@ -94,7 +94,7 @@ do
   "$PYTHON3_BIN" tools_webrtc/android/build_aar.py \
     --build-dir $OUTPUT_DIR \
     --output $OUTPUT_DIR/libwebrtc.aar \
-    --arch arm64-v8a \
+    --arch x86_64 \
     --extra-gn-args "is_debug=${is_debug} \
       is_java_debug=${is_debug} \
       rtc_use_h264=false \
diff --git a/BuildScripts~/build_plugin_android.sh b/BuildScripts~/build_plugin_android.sh
index 04873c1..1c42fec 100755
--- a/BuildScripts~/build_plugin_android.sh
+++ b/BuildScripts~/build_plugin_android.sh
@@ -3,14 +3,18 @@
 export LIBWEBRTC_DOWNLOAD_URL=https://github.com/Unity-Technologies/com.unity.webrtc/releases/download/M112/webrtc-android.zip
 export SOLUTION_DIR=$(pwd)/Plugin~
 export PLUGIN_DIR=$(pwd)/Runtime/Plugins/Android
-export ARCH_ABI=arm64-v8a
+export ARCH_ABI=x86_64

 # Download LibWebRTC 
-curl -L $LIBWEBRTC_DOWNLOAD_URL > webrtc.zip
-unzip -d $SOLUTION_DIR/webrtc webrtc.zip 
+# TODO INSTEAD COPY OVER THE RESULTS OF build_libwebrtc_android.sh
+# Make sure the keep the same folder structure!
+# curl -L $LIBWEBRTC_DOWNLOAD_URL > webrtc.zip
+# unzip -d $SOLUTION_DIR/webrtc webrtc.zip 

 # Build UnityRenderStreaming Plugin 
 cd "$SOLUTION_DIR"
+# for ARCH_ABI in "arm64-v8a" "x86_64"
+# do
 cmake . \
   -B build \
   -D CMAKE_SYSTEM_NAME=Android \
@@ -32,4 +36,6 @@ mkdir -p jni/$ARCH_ABI
 mv libwebrtc.so jni/$ARCH_ABI
 zip -g libwebrtc.aar jni/$ARCH_ABI/libwebrtc.so
 rm -r jni
-popd
\ No newline at end of file
+popd
+# rm -rf build
+# done
\ No newline at end of file
diff --git a/Plugin~/cmake/FindWebRTC.cmake b/Plugin~/cmake/FindWebRTC.cmake
index 55fe5b3..9c9a7f6 100644
--- a/Plugin~/cmake/FindWebRTC.cmake
+++ b/Plugin~/cmake/FindWebRTC.cmake
@@ -26,6 +26,7 @@ if(Windows OR Linux)
   set(WEBRTC_LIBRARY_DIR ${WEBRTC_LIBRARY_DIR}/${SYSTEM_PROCESSOR})
 endif()
 if(Android)
+  set(CMAKE_ANDROID_ARCH "x64")
   set(WEBRTC_LIBRARY_DIR ${WEBRTC_LIBRARY_DIR}/${CMAKE_ANDROID_ARCH})
 endif()

lumin-leaper commented 1 year ago

@oculavis-cvar , its a bit hard to say for sure with what you posted. Some of the changes are a little bit different to what I posted, but mostly it looks correct. This does not look correct though. +# for ARCH_ABI in "arm64-v8a" "x86_64" This loop will also try to include the arm binaries. Given your comment about only building for x64, i'm not sure what would happen if they didn't exist. I presume the script would error out. But perhaps not.

This is also a bit ambiguous:

+# TODO INSTEAD COPY OVER THE RESULTS OF build_libwebrtc_android.sh
+# Make sure the keep the same folder structure!
+# curl -L $LIBWEBRTC_DOWNLOAD_URL > webrtc.zip
+# unzip -d $SOLUTION_DIR/webrtc webrtc.zip

You might be doing the right thing here, but it's hard to tell for sure. If you unzip the final version of Runtime/Plugins/Android/libwebrtc.aar and print the folder/file structure it might help to confirm it is okay. I'd also suggest running objdump on the binaries that you built to confirm that they were actually built for x86_64 (I managed once to generate x86_64 names files that were actually arm). Although if this were the case, you should see a message in logcat that the webrtc library failed to load (it doesn't crash the app though).

we then copied all folders relevant to import into Unity in a "com.unity.webrtc" folder I'm a bit unclear if you mean you are using the folders as the basis for a new unity project or if you are still using the artifacts as a plugin. I've only ever used these artifacts as a plugin. So i'm not sure what would happen if you tried to turn it directly into a project.

The only other thing that comes to mind (if all of the above is ok) is that you have correctly added all of the scene to your build target. I suspect if you didn't add any scenes, you might get behavior like what you describe, but again. I'm not sure. It may or may not be obvious, but your build settings need to target Android, you need to build for il2cpp and you need to target x86_64.

Hopefully that helps. If not, you could try regenerating the plugin using the script I have in this branch: https://github.com/lumin-leaper/com.unity.webrtc/tree/issue-802-part-2-build-test It is designed to build the plugin from a custom artifact rather than from the downloaded url. You just need to copy the webrtc-android.zip file (generated from the pervious libwebrtc script) and it'll do the rest. This would eliminate any difference in our process.

crucial-sailor commented 1 year ago

@lumin-leaper Regarding the lines you commented on, you are right that they could be problematic, but I commented them out - this is hard to read in the diff, so for that it may have been better to remove them, to avoid confusion. Also, it's sadly not a scene issue - if I remove the libwebrtc.aar file from the Unity project, the scene loads and everything works fine (as long as I don't load the modules that use WebRTC of course). As soon as libwebrtc.aar is in the project, the app never loads and stays blank - most likely because of "load on startup" for libwebrtc.aar, so something must be wrong with the build output for sure, which prevents the app from loading any further.

As for objdump, objdump -p libwebrtc.so gives libwebrtc.so: file format elf64-x86-64 (and file format elf64-little for the arm folder). So that should be fine as well. The folder structure of libwebrtc.aar also looks fine.

I tried to use only your changes now - thanks for also linking https://github.com/lumin-leaper/com.unity.webrtc/tree/issue-802-part-2-build-test. I adapted the three files, built the library and exchanged libwebrtc.aar only in Unity. Sadly, I see the same behaviour as before.

One reason could be that I'm not sure if https://github.com/Unity-Technologies/com.unity.webrtc/blob/main/Plugin~/README.md still is up to date. For example, it states to download android-ndk-r21b-linux-x86_64.zip, then references android-ndk-r21d/ in the next line. I am using r21e. I hope that does not cause issues. Also, maybe the cmake setup went wrong, although cmake --version gives me the correct info after the build, and I doubt that the build would finish without errors if cmake would be broken.

PS: Here are the only outstanding logcat outputs (I replaced some parts with "..."):

2023/07/10 10:07:09.241 10071 10071 Error ... Not starting debugger since process cannot load the jdwp agent.
2023/07/10 10:07:09.811 10071 10092 Error libprocessgroup set_timerslack_ns write failed: Operation not permitted
2023/07/10 10:07:09.899 10071 10071 Warn UnityMain type=1400 audit(0.0:338): avc: denied { search } for name="traces" dev="..." ino=... scontext=u:r:untrusted_app:s0:c109,c256,c512,c768 tcontext=u:object_r:trace_data_file:s0 tclass=dir permissive=0

crucial-sailor commented 1 year ago

@lumin-leaper To add to that, I also shortly tried to run the project on an arm64 device. I could not test an actual WebRTC call there, but I could check if the custom build plugin prevents the app from loading. On ARM64, the app loads fine. I still get errors like the one above though, e.g.:

2023/07/11 11:25:07.901 14377 14394 Error libprocessgroup set_timerslack_ns write failed: Operation not permitted

That may also be Unity-related though.

karasusan commented 1 year ago

@lumin-leaper I merged your PR #932 and uploaded the libwebrtc m112 on GitHub Release.

crucial-sailor commented 1 year ago

It appears that there may be an issue with either the build_plugin_android.sh or our build setup for the Unity project. Despite using a fresh clone and the newly released webrtc_android.zip for M112, we are still encountering a black screen in the project when we include the resulting .aar file on Magic Leap 2. We're continuing to investigate this.

To check if our project setup is the cause of the issue, we opened the MagicLeap Examples Unity project and built it, with the HelloCube scene as the first scene in build. It loaded fine - even when adding the custom WebRTC build. BUT: As soon as any script that uses WebRTC is added (which is probably what triggers Unity to import the .aar at all during build), we have the same issue again: a black screen is greeting us. The test script is shown below. (Note: The app goes black immediately, even though we don't use WebRTC in the first 10 seconds). It works fine again if we remove the .aar (it will throw an exception then after 10 seconds).

public class TestScript : MonoBehaviour
{
    [SerializeField] Text TestText;

    // Start is called before the first frame update
    void Start()
    {
        StartCoroutine(TestPublisher());
    }

    private IEnumerator TestPublisher()
    {
        yield return new WaitForSecondsRealtime(10);

        TestText.text += "Creating and destroying a participant\n";
        // Testing Unity WebRTC with something very simple:
        var participant = new RTCPeerConnection();
        UnityEngine.Debug.Log("Created");
        TestText.text += "Created\n";

        yield return new WaitForSecondsRealtime(10);

        participant.Close();
        participant.Dispose();
        TestText.text += "Destroyed\n";
    }
}

lumin-leaper commented 1 year ago

@lumin-leaper I merged your PR #932 and uploaded the libwebrtc m112 on GitHub Release.

Thanks @karasusan! I'll re-test and prepare the second Pull Request.

lumin-leaper commented 1 year ago

@oculavis-cvar, In your testing, are you always using some Magic Leap SDK/plugin? Or have you also tested the WebRTC changes without any Magic Leap code?

crucial-sailor commented 1 year ago

@lumin-leaper Good idea, we'll test it in an empty project without importing the MagicLeap SDK tomorrow.

crucial-sailor commented 1 year ago

@lumin-leaper Thanks again for the suggestion! The app starts and we can create a participant and an SDP without the MagicLeap SDK. We now assume that there must be a conflict with the MLWebRTC lib in the MagicLeap SDK. Also, thank you for your work to support Unity WebRTC on x86 Android devices!

lumin-leaper commented 1 year ago

@oculavis-cvar, thanks for the update, that is good to know. If you need help root causing the issue as it relates to Magic Leap SDK, I'd recommend raising the issue on https://forum.magicleap.cloud/

lumin-leaper commented 1 year ago

@karasusan , you'll likely see that I have created the second pull request: https://github.com/Unity-Technologies/com.unity.webrtc/pull/952 which adds building of x86_64 Android Plugin binaries.

karasusan commented 1 year ago

@lumin-leaper I am checking your PR. Thanks. However, we don't have Android device for x64, so I will test it on the Android Simulatror.

karasusan commented 1 year ago

Merged #952. Do you have other topics to solve this issue?

lumin-leaper commented 1 year ago

Thanks @karasusan . The only item left from my perspective is to commit the new version of libwebrtc.aar to Runtime/Plugins/Android/libwebrtc.aar I presume that this would need to be done from your side as you generate the binary? Do you have a plan for when that might happen?

karasusan commented 1 year ago

@lumin-leaper Please use this changeset. https://github.com/Unity-Technologies/com.unity.webrtc/commit/2e76cf79e4051b20e80f33e4d5f6f4c3d6a64c87

We will push the binaries to main branch afetr testing the package.

lumin-leaper commented 1 year ago

ok, thanks @karasusan , looks good to me :)

lumin-leaper commented 1 year ago

@karasusan are you able to provide any guidance one when this will be released?

karasusan commented 1 year ago

@lumin-leaper It would be released in next month.

crucial-sailor commented 1 year ago

Greetings,

now that Unity WebRTC works on the ML2, we are having issues with the performance with VP8. We are using the patches for the next release of Unity WebRTC.

Even though this device only supports software encoding for VP8, we expected acceptable performance, as any laptop with a similar processor should also be able to provide at least 30FPS at 720p resolution – we are only at 15FPS though.

Also, H264 encoding did not work for us at all.

We are wondering if any of the patches or the downgrade of the Java version may affect performance on the MagicLeap, or if there are compilation flags that may need to be added to optimize performance for x64 Android?

We also invite others to share their experience with WebRTC performance on the MagicLeap 2 or other x64 Android devices here. Maybe they have a better experience than we do.

We also want to share some additional information from debugging:

According to CPU statistics from the Unity Editor, the MagicLeap 2 can keep constant 60FPS for the user.
When publishing a single video stream, CPU usage (according to Android Studio) goes up from 11% to 28%. While this is a huge increase, there still seems to be a lot of unused CPU time.
On the Meta Quest Pro, we see a similar increase, but video FPS are much higher. The chipset of the Meta Quest Pro should not be more powerful than the one used in the Magic Leap 2, according to our short research.

We get the following results when we compare different platforms, when taking a look at the average frame encoding time in ms, with VP8, software encoding, FullHD:

Meta Quest Pro (Android, ARM64): 40ms
MagicLeap 2 (Android, x64): 159ms
Unity Desktop (Windows, x64): 14ms

The performance on the Meta Quest Pro is very unstable though, the worst encode time was at 80ms. Desktop and ML2 are both very stable regarding their encode times, no matter how much the user is moving their head / how much the image is changing.

karasusan commented 1 year ago

@crucial-sailor Thank you for sharing detail. Which Graphics API did you use for this testing?

crucial-sailor commented 1 year ago

@karasusan For the Meta Quest Pro, the list contains Vulkan as a first entry, then OpenGLES3, so I would assume it uses Vulkan. For the Magic Leap 2, we use Vulkan as well.

But I'm also not sure how big of an influence Unity settings have on the underlying native WebRTC package. The performance metrics come directly from the WebRTC stats, so it's the frame encoding itself which is surprisingly slow on x64 Android, and not the way frames are passed down to the native library.

Unity-Technologies / com.unity.webrtc