search

OpenNetLab / AlphaRTC

Evaluation framework for RL-based bitrate control for WebRTC
BSD 3-Clause "New" or "Revised" License
94 stars 37 forks source link

How to compile shared libraries & docker build gets stuck #74

Closed AerysNan closed 3 years ago

AerysNan commented 3 years ago

Hello, I wonder whether there are any instructions about how the shared libraries under /modules/third_party/onnxinfer/lib are compiled. Since their source codes seem not to be included in this repo, it feels a little bit hard to me to imagine how to fit an onnx model into it. Though we can directly specify the onnx model path in the configuration file, what the input and output look like is still unclear to me. I'd appreciate it if some detailed descriptions (or the source codes of the shared libraries, directly) could be provided. Thanks a lot.

Pterosaur commented 3 years ago

Hello , sorry for too late response. If you want to use ONNX model, we assume you will have a pre-compiled shared library and put it in https://github.com/OpenNetLab/AlphaRTC/tree/main/modules/third_party/onnxinfer/lib. But we will not release the source codes of shared libraries due to some company policy.

But right now, we also provide a python interface for AlphaRTC to control the bandwidth (https://github.com/OpenNetLab/AlphaRTC#pyinfer), It will be easier to developer. Please try it. And here is an example about this interface: https://github.com/OpenNetLab/Challenge-Example

AerysNan commented 3 years ago

Hello , sorry for too late response. If you want to use ONNX model, we assume you will have a pre-compiled shared library and put it in https://github.com/OpenNetLab/AlphaRTC/tree/main/modules/third_party/onnxinfer/lib. But we will not release the source codes of shared libraries due to some company policy.

But right now, we also provide a python interface for AlphaRTC to control the bandwidth (https://github.com/OpenNetLab/AlphaRTC#pyinfer), It will be easier to developer. Please try it. And here is an example about this interface: https://github.com/OpenNetLab/Challenge-Example

Thanks for your kind reply. But recently I encountered another problem. When trying to build AlphaRTC using the command make all, the command will get stuck at gclient sync in docker when running hooks. Part of the output message is as follow:

________ running 'vpython src/build/landmines.py --landmine-scripts src/tools_webrtc/get_landmines.py --src-dir src' in '/app/AlphaRTC'
/home/onl/.vpython-root/fe8f2e/bin/python: can't open file 'src/tools_webrtc/get_landmines.py': [Errno 2] No such file or directory

Hook 'vpython src/build/landmines.py --landmine-scripts src/tools_webrtc/get_landmines.py --src-dir src' took 10.53 secs
________ running 'vpython src/third_party/depot_tools/update_depot_tools_toggle.py --disable' in '/app/AlphaRTC'
________ running 'vpython src/build/linux/sysroot_scripts/install-sysroot.py --arch=i386' in '/app/AlphaRTC'
Installing Debian sid i386 root image: /app/AlphaRTC/src/build/linux/debian_sid_i386-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/36ed2ee83109acdd44af2e522a5e9e0082b074a6/debian_sid_i386_sysroot.tar.xz
________ running 'vpython src/build/linux/sysroot_scripts/install-sysroot.py --arch=amd64' in '/app/AlphaRTC'
Installing Debian sid amd64 root image: /app/AlphaRTC/src/build/linux/debian_sid_amd64-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/52cf2961a3cddc0d46e1a2f7d9bf376fc16a61de/debian_sid_amd64_sysroot.tar.xz
________ running 'vpython src/tools/clang/scripts/update.py' in '/app/AlphaRTC'
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-n353803-99ac9ce7-1.tgz .......... Done.
________ running 'vpython src/build/util/lastchange.py -o src/build/util/LASTCHANGE' in '/app/AlphaRTC'
________ running 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s src/buildtools/linux64/clang-format.sha1' in '/app/AlphaRTC'
0> Downloading src/buildtools/linux64/clang-format@1baf0089e895c989a311b6a38ed94d0e8be4c0a7...
Downloading 1 files took 22.099044 second(s)
Hook 'download_from_google_storage --no_resume '--platform=linux*' --no_auth --bucket chromium-clang-format -s src/buildtools/linux64/clang-format.sha1' took 22.49 secs
________ running 'download_from_google_storage --no_resume --extract --no_auth --bucket chromium-fonts -s src/third_party/test_fonts/test_fonts.tar.gz.sha1' in '/app/AlphaRTC'
0> Downloading src/third_party/test_fonts/test_fonts.tar.gz@a22de844e32a3f720d219e3911c3da3478039f89...
0> Extracting 30 entries from src/third_party/test_fonts/test_fonts.tar.gz to src/third_party/test_fonts/test_fonts
Downloading 1 files took 8.430969 second(s)

After the last line, the make process gets stuck. Is there any hint about why this happens? Thanks a lot.

Pterosaur commented 3 years ago

Hello , sorry for too late response. If you want to use ONNX model, we assume you will have a pre-compiled shared library and put it in https://github.com/OpenNetLab/AlphaRTC/tree/main/modules/third_party/onnxinfer/lib. But we will not release the source codes of shared libraries due to some company policy. But right now, we also provide a python interface for AlphaRTC to control the bandwidth (https://github.com/OpenNetLab/AlphaRTC#pyinfer), It will be easier to developer. Please try it. And here is an example about this interface: https://github.com/OpenNetLab/Challenge-Example

Thanks for your kind reply. But recently I encountered another problem. When trying to build AlphaRTC using the command make all, the command will get stuck at gclient sync in docker when running hooks. Part of the output message is as follow:

________ running 'vpython src/build/landmines.py --landmine-scripts src/tools_webrtc/get_landmines.py --src-dir src' in '/app/AlphaRTC'
/home/onl/.vpython-root/fe8f2e/bin/python: can't open file 'src/tools_webrtc/get_landmines.py': [Errno 2] No such file or directory

Hook 'vpython src/build/landmines.py --landmine-scripts src/tools_webrtc/get_landmines.py --src-dir src' took 10.53 secs
________ running 'vpython src/third_party/depot_tools/update_depot_tools_toggle.py --disable' in '/app/AlphaRTC'
________ running 'vpython src/build/linux/sysroot_scripts/install-sysroot.py --arch=i386' in '/app/AlphaRTC'
Installing Debian sid i386 root image: /app/AlphaRTC/src/build/linux/debian_sid_i386-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/36ed2ee83109acdd44af2e522a5e9e0082b074a6/debian_sid_i386_sysroot.tar.xz
________ running 'vpython src/build/linux/sysroot_scripts/install-sysroot.py --arch=amd64' in '/app/AlphaRTC'
Installing Debian sid amd64 root image: /app/AlphaRTC/src/build/linux/debian_sid_amd64-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/52cf2961a3cddc0d46e1a2f7d9bf376fc16a61de/debian_sid_amd64_sysroot.tar.xz
________ running 'vpython src/tools/clang/scripts/update.py' in '/app/AlphaRTC'
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-n353803-99ac9ce7-1.tgz .......... Done.
________ running 'vpython src/build/util/lastchange.py -o src/build/util/LASTCHANGE' in '/app/AlphaRTC'
________ running 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s src/buildtools/linux64/clang-format.sha1' in '/app/AlphaRTC'
0> Downloading src/buildtools/linux64/clang-format@1baf0089e895c989a311b6a38ed94d0e8be4c0a7...
Downloading 1 files took 22.099044 second(s)
Hook 'download_from_google_storage --no_resume '--platform=linux*' --no_auth --bucket chromium-clang-format -s src/buildtools/linux64/clang-format.sha1' took 22.49 secs
________ running 'download_from_google_storage --no_resume --extract --no_auth --bucket chromium-fonts -s src/third_party/test_fonts/test_fonts.tar.gz.sha1' in '/app/AlphaRTC'
0> Downloading src/third_party/test_fonts/test_fonts.tar.gz@a22de844e32a3f720d219e3911c3da3478039f89...
0> Extracting 30 entries from src/third_party/test_fonts/test_fonts.tar.gz to src/third_party/test_fonts/test_fonts
Downloading 1 files took 8.430969 second(s)

After the last line, the make process gets stuck. Is there any hint about why this happens? Thanks a lot.

Hi, it's a real issue and we are trying to dig out it ASAP.

If you just want to join our challenge, I think you don't need to compile it by yourself and you can fetch the precompiled AlphaRTC by docker pull opennetlab.azurecr.io/alphartc.

Azson commented 3 years ago

Hello , sorry for too late response. If you want to use ONNX model, we assume you will have a pre-compiled shared library and put it in https://github.com/OpenNetLab/AlphaRTC/tree/main/modules/third_party/onnxinfer/lib. But we will not release the source codes of shared libraries due to some company policy. But right now, we also provide a python interface for AlphaRTC to control the bandwidth (https://github.com/OpenNetLab/AlphaRTC#pyinfer), It will be easier to developer. Please try it. And here is an example about this interface: https://github.com/OpenNetLab/Challenge-Example

Thanks for your kind reply. But recently I encountered another problem. When trying to build AlphaRTC using the command make all, the command will get stuck at gclient sync in docker when running hooks. Part of the output message is as follow:

________ running 'vpython src/build/landmines.py --landmine-scripts src/tools_webrtc/get_landmines.py --src-dir src' in '/app/AlphaRTC'
/home/onl/.vpython-root/fe8f2e/bin/python: can't open file 'src/tools_webrtc/get_landmines.py': [Errno 2] No such file or directory

Hook 'vpython src/build/landmines.py --landmine-scripts src/tools_webrtc/get_landmines.py --src-dir src' took 10.53 secs
________ running 'vpython src/third_party/depot_tools/update_depot_tools_toggle.py --disable' in '/app/AlphaRTC'
________ running 'vpython src/build/linux/sysroot_scripts/install-sysroot.py --arch=i386' in '/app/AlphaRTC'
Installing Debian sid i386 root image: /app/AlphaRTC/src/build/linux/debian_sid_i386-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/36ed2ee83109acdd44af2e522a5e9e0082b074a6/debian_sid_i386_sysroot.tar.xz
________ running 'vpython src/build/linux/sysroot_scripts/install-sysroot.py --arch=amd64' in '/app/AlphaRTC'
Installing Debian sid amd64 root image: /app/AlphaRTC/src/build/linux/debian_sid_amd64-sysroot
Downloading https://commondatastorage.googleapis.com/chrome-linux-sysroot/toolchain/52cf2961a3cddc0d46e1a2f7d9bf376fc16a61de/debian_sid_amd64_sysroot.tar.xz
________ running 'vpython src/tools/clang/scripts/update.py' in '/app/AlphaRTC'
Downloading https://commondatastorage.googleapis.com/chromium-browser-clang/Linux_x64/clang-n353803-99ac9ce7-1.tgz .......... Done.
________ running 'vpython src/build/util/lastchange.py -o src/build/util/LASTCHANGE' in '/app/AlphaRTC'
________ running 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s src/buildtools/linux64/clang-format.sha1' in '/app/AlphaRTC'
0> Downloading src/buildtools/linux64/clang-format@1baf0089e895c989a311b6a38ed94d0e8be4c0a7...
Downloading 1 files took 22.099044 second(s)
Hook 'download_from_google_storage --no_resume '--platform=linux*' --no_auth --bucket chromium-clang-format -s src/buildtools/linux64/clang-format.sha1' took 22.49 secs
________ running 'download_from_google_storage --no_resume --extract --no_auth --bucket chromium-fonts -s src/third_party/test_fonts/test_fonts.tar.gz.sha1' in '/app/AlphaRTC'
0> Downloading src/third_party/test_fonts/test_fonts.tar.gz@a22de844e32a3f720d219e3911c3da3478039f89...
0> Extracting 30 entries from src/third_party/test_fonts/test_fonts.tar.gz to src/third_party/test_fonts/test_fonts
Downloading 1 files took 8.430969 second(s)

After the last line, the make process gets stuck. Is there any hint about why this happens? Thanks a lot.

Hello, AerysNan. We have do some update in https://github.com/OpenNetLab/AlphaRTC/pull/80. And you can update your repo and try again it.