Carla on ppc64le building of Unreal Engine fails

[robertengelmann]

Hi everyone,

I am trying to build Carla for the ppc64le architecture on a POWER8 machine. The machine runs RHEL 7.5 and I am trying to build in a Docker container based on an NVIDIA CUDA Ubuntu 16.04 image. So far I have basically followed the documentation for building on Linux (https://carla.readthedocs.io/en/latest/how_to_build_on_linux/), but when compiling the Unreal Engine, I get two error messages I have not been able to fix, yet: When executing Setup.sh it exits successfully, but I get:
Register the engine installation... ./Setup.sh: line 45: ./UnrealVersionSelector-Linux-Shipping: cannot execute binary file: Exec format error
Looks to me like this binary is not compiled for ppc64le, but I don't even know if it is important. When running GenerateProjectFiles.sh:
/usr/lib/mono/4.5/Microsoft.CSharp.targets: error : Compiler crashed with code: 139. Cannot open assembly './../../../Binaries/DotNET/UnrealBuildTool.exe': No such file or directory.
I have checked, the file UnrealBuildTool.exe is really not there. I suppose it should have been generated in a previous step.
I have found this pull request to the UE project https://github.com/EpicGames/UnrealEngine/pull/2585, but have not been able to successfully incorporate it.

Has anyone tried this before? Any experiences or ideas to share? Am I missing something?
Please ask, if I have left anything obscured. I have also posted this in the Unreal Engine forums: https://forums.unrealengine.com/development-discussion/engine-source-github/1611491-building-unreal-engine-on-ppc64le

Any help would be appreciated :-)

[klauspower]

i think the dll is 32bit .. (must change mono ) and dll thx klaus

[klauspower]

check: > https://www.mono-project.com/archived/installerinstructions/ Will the installer work on x86_64?

No, this installer is only built to run on 32 bit systems. Some distributions have both 32 bit and 64 bit libraries and programs installed. If a 64 bit system has all the necessary 32 bit components installed, it should work, although this is not a tested scenario.

so .. power8 i a native 64bit system .. ps. will try it on power9 .. () I don’t know that for sure yet !

[klauspower]

look .. yes .. must build an VM .. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.6_release_notes/index

[1] Note that the Red Hat Enterprise Linux 7.6 installation is supported only on 64-bit hardware. Red Hat Enterprise Linux 7.6 is able to run 32-bit operating systems, including previous versions of Red Hat Enterprise Linux, as virtual machines.

[robertengelmann]

Thank you for your input, @klauspower . When you say

i think the dll is 32bit .. (must change mono ) and dll

do you talk about ./UnrealVersionSelector-Linux-Shipping ? Are you suggesting the Unreal Engine can only be built on 32 bit systems?

[klauspower]

no but x86 can both 64 and 32bit .. power only 64-bit -- mono is only 32-bit .. to. docker .. https://forums.docker.com/t/support-for-32-bit-images-containers-on-64-bit-hosts/10010 to vm .. need kvm ! understand .. try to build () UE4 on x86 .. after make it on power8 / 9 you must understand mono .. no native .. need docker or vm and try ..
ps. will try it on power9 ( )I'll let you know when it's ready.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[edowson]

I'm porting Raptor Engineering's port of UE4.20.3-release from the kb9vqf/UnrealEngine/tree/release branch.

You can track it's progress in the edowson/UnrealEngine/tree/4.22-ppc64le branch.

Additional notes for setting up and installing Ubuntu-18.04 on a Power9 system like the Talos II Secure Workstation can be found in the edowson/raptor-talos-ii-power9-technote repo. It includes notes for setting NVIDIA Tesla GPU drivers and CUDA.

I still have to port several fixes for third party libraries to ppc64el.

In the meantime, you can prepare a Power9 host using the technotes, as well as install the following packages on your system, in order to get the a basic environment to compile the UE4.22.3 sources on ppc64le

Install common development packages:

sudo apt install --no-install-recommends -q -y \
python-dev \
python3-dev

sudo apt-get install --no-install-recommends -q -y \
autotools-dev \
autoconf \
automake \
cmake \
dos2unix \
libtool

sudo apt-get install --no-install-recommends -q -y \
libtool libc++-dev libc++abi-dev \
libglib2.0-dev libhdf5-dev libwebsockets-dev \
freeglut3-dev  libvulkan-dev wayland-protocols \
libegl1-mesa-dev libglu1-mesa-dev libgl1-mesa-dev libgles2-mesa-dev mesa-common-dev

Install required libraries:

sudo apt-get install --no-install-recommends -q -y \
libfreetype6-dev \
libomp-dev

Install X11 development libraries:

sudo apt-get install --no-install-recommends -q -y \
libx11-dev \
libxcursor-dev \
libxext-dev \
libxi-dev \
libxt-dev \
libxv-dev \
libxinerama-dev \
libxkbcommon-dev \
libxrandr-dev \
libxrender-dev \
x11-xkb-utils \
x11proto-input-dev \
x11proto-randr-dev \
x11proto-scrnsaver-dev \
x11proto-xinerama-dev \
x11-xserver-utils \
xauth \
xdg-user-dirs \
xdg-utils \
xfonts-base \
xkb-data \

Install mono development tools:

# setup sources.list
sudo apt install gnupg ca-certificates
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update

sudo apt-get install --no-install-recommends -q -y \
mono-devel \
mono-dmcs

Build clang-8.0.1 on linux by running the Engine/Build/BatchFiles/Linux/Toolchain/BuildClangOnLinux.sh script.

[qhaas]

Thanks, @edowson . I've been monitoring your work on your 4.22 ppc64le fork of ue4. Is the above your current development environment? If you have it compiling, a text file with the results of apt list --installed would be quite useful to us.

[edowson]

@qhaas The latest set of patches can be found in the edowson/UnrealEngine/commits/4.23-ppc64le branch.

[edowson]

@qhaas I've attached a text file with the output of apt list --installed on the power9 system below: 201911070600-ubuntu-18.04-apt-list-installed-ppc64le.txt

[edowson]

@qhaas The current status is that I'm down to only a few more libraries need to be added to the Engine/Build/BatchFiles/Linux/BuildThirdParty.sh script and to fix some linker errors on ppc64le.

KissFFT, Steam API, WebRTC
Opus
libStrophe
FBX: some more to be disabled

Run this script to build the ppc64le toolchain. Engine/Build/BatchFiles/Linux/Toolchain/DockerOnWindows/build_linux_toolchain/build_linux_toolchain.sh

Just modify line 27 to build just one targets to save time:

TARGETS="powerpc64le-unknown-linux-gnu"

The BuildThirdParty.sh has detection logic for the clang toolchain. An earlier set of patches had that for the Linux/Setup.sh script, which I will have to reintroduce. This way, if a user runs Setup.sh, it will check for existence of the clang-8 toolchain and trigger a build. I'll parameterize the build_linux_toolchain.sh script later, to accept an argument to over-ride the default target list, to save time during the build process.

[qhaas]

Thanks, @edowson . We were barking up the wrong branch (4.22-ppc64le), it would seem. We will focus on getting your 4.23-ppc64le branch to compile on our end. We had gotten to the point where we ran into the LLVM bug report you submitted (bug report 43072) when compiling the Epic's SDL2 fork and I backported the fix from LLVM's master branch to the 8.0.0 version that ships with Ubuntu 18.04 ppc64le. But, you might have already worked around that issue in your 4.23 branch. Thanks for the tip on using the toolchain script in the DockerOnWindows folder, we will try that.

Our end goal is to get CARLA running on a ppc64le / Tesla V100 based HPC in a RHEL7 environment, so it is doubtful we need the full UE4 feature-set. We have a Raptor Talos 2 for development purposes, and we are using containers to replicate your Ubuntu 18.04 development environment. We will likely build an Ubuntu 18.04 ppc64le runtime container on the HPC since Singularity is available.

[edowson]

@qhaas The early tests were performed using a Raptor system with a GV100 GPU. You don't necessarily have to have a GV100 for these tests, an RTX2070 GPU will also work fine with a Raptor Talos II workstation.

Send me a pm with your email, and we can sync our efforts directly.

I've containerized UE4 for x86_64 as well and did some experimental trials running a containerized version of the ue4-first-person sample from within a Kubernetes cluster. My end-goal is to support distributed reinforcement learning for autonomous agents.

[edowson]

@qhaas There is a pull request for general script fixes here: https://github.com/EpicGames/UnrealEngine/pull/6279

Hopefully, if that gets accepted into the master branch, then the delta ppc64le patches for future version 4.24 and above, will be much smaller.

[qhaas]

@edowson , since GitHub doesn't seem to have private messages anymore, I sent an email to your gmail account that was listed in the commit logs.

[qhaas]

Using @edowson 's 4.23.1-ppc64le fork mentioned above, we now have CARLA 0.9.8 running on our IBM AC922 HPC, also known as the Summit Supercomputer and our Raptor Talos 2 workstations! The latter was easier, for supercomputers have their own nuances...

Unfortunately, a few textures are missing (note fewer details of the plants on the right with ppc64le), likely because Epic doesn't have fallbacks for some x86-64 specific SIMD/library calls:

LogMeshUtilities: Error: Couldn't generate distance field for mesh, platform is missing required Vector intrinsics.

The first known image rendered with CARLA 0.9.8 on Summit (Power9+Tesla V100) using Dowson's 4.23.1-ppc64le UE fork and a slightly modified tutorial.py script: x86-64 + Kepler era GPU using Epic's official UE 4.23.1 for comparison:

We are having to run it via the UnrealEditor due to issues with make package, but these issues are occurring on Epic's 4.23.1 release on x86-64 as well, so likely an artifact of using a version of CARLA (and its assets) meant for UE 4.22 with UE 4.23.1:

LogInit: Display: LogBlueprint: Error: [Compiler BaseVehiclePawn] The current value of the ' Target Array ' pin is invalid: Array inputs (like 'Target Array') must have an input wired into them (try connecting a MakeArray node). from Source: /Game/Carla/Blueprints/Vehicles/BaseVehiclePawn.BaseVehiclePawn
LogInit: Display: LogBlueprint: Error: [Compiler BaseVehiclePawn] The current value of the ' Target Array ' pin is invalid: Array inputs (like 'Target Array') must have an input wired into them (try connecting a MakeArray node). from Source: /Game/Carla/Blueprints/Vehicles/BaseVehiclePawn.BaseVehiclePawn
LogInit: Display: LogBlueprint: Error: [Compiler BaseVehiclePawn] The current value of the ' Array ' pin is invalid: Array inputs (like 'Array') must have an input wired into them (try connecting a MakeArray node). from Source: /Game/Carla/Blueprints/Vehicles/BaseVehiclePawn.BaseVehiclePawn
LogInit: Display: LogBlueprint: Error: [Compiler BaseVehiclePawn] The current value of the ' Array ' pin is invalid: Array inputs (like 'Array') must have an input wired into them (try connecting a MakeArray node). from Source: /Game/Carla/Blueprints/Vehicles/BaseVehiclePawn.BaseVehiclePawn

@germanros1987 , you were correct about how much time it would take when we mentioned this effort to you last year on a teleconference call, ha.

UPDATE: The 'BaseVehiclePawn' issue was cleared by wiring the blueprint similar to how it is wired in CARLA 0.9.9 with UE4.24. This allowed make package to succeed on my parallel x86-64 build with clang-8 and (official) UE 4.23.1. However, the ppc64le flavor still suffers from the earlier error reported by 'LogMeshUtilities' above and refused to make package due to said errors.

[qhaas]

We now have CARLA 0.9.8 packaging in 'Shipping' mode with the unofficial ppc64le UnrealEngine 4.23.1 fork and clang-8.

The Error: Couldn't generate distance field for mesh was fixed with this commit to the fork. A separate packaging issue described in Issue #3522 was cleared in different commit to said fork.

Unfortunately, while we can package and run CARLA now on pp64le, the origin of the visual artifacts / lack of detail on the trees/shrubs depicted above remain elusive. They still occur even with said fixes. We also cannot force the issue on x86-64, even with 'low' detail settings. There are no warnings/errors to indicate what the problem is that we have discovered. We are assuming a driver bug at this point.

[qhaas]

Per a similar issuing occurring on ARM, DXTC is apparently misbehaving on ppc64le. When we disabled it (per the instructions in the guide linked), the artifacts / missing textures appear resolved.

With DXTC on ppc64le: Without DXTC on ppc64le:

This was the last of the major outstanding issues with ppc64le!