itwin-unreal-plugin

Introduction

As part of Bentley's commitment to keep evolving its solutions alongside the increasing complexity of the infrastructure industry, Bentley's cloud-based iTwin platform serves the central hub for bringing all elements of Digital Twins together in a collaborative, scalable environment which allows insights into all available types of data and planning stages of Digital Twins, including IoT, change tracking, maintenance and issue anticipation and much more. Together, all this data is known as an iTwin. Bentley's applications directly plug into the iTwin cloud platform, allowing all stakeholders to consume the Digital Twin from a single point of truth.

The iTwin for Unreal SDK enables streaming of iTwins and reality data from the iTwin cloud into Unreal Engine for high-fidelity visualization and consumption. Additionally, the SDK allows Unreal Engine developers to create custom applications which can expand upon the capabilities provided by the SDK out of the box. This enables the creation of tailored interactive experiences around iTwins. In addition, we provide a pre-compiled plugin version of the SDK.
The streaming technology is based on the open-source Cesium 3D tiles (link to the Cesium github) and offers great performance even with large datasets. The SDK is based on the experience gathered through the technology previews introduced by the Datasmith Connector and the 3DFT plugin during the last year.

The iTwin for Unreal SDK will supersede these technology demos and become the foundation of our upcoming advanced visualization products. It embodies Bentley's long-term commitment to offering scalable and future-proven technologies and open standards for any visualization needs of infrastructure and Digital Twins.
This SDK will be regularly updated. We appreciate your feedback to turn this exciting new technology into the leading foundation for advanced visualization of Digital Twins leveraging game engine technology. We encourage you to participate in the SDK's development with your ideas, requests and wishes, and help us shape the future of visualization together.

Supported features

• Real-time 3D rendering, navigation and visualization of iModels with high performance; other iTwin related data will be added in future updates.
• Accessing saved views
• Exposed API for custom blueprints (for loading reality data, for example)

Please note that 4D animation is not part of this initial build, but it is one of our top priorities and will be added as soon as possible.

Ability to load external tilesets

It is possible to load "external" tilesets (ie. non-iModels, like tileset coming from Cesium Ion server) by following these steps:

• Drag and drop an ITwinCesium3DTileset actor from the Content Browser into your level.
  This actor can be found in the content browser inside folder Plugins/iTwin for Unreal C++ Classes/ITwinCesiumRuntime/Public.
  
• In the actor's Details panel, in the Cesium section, fill the Source, Url, Ion Asset ID, Ion Access token fields depending on the source of your tileset.

Supported platforms & requirements

To use the iTwin plugin, you need to create an account at developer.bentley.com. There is a free 90 day trial available. The developer account is required for utilizing the streaming API within the plugin. If your trial period is over and you would like to continue using and testing the plugin, please get in touch with us.
This initial release supports Windows 11; Windows 10 might work, but has not been tested officially (you may conduct tests on Windows 10 yourself if you would like). A version for Mac is in development.
Only iModels and their Saved Views are supported at the moment; Reality Data can be loaded through blueprints only (no GUI is available in the ITwinTestApp for them at the moment).
Unreal Engine 5.3 is the currently supported version. Other Unreal Engine versions will be supported in future updates.

To run Unreal Engine, make sure you are using a dedicated GPU. The performance largely depends on the power of your graphics card. For more information on recommended system specs for Unreal Engine, please visit Epic's website.

We recorded a quick start video on YouTube. Read on for the written steps of the installation process.

1. Installing the precompiled iTwin plugin for Unreal

1. Go to the Releases page.
2. Download ITwinForUnreal.zip from the latest release.
3. Extract the zip archive to Unreal Engine's Engine/Plugins/Marketplace folder.
   For example, on Unreal Engine 5.3 on Windows, this is typically C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\Marketplace.
   You may need to create the Marketplace folder yourself.
   After extraction, you should have a folder Engine/Plugins/Marketplace/ITwinForUnreal/Binaries for example.
4. The plugin is now ready to use.

If you do not want to install the plugin in the Unreal Engine folder and instead prefer to put it directly in your Unreal project, just extract the zip archive inside your app's Plugins folder (you may need to create this folder).
For example: C:\MyUnrealApp\Plugins\ITwinForUnreal.

2. Using the installed iTwin plugin inside Unreal Engine

In a new project

1. If you installed the plugin in the Unreal Engine folder, enable the plugin iTwin for Unreal as explained here.
2. If not done yet, configure your iTwin Platform account and take note of your app's client ID.
3. Make sure the plugin and its content is visible in the content browser: in the "Settings" menu of the content browser window, enable "Show Engine Content" and "Show Plugin Content".
4. To access your iModels from your Unreal app, you will have to provide the iTwin app ID to the plugin.
   This can be done in several ways:
   • Drag and drop an ITwinAppIdHelper actor from the Content Browser into your level.
     This actor can be found in the content browser inside folder Plugins/iTwin for Unreal C++ Classes/ITwinRuntime/Public.
     Then, in the actor's Details panel, paste your iTwin app ID inside the field ITwin/App Id and validate.
     Now, the plugin will use this app ID whenever this level is loaded.
     This method is useful if you want to simply add an iModel manually into your level inside the Unreal Editor (see below).
   • If you use C++ code or Blueprint, you can instead directly call the static function AITwinServerConnection::SetITwinAppID().
     This is typically done in your app's module StartupModule() function (if using C++), or in the BeginPlay event of your Game Mode (if using Blueprints).
5. To manually add an iModel into your level inside the Unreal Editor, drag and drop an ITwinIModel actor from the Content Browser into your level.
   This actor can be found in the content browser inside folder Plugins/iTwin for Unreal C++ Classes/ITwinRuntime/Public.
   Then, in the actor's Details panel, go to the Loading section and fill in these fields:
   • Loading Method: Automatic
   • iModel Id: the ID of your iModel
   • Changeset Id: the ID of the changeset you want to import Then the iModel should appear in the viewport. If the selected iModel/changeset has never been imported yet, the iTwin server needs to convert (ie. export) it into the Cesium format.
     In such case the export will be automatically started, and the "Export Status" label will say "In progress" until the export is complete.
     This can take a long time, depending on the complexity of your iModel. Once the export is complete, the iModel will appear in the viewport.

In an existing project which uses the 3DFT plugin

1. Make sure your project is using Unreal Engine version 5.3.
   See "Change a Project's Unreal Engine Version" here.
2. Remove the dependency to the 3DFT plugin. This is done by following these steps:
   • If the 3DFT plugin folder is located inside your project:
     1. Make sure your project is not open in Unreal Editor.
     2. Remove the folder Plugins\iTwin from your project.
   • If the 3DFT plugin folder is located inside in the Unreal Engine folder:
     In the Unreal Editor, disable the iTwin plugin as explained here.
3. Make sure your project is not open in the Unreal Editor.
4. If not done yet, install the ITwinForUnreal plugin.
5. Remove the folders Binaries and Intermediate that may exist at the root of your project's folder.
6. If your project contains C++ code, these additional steps are needed:
   • Enable C++20 support; this can be done by adding/updating this line in your *.Target.cs files:
     DefaultBuildSettings = BuildSettingsVersion.V4;
   • Open your .uproject file in a text editor, and remove any dependency on module iTwin inside the Modules/AdditionalDependencies section.
   • Some classes have been renamed (eg. AiModel -> AITwinIModel), you may need to update your code accordingly.
     Please refer to file ITwinForUnreal/Config/BaseITwinForUnreal.ini for the list of changes.
7. Modify your app so that it provides the iTwin app ID to the plugin, as explained in the "using the plugin in a new project" section.

Now, if you open your .uproject in the Unreal Editor, it should (build and) run without any error.

3. Configure access to the iTwin Platform

Create and configure your iTwin Platform account:

1. Go to the iTwin Platform developer portal and create an account.
   
2. Go to My Models and create a new iModel.
   
3. Go to My Apps and register a new iTwin App:
   • Application type: Native
   • Redirect URIs: http://localhost:3000/signin-callback

4. Developing with Unreal Engine: Compiling the plugin yourself

Prerequisites

• CMake 3.28 or newer
• Python 3.9 or newer
• Visual Studio 2022

Configure your machine for development

Developer Mode for Windows must be enabled, as explained here.

Configure the iTwin app ID

1. If not done yet, configure your iTwin Platform account and take note of your app's client ID.
2. Configure your iTwin app ID in the plugin: File: "\Public\CMake\main.cmake"
   set (ITWIN_TEST_APP_ID "your_client_id_goes_here")
   

Build and run on windows

1. Clone the vcpkg repo somewhere on your machine.
   git clone https://github.com/microsoft/vcpkg.git
2. Create an environment variable VCPKG_ROOT pointing to your local vcpkg repo. You can also set the VCPKG_ROOT as a cmake variable.
3. Clone the repo and its submodules.
   git clone https://github.com/iTwin/itwin-unreal-plugin.git
cd itwin-unreal-plugin
git submodule update --init --recursive
4. Run CMake (command-line or GUI) and generate your build folder.
   You need to select CMake preset "win64Unreal".
   Command-line example, supposing current directory is your repo root:
   cmake -S . -B C:\Build\itwin-unreal-plugin --preset win64Unreal
5. Open ITwinTestApp.sln (in VS 2022) located inside the generated build directory.
6. Hit F5 to build and run the test app (UnrealProjects/ITwinTestApp in the solution explorer).
7. Once compilation is finished and VS has launched UnrealEditor, hit alt+P (play in editor) to actually start the app.

Build and run on MAC ARM

1. Clone the vcpkg repo somewhere on your machine.
   git clone https://github.com/microsoft/vcpkg.git
2. Create an environment variable VCPKG_ROOT pointing to your local vcpkg repo. You can also set the VCPKG_ROOT as a cmake variable.
3. Clone the repo and its submodules.
   git clone https://github.com/iTwin/itwin-unreal-plugin.git
cd itwin-unreal-plugin
git submodule update --init --recursive
   
4. Run CMake (command-line or GUI) and generate your build folder.
   You need to select CMake preset "macosXcodeARM".
   You also have to ensure that clang will compile for arm and not x86.
   Command-line example, supposing current directory is your repo root:
   cmake -S . -B ../Build/itwin-unreal-plugin --preset macosXcodeARM -DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64
5. Build the targets InstallCesiumDependencies and then ITwinTestApp.
   Some of the targets use xcodebuild that does not handle well multithreading, we recommend that you use --parallel 1:
   cmake --build . --parallel 12 --target InstallCesiumDependencies
cmake --build . --parallel 1 --target ITwinTestApp
6. To launch ITwinTestApp in Unreal, use the target Run_ITwinTestApp_Editor:
   cmake --build . --parallel 1 --target Run_ITwinTestApp_Editor
7. Once compilation is finished and cmake has launched UnrealEditor, hit alt+P (play in editor) to actually start the app.

Clean Build

1. remove your cmake Build folder
2. call git clean -dfX in your source folder

Known issues and limitations

• Packaging an application that uses both the iTwin plugin and the Cesium plugin will not work (errors about duplicate c++ symbols).
  As a workaround, you can disable the Cesium plugin and add your tilesets as explained here.
• On MacOS, the synchro schedules are not yet supported.

Support

We are looking forward to your feedback and your ideas for the plugin. If you encounter any bugs, please use the Issues tab to report any bugs.

Getting 4D animations from Synchro working in the plugin

Build 0.1.10 offers preliminary support for playing back 4D animations. Since this is an early access version, there are a few steps and limitations to consider.

Current limitations related to 4D schedules:

• The macOS version of the plugin does not support the 4D schedules yet.
• Transformations and animations along 3D paths are not supported yet (even if you uncheck the checkbox in the Replay settings).
• When new geometry is received from the streaming server, typically better Level-of-Detail meshes, the animation is not immediately known, nor instantly applied to the visualization. This results in chunks of geometry appearing even though they should be invisible at the current time in the schedule, for example. The visual should correct itself after some time has passed.
• Please do not change the materials used by the AITwinCesiumTileset actor as this would most likely prevent the animation (including visibilities and cutting planes) from being applied.

Please follow along the following process:

1. To be able to access schedules data, the project must have been resynchronized recently: projects not actively used and/or synchronized before mid-July of this year (mid-August for projects hosted on East US datacenter) will definitely need to be resynchronized, either by submitting actual changes to the project, or by changing any of the synchronization settings in the Edit iModel panel for the iModel bearing the 4D schedule (actual list of available settings can vary depending on the iModel's history):
   
   
   
2. After setting up the app Id and creating the iModel actor in your Unreal level, the geometries start appearing, and the schedule data is queried for all received geometries
   This can take some time and you can see the pending queries in the "Output log" panel. Moving in the scene will stack more pending batches, when new tiles become visible or resolution changes occur. When waiting long enough, all pending queries will be handled, and a summary is visible in the Output log:
   
   
3. Select the iModel in the Outliner, and select its Synchro4DSchedules component in the Details panel: you will see the Replay settings there, where you can:
   • "Jump to beginning": jump to the start of the schedule (the default is a time far in the future in order to see the completed project)
   • "Jump to end" of schedule
   • Set the replay speed (as an FTimespan of schedule time per second of replay time), or use "Auto replay speed" to fit the whole animation in 30 seconds of replay time.
   • You can edit the "Schedule Time" field manually, to jump to a specific time point.
   • "Play"/"Pause" the schedule animation
   • "Stop" will pause the animation but also display all geometries with their original color and opacities, as if there were no schedules at all.
     
     
4. There are also advanced Replay settings to optionally disable coloring, visibilities or cutting planes during the animation, or to 'Mask out non-animated Elements' which may be confusing when looking at the animation.