This sample is part of the larger Omniverse Embedded Web Viewer Example. The sample demonstrates how a front end client can present a streamed Omniverse Kit application and how to send messages back and forth between the two apps.
This application is designed to be used with the USD Viewer Sample
in https://github.com/NVIDIA-Omniverse/kit-app-template.
The messages sent and handled by this sample expects the USD Viewer Sample. However, with some editing it can also be
used to stream any other Kit application as well.
This is a React application that has been build using the Vite framework (https://github.com/vitejs).
USD Viewer Template
based application created in kit-app-template using Kit 106.1.0 or more recent. Section Viewport Only - a Simpler UI describes how to use other application.This section describes how to run the solution in dev mode on a local workstation.
Make sure the prerequisites are fulfilled.
Launch a streaming Kit application based on the USD Viewer Template version 106.1
or more recent.
NOTE: If you use a modified version of a USD Viewer Template based application - or a different Template all together - you need to refer to Viewport Only - a Simpler UI to reveal the stream. The default state of this sample has code that depends on USD Viewer to be streamed.
git clone https://github.com/NVIDIA-Omniverse/web-viewer-sample.git
NOTE: Due to path length limitations on Windows it is recommended to place repository artifacts in a location closer to the root of the drive. This will help avoid issues with the path lengths when building and packaging applications. If long paths cannot be avoided then executing
git config --global core.longpaths true
prior to cloning might resolve the issue.
cd web-viewer-sample
npm install
npm run dev
At this point you should see the Kit application streamed into the web page.
The streamed RTX viewport within this client is interactable:
The USD Asset
selector tells the streamed application which OpenUSD asset to load.
The USD Stage
presents the contents of the OpenUSD asset.
The Web Viewer Sample is configured by default to connect to the USD Viewer application template and includes web UI elements for sending messages to a running Kit application. This is necessary for the USD Viewer template, which in the default use case requires a client to send a request to open a file. Use the ViewportOnly component for other use cases such as when streaming USD Viewer loading a file on startup or for completely different Kit applications.
To switch to a Viewport-only interface simply edit the import statement in ./src/App.tsx:
From:
import Window from './Window';
To:
import Window from './ViewportOnly';
As an exercise, create an application from a different template (such as USD Composer
) in the kit-app-template repository.
Run the streaming version of that application and this client.
NOTE: Without any
custom messages
being sent you can interact with the application within the client interface as if you were using the actual application.
If this sample ticks all the boxes for what you want to develop then you are off to a good
start. Simply make it your own and continue developing in it. However, it’s likely that you
have an existing client that you want to embed the viewer into.
In the section Embed Viewer in an Existing Client
we provide instructions for how to add the omniverse-webrtc-streaming-library
as a dependency.
The library will provide pixel streaming and messaging to your client. The remainder of the topics
here are presented in context of this sample so that it’s easy to follow along and try things out.
Use this sample as a reference for implementing in your own client.
To embed the viewer in your existing client you’ll need to add the
omniverse-webrtc-streaming-library
as a dependency to your project:
dependencies
section.The most important part of this sample is the ./src/AppStream.tsx file and its
use of the AppStreamer
class imported from the omniverse-webrtc-streaming-library
.
The AppStreamer
is used for any implementation and AppStream.tsx
is a reference implementation
for initializing the stream and providing bi-directional messaging between the front end client
and the Kit application.
The video HTML element presenting the streamed application reacts to keyboard and mouse
events in order to support camera movement, selection, and other 3D viewport interactions.
The tabIndex
of the <div>
wrapping the video element needs to have the tabIndex set to
tabIndex={0}
as seen in ./src/AppStream.tsx. One - yes one - other interactive elements
in the client also needs this tabIndex setting such as the selectable items in
./src/USDStage.tsx. These tabIndex setting allows each element to receive
the right focus and interactivity. Without it you may find that only the viewport reacts to
keyboard events.
The AppStreamer
’s setup()
function initializes the streaming and messaging. Here you provide a
streamConfig
object with configuration settings and a set of functions to handle messages.
This sample provides configuration via the stream.config.json file.
The default source
is set to local
which is the setting to use unless you are embedding a
stream from GDN.
For GDN you need to contact your NVIDIA representative and get the appropriate configuration details.
For local
configuration in stream.config.json you change the server
to the ip address where the Kit application is streamed from. With the ip address set you can see how
AppStream.tsx constructs a streamConfig
object providing a stream resolution and framerate:
const server = this.props.streamConfig.server;
const width = 1920;
const height = 1080;
const fps = 60;
const url = `server=${server}&resolution=${width}:${height}&fps=${fps}&mic=0&cursor=free&autolaunch=true`;
streamConfig = {
source: 'local',
videoElementId: 'remote-video',
audioElementId: 'remote-audio',
messageElementId: 'message-display',
urlLocation: { search: url }
};
NOTE: The "stream resolution” is the size of the Kit application. It is not the resolution in the Kit application viewport. If you want to add the ability to change the viewport resolution during an active session you could send a custom message from the front end client to request the change.
NOTE: The
gfn-client-sdk.js
script source in index.html can be removed when usinglocal
stream source. That library reference is only needed when streaming from GDN/GFN.
There are two critical things to recognize when working with AppStreamer and custom messages:
AppStreamer.sendMessage()
lets you send a custom message.AppStreamer.setup()
function lets you register a handler for incoming messages via onCustomEvent
:AppStreamer.setup({
streamConfig: streamConfig,
onUpdate: (message) => this._onUpdate(message),
onStart: (message) => this._onStart(message),
onCustomEvent: (message) => this._onCustomEvent(message)
})
All custom messages exchanged between the front end client and the streamed Omniverse Kit application follows the same format:
an object with properties event_type
and payload
that is JSON stringified prior to being sent off.
{
event_type: "myEvent",
payload: {
property_name : value
}
}
On the receiving end, the Omniverse Kit application will need an Extension that handles myEvent
and it's payload
. The Kit
application sends similar messages for this client to handle. Below we explore how messages are used in this solution for
opening a USD stage,
Messages sent by AppStreamer
are strings. To make a message usable by your custom Kit Extension based on
omni.kit.livestream.messaging
usage you’ll need to comply with the message format stated above.
A practical and easy to read approach to do this is to first create an object:
const message = {
event_type: 'changeResolutionRequest',
payload: {
width: 2048,
height: 1152
}
}
Then use json to stringify the message object and ask AppStreamer to send it:
AppStreamer.sendMessage(JSON.stringify(message));
This sample's Window.tsx shows many examples of sending messages.
The function registered for custom events with AppStreamer.setup()
should expect the same message object
structure used to send messages.
private _myCustomMessageHander (event: any): void {
if (!event) {
return;
}
if (event.event_type === 'changeResolutionConfirmation') {
console.log('Resolution was changed in Kit app: ' + event.payload.resolution);
}
}
This sample's Window.tsx has a _handleCustomEvent
that shows many examples of handling messages.
The below function from Windows.tsx provides an example of sending a message to the streamed
Omniverse Kit application. The client sends a openStageRequest
with a url
property in the payload
.
_openSelectedAsset = () => {
...
const message: AppStreamMessageType = {
event_type: "openStageRequest",
payload: {
url: this.state.selectedUSDAsset.url
}
};
AppStream.sendMessage(JSON.stringify(message));
}
The Kit application has a handler for openStageRequest
that opens the USD asset with the provided url
. Once that
asset has loaded the Kit application sends a openedStageResult
which is handled by the client as shown below.
The openedStageResult
handler to the request to open the asset can found in src/Windows.tsx
as well:
_handleCustomEvent = (event: any) => {
...
// Streamed app notification of asset loaded.
else if (event.event_type === "openedStageResult") {
if (event.payload.result === "success") {
console.log('Kit App communicates an asset was loaded: ' + event.payload.url);
this._getChildren(null); // Hide progress indicator
} else {
console.error('Kit App communicates there was an error loading: ' + event.payload.url);
this._toggleLoadingState(false); // Hide progress indicator
}
}
...
}
The event_type
is used by both applications to triage how to handle a message. The payload
is the data of the message.
This payload
can contain whatever data is desired.
The above is a custom capability of this example solution. Developers should decide on what messages and payloads to implement for their solutions.
The omniverse-webrtc-streaming-library
is updated over time. To get the most recent version:
./node_modules
directory if it exists.npm install
Things don't always turn out as expected. If you are not getting the expected results use the below steps to troubleshoot.
npm run dev
Kit app streaming is designed to start, run, and end; however, while developing the solution you may want to refresh the client in a browser and thus interrupt the normal flow.
Restarting the client could cause errors on the Kit application side. If the stream does not immediately re-appear just wait a few seconds. If it still does not appear try refreshing the browser one more time. If this doesn't work you may need to restart the Kit application and client.
Development using the Omniverse Kit SDK is subject to the licensing terms detailed here.
We provide this source code as-is and are currently not accepting outside contributions.