search

NVIDIA-Omniverse / web-viewer-sample

This sample demonstrates how a front-end client can present a streamed Omniverse Kit application and how to send messages between the two apps.
Other
31 stars 13 forks source link

readme

Omniverse Web Viewer Sample Application

This sample is part of the larger Omniverse Embedded Web Viewer Example. The sample demonstrates how a front end client can present a streamed Omniverse Kit application and how to send messages back and forth between the two apps.

This application is designed to be used with the USD Viewer Sample in https://github.com/NVIDIA-Omniverse/kit-app-template. The messages sent and handled by this sample expects the USD Viewer Sample. However, with some editing it can also be used to stream any other Kit application as well.

This is a React application that has been build using the Vite framework (https://github.com/vitejs).

Table of Contents

Prerequisites

Quick Start

This section describes how to run the solution in dev mode on a local workstation.

  1. Make sure the prerequisites are fulfilled.

  2. Launch a streaming Kit application based on the USD Viewer Template.

  3. Clone this repository:

git clone https://github.com/NVIDIA-Omniverse/web-viewer-sample.git
  1. Navigate into the project:
cd web-viewer-sample
  1. Install dependencies:
npm install
  1. Run the client:
npm run dev
  1. Open a Chromium browser and navigate to localhost:5173.

At this point you should see options for configuring the client.

NOTE: When running this application in a container, you can serve the optimized production files generated by npm run build, rather than starting the development server. After running npm run build, the static files will be located in the dist directory.

Client Options

When running this client, you will be presented with options to configure options prior to the stream launching. These options will vary depending on the stream source defined in the stream.config.json file.

All stream sources will initally display a Include Web UI? prompt. Selecting Yes will include web UI elements for sending messages to a running Kit application. This is necessary for the USD Viewer template, which in the default use case requires a client to send a request to open a file. Select No for other use cases such as when streaming USD Viewer loading a file on startup or for completely different Kit applications.

As an exercise, create an application from a different template (such as USD Composer) in the kit-app-template repository. Run the streaming version of that application and this client.

Note that without any custom messages being sent you can interact with the application within the client interface as if you were using the actual application.

When running the client using stream as the source in the stream.config.json file, you will be presented with addtitional options for entering an application and stream server, selecting an application, application version and application profile prior to creating the streaming session.

Interacting with the solution

The streamed RTX viewport within this client is interactable:

The USD Asset selector tells the streamed application which OpenUSD asset to load.

The USD Stage presents the contents of the OpenUSD asset.

Front End Client Development

If this sample ticks all the boxes for what you want to develop then you are off to a good start. Simply make it your own and continue developing in it. However, it’s likely that you have an existing client that you want to embed the viewer into. In the section Embed Viewer in an Existing Client we provide instructions for how to add the omniverse-webrtc-streaming-library as a dependency. The library will provide pixel streaming and messaging to your client. The remainder of the topics here are presented in context of this sample so that it’s easy to follow along and try things out. Use this sample as a reference for implementing in your own client.

Embed Viewer in an Existing Client

To embed the viewer in your existing client you’ll need to add the omniverse-webrtc-streaming-library as a dependency to your project:

AppStreamer

The most important part of this sample is the ./src/AppStream.tsx file and its use of the AppStreamer class imported from the omniverse-webrtc-streaming-library. The AppStreamer is used for any implementation and AppStream.tsx is a reference implementation for initializing the stream and providing bi-directional messaging between the front end client and the Kit application.

Keyboard Event Management

The video HTML element presenting the streamed application reacts to keyboard and mouse events in order to support camera movement, selection, and other 3D viewport interactions. The tabIndex of the <div> wrapping the video element needs to have the tabIndex set to tabIndex={0} as seen in ./src/AppStream.tsx. One - yes one - other interactive elements in the client also needs this tabIndex setting such as the selectable items in ./src/USDStage.tsx. These tabIndex setting allows each element to receive the right focus and interactivity. Without it you may find that only the viewport reacts to keyboard events.

Initialize the Stream

The AppStreamer’s connect() function initializes the streaming and messaging. Here you provide a streamConfig object with configuration settings and a set of functions to handle messages.

This sample provides configuration via the stream.config.json file. There are three different source values that are supported: local, gfn and stream. The default source is set to local which is the setting to use unless you are embedding a stream from GDN. For GDN you need to contact your NVIDIA representative and get the appropriate configuration details.

For local configuration in stream.config.json you change the server to the ip address where the Kit application is streamed from. With the ip address set you can see how AppStream.tsx constructs a streamConfig object providing a stream resolution and framerate (view code example here).

For stream configuration in stream.config.json, you can enter default streamServer and appServer values, but these values are also configurable prior to creating the streaming session via the front-end when the application is launched.

It’s important to note that the "stream resolution” is the size of the Kit application. It is not the resolution in the Kit application viewport. By default, the USD Viewer template is set to change the viewport resolution to fit the size of the viewport; meaning, the resolution is adjusted as the application window is resized. If you want to add the ability to change the viewport resolution during an active session you could send a custom message from the front end client to request the change.

NOTE: The gfn-client-sdk.js script source in index.html can be removed when using local or stream sources. That library reference is only needed when streaming from GDN/GFN.

Custom Messages with AppStreamer

There are two critical things to recognize when working with AppStreamer and custom messages:

Message Format

All custom messages exchanged between the front end client and the streamed Omniverse Kit application follows the same format: an object with properties event_type and payload that is JSON stringified prior to being sent off.

{
    event_type: "myEvent",
    payload: {
        property_name  : value
    }
}

On the receiving end, the Omniverse Kit application will need an Extension that handles myEvent and it's payload. The Kit application sends similar messages for this client to handle. Below we explore how messages are used in this solution for opening a USD stage,

Send a Custom Message

Messages sent by AppStreamer are strings. To make a message usable by your custom Kit Extension based on omni.kit.livestream.messaging usage you’ll need to comply with the message format stated above. A practical and easy to read approach to do this is to first create an object:

const message = {
  event_type: 'changeResolutionRequest',
    payload: {
      width: 2048,
      height: 1152
  }
}

Then use json to stringify the message object and ask AppStreamer to send it:

AppStreamer.sendMessage(JSON.stringify(message));

This sample's Window.tsx shows many examples of sending messages.

Receive a Custom Message

The function registered for custom events with AppStreamer.connect() should expect the same message object structure used to send messages.

private _myCustomMessageHander (event: any): void {
    if (!event) {
        return;
    }
    if (event.event_type === 'changeResolutionConfirmation') {
        console.log('Resolution was changed in Kit app: ' + event.payload.resolution);
    }
}

This sample's Window.tsx has a _handleCustomEvent that shows many examples of handling messages.

Sample Message Loop

The below function from Windows.tsx provides an example of sending a message to the streamed Omniverse Kit application. The client sends a openStageRequest with a url property in the payload.

_openSelectedAsset = () => {
        ...
        const message: AppStreamMessageType = {
            event_type: "openStageRequest",
            payload: {
                url: this.state.selectedUSDAsset.url
            }
        };
        AppStream.sendMessage(JSON.stringify(message));
    }

The Kit application has a handler for openStageRequest that opens the USD asset with the provided url. Once that asset has loaded the Kit application sends a openedStageResult which is handled by the client as shown below.

The openedStageResult handler to the request to open the asset can found in src/Windows.tsx as well:

_handleCustomEvent = (event: any) => {
    ...
    // response received once a USD asset is fully loaded
    else if (event.event_type === "openedStageResult") {
        if (event.payload.result === "success") {
            this._queryLoadingState() 
            console.log('Kit App communicates an asset was loaded: ' + event.payload.url);
            this._getChildren(null); // Hide progress indicator
        } else {
            console.error('Kit App communicates there was an error loading: ' + event.payload.url);
        }
    }
    ...
}

The event_type is used by both applications to triage how to handle a message. The payload is the data of the message. This payload can contain whatever data is desired.

The above is a custom capability of this example solution. Developers should decide on what messages and payloads to implement for their solutions.

Updating Dependencies

The omniverse-webrtc-streaming-library is updated over time. To get the most recent version:

  1. Delete the ./node_modules directory if it exists.
  2. Delete the package-lock.json file.
  3. Pull dependencies:
npm install

Troubleshooting

Things don't always turn out as expected. If you are not getting the expected results use the below steps to troubleshoot.

General steps

Refreshing the Client

Kit app streaming is designed to start, run, and end; however, while developing the solution you may want to refresh the client in a browser and thus interrupt the normal flow.

Restarting the client could cause errors on the Kit application side. If the stream does not immediately re-appear just wait a few seconds. If it still does not appear try refreshing the browser one more time. If this doesn't work you may need to restart the Kit application and client.

License

Development using the Omniverse Kit SDK is subject to the licensing terms detailed here.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Contributing

We provide this source code as-is and are currently not accepting outside contributions.