The vuforia-spatial-toolbox-userinterface contains the large majority of the code for rendering the user interface and for client-user interactions. It is a web application that is built to run in a WebView in the vuforia-spatial-toolbox-ios application, which provides device-specific APIs such as AR tracking. All content rendering, user interactions, and most communication with Vuforia Spatial Edge Servers takes place in the userinterface repository.
The Vuforia Spatial Toolbox and Vuforia Spatial Edge Server make up a shared research platform for exploring spatial computing as a community. This research platform is not an out of the box production-ready enterprise solution. Please read the MPL 2.0 license before use.
Join the conversations in our discourse forum if you have questions, ideas want to collaborate or just say hi.
Installation instructions for iOS can be found in the Vuforia Toolbox iOS repository.
Installation instructions for Android, when available, will be found in the Vuforia Toolbox Android repository.
If you just want to compile and run the app as it is currently implemented, that is all you need to know about the userinterface.
If, however, you want to understand and contribute to the codebase, the following sections will give you an overview of what you need to know.
Beyond this README, please refer to our Spatial Toolbox Forum for additional questions and answers.
index.html
is the entry-point for this codebase, and mostly consists of including all the
relevant scripts.
The src
directory contains all of the JavaScript files defining the application behavior.
It contains the following tree:
addons
app
device
gui
ar
crafting
memory
settings
network
Outside of the src
directory, we also have directories for additional resources including: css
styles, svg
and png
images (for menu and button icons, etc), nodes
(the HTML for how to
render each node type), and thirdPartyCode
, which contains additional libraries used by the
userinterface.
The userinterface has an addon system where it can load additional content_scripts from the addon packages installed on the local vuforia-spatial-edge-server running within the app, such as the Core Addon, whose README contains an example of how to structure such a script.
You will have to recompile the app to update the userinterface with its new content_scripts.
"Plugin" features that are optional to the core functionality of the app are best constructed as addons. Core services that other modules depend on should instead be included directly in this repository.
To understand how the userinterface works, you should start with a handful of core concepts and entry-points into the codebase:
objects
which contains the entire set of recognizable
objects (loaded from any vuforia-spatial-edge-servers in the local network) and their associated
AR content. By default, this will contain one "world object" where all of your AR content will
be placed if you don't connect to any external edge servers.objects
data structure is composed hierarchically of Objects
, Frames
(also known as
Tools
), and Nodes
. Refer to additional documentation about the role each of these entities
plays in the sytem.src/addons/index.js
function named onInit
can be accessed at realityEditor.addons.onInit
,
and and the src/app/targetDownloader.js
function named onTargetFileDownloaded
can be
accessed with realityEditor.app.targetDownloader.onTargetFileDownloaded
.src/gui/ar/draw.js
file contains the main update
function, which iterates over the
current set of visibleObjects
detected by Vuforia every frame, and renders all AR content
relative to the objects. The visibleObjects
will always contain the set of objects Vuforia
can see right now, as a set of their unique IDs (referred to as a uuid
or objectKey
)
mapped to a 4x4 transformation matrix (its model matrix relative to some origin point in
the world).src/gui/ar/utilities.js
for efficiently manipulating
these matrices.realityEditor.app.callbacks.receiveCameraMatricesFromAR
function, which uses Vuforia's
PositionalDeviceTracker to maintain knowledge of the phone's position in space.src/device/onLoad.js
file contains the onload
function which initializes the
application, including initializing any core or addon modules, building out the menus and 2D UI
elements, and asking the native app to initialize the Vuforia Engine.receivedUDPMessage
function in src/app/callbacks.js
.
When edge-servers periodically broadcast heartbeat messages for each object they contain, the
userinterface will call realityEditor.network.addHeartbeatObject
to download all data from
that object, add its target data to the Vuforia tracker, and add it to the global objects
.src/device/index.js
file contains most of the touch event listeners for interacting with
AR content, such as moving frames around and drawing links between programming nodes.src/gui/pocket.js
will load the set of frames (tools) from all the
connected edge servers (including the aforementioned local edge server). Thus, to create a new
type of frame (tool) that you can drop into the world, you should create an addon with a tools
directory rather than edit the userinterface.If you are debugging the userinterface and need to inspect the HTML Elements, you may want to look at these:
<div id="GUI">
<div id="object+[frameId]">
is a container for the entire frame, which has a CSS matrix3d
transformation applied to position it in 3D space.
<iframe id="iframe+[frameId]">
is the iframe containing the frame's index.html web
content.<div id="[frameId]">
is an invisible element covering the frame, which catches all the
touch events.<svg id="svg+[frameId]">
is a deprecated element that can provide visual feedback
when this frame is selected.<div id="craftingBoard">
contains the entire interface for the grid-based programming mode
, referred to in the code as "logic crafting".<div id="UIButtons">
contains the sidebar menu buttons and all of their variations.<div class="pocket">
contains the pocket screen that holds the palette of available frames
that can be dropped into the world.<canvas id="canvas">
is a fullscreen canvas behind the AR content where the links are
rendered onto in pseudo-3D.<div id="overlay">
is a small circular icon that gives visual feedback when you touch the
screen.<iframe id="settingsIframe">
contains the settings menu and all of its sub-menus.You should learn several key vocabulary terms, which correspond to classes or other data structures that compose the system's data model.
In-depth explanations of each of these concepts are available in the Data Model documentation.
Most code is documented using JSDoc comments, which can be compiled into human-readable pages of documentation using documentation.js.
You should first install the documentation.js node package:
npm install -g documentation
Then you can generate up-to-date documentation of each file by running the included
generate_docs
script:
bash ./generate_docs.sh
This will generate a docs
directory with a hierarchical set of folders mirroring the structure
of the src
directory. Open each index.html
to view the documented members of each module.
The userinterface source code may optionally be bundled using npm run build
.
This will create a subdirectory dist
with the compiled, optimized files. This
subdirectory may be used anywhere the normal root of the userinterface would
have gone previously. For example, the remote operator hardware interface can
be configured to point to <path to userinterface>/dist
for a significant
speedup once the code is compiled using npm run build
.