Godot Android Streaming Texture (GAST)

The GAST library leverages Godot's support for OpenGL external textures to enable rendering and interaction with Android views in a Godot project's scenegraph.

License

The library is released under the MIT license.

Godot XrApp Framework

The Godot XrApp framework leverages the GAST library to turn regular Android apps into Quest VR apps.

Godot XrApp Framework

GAST Plugins

These are Godot Android plugins that leverage the GAST library and the XrApp framework to support access and use of Android views within a Godot project.

Note: They are only supported starting with Godot version 3.2.2.

Contributions

The project is in alpha state (base functionality is available but API is subject to change). Contributions are welcome!

Setup

Clone the repository and initialize the submodules with git submodule update --init --recursive.

Note: When you've pulled a newer version, make sure to run git submodule update --init --recursive.

Godot CPP library

The git submodule update --init --recursive command should have checked out the godot-cpp repo under the core/libs/godot-cpp directory.
Navigate to the core/libs/godot-cpp directory and follow these steps:
- Generate the godot-cpp bindings and static libraries with the following commands:
- scons platform=android generate_bindings=yes android_arch=arm64v8 target=release ANDROID_NDK_ROOT="/PATH-TO-ANDROID-NDK/"
- scons platform=android generate_bindings=yes android_arch=arm64v8 target=debug ANDROID_NDK_ROOT="/PATH-TO-ANDROID-NDK/"

Note: If you run into issues, check out the repo setup instructions.

IDE

The project uses Android Studio for development. Make sure you have Android Studio version 4.0 or higher and open the project within Android Studio.

Architecture

The GAST plugin is written in C++ and Kotlin.

The Kotlin layer leverages the Godot Android Plugin framework to integrate with a Godot Android application. It provides the public APIs for Android clients to access the plugin core functionality in order to render, interact with and manipulate content generated by the Android View system in Godot’s scenegrah.

The core functionality is written in C++. It leverages the Godot GDNative C++ bindings in order to access and use the GDNative API. Through that API, it’s able to access at runtime the project scenegraph and manipulate it by adding, updating and/or removing Godot nodes and/or resources.

GastNode

This is the element in the scenegraph responsible for rendering and supporting interaction with an Android view.

Structure

GastNode structure

A GastNode is made of the following Godot nodes and resources:

StaticBody node as the root node, which allows the GastNode to detect collision events (if enabled by the user).
A StaticBody node requires a Shape resource to define the object’s collision bounds. For the GastNode, we’re using a CollisionShape node who's shape is set whenever the child's mesh is updated.
Since our GastNode is ultimately a mesh in 3D space, we make use of Godot’s MeshInstance node to render the GastNode.

Once a GastNode is created, the client gains the ability to retrieve a Surface instance via the GastNode#bindSurface() API. The Surface instance can be used as an output destination onto which Android Views, images or videos can be rendered. This is the process used by the GAST-Video and GAST-WebView plugins.

Input Handling

Using the GastInputListener interface via the GastManager#registerGastInputListener(...) and GastManager#unregisterGastInputListener(...) methods, the GastManager instance relays interaction events from the generated node(s) to the client. This provides the client with enough information to generate and feed Android MotionEvent events to the Android view.

Two types of events are supported:

Input Events

The plugin uses the Godot Input API to subscribe to dispatched input action events (e.g: button press). It listens for the events registered by the client via the GastInputListener#getInputActionsToMonitor() method, and notifies the client accordingly via the GastInputListener#onMainInputAction(...) callback.

Collision Events

Godot’s Physics API is used in order to detect collision events between the GastNode and a ray cast from the input pointer (in VR the input pointer consists of a tracked controller/hand). This is used to provide support for detecting click, hover and scroll events targeted at the GastNode.

These events are dispatched to the GDScript code via the following signals:

hover_input_event for hover events
press_input_event for click press events
release_input_event for click release events
scroll_input_event for scroll events

Similarly, the Android code can listen to these events using the corresponding methods:

GastInputListener#onMainInputHover(...) for hover events
GastInputListener#onMainInputPress(...) for click press events
GastInputListener#onMainInputRelease(...) for click release events
GastInputListener#onMainInputScroll(...) for scroll events

The dispatched events include the nodepath of the targeted Gast node, the node name of the colliding ray cast and information specific to the type of the event (e.g: hover location for a hover event).

Example code:

func _ready():
    ...
    var gast_loader = load("res://godot/plugin/v1/gast/GastLoader.gdns").new()
    gast_loader.initialize()
    gast_loader.connect("hover_input_event", self, "_on_gast_hover_input_event")
    gast_loader.connect("press_input_event", self, "_on_gast_press_input_event")
    gast_loader.connect("release_input_event", self, "_on_gast_release_input_event")
    gast_loader.connect("scroll_input_event", self, "_on_gast_scroll_input_event")
    ...

func _on_gast_hover_input_event(node_path: String, event_origin_id: String, x_percent: float, y_percent: float):
    pass

func _on_gast_press_input_event(node_path: String, event_origin_id: String, x_percent: float, y_percent: float):
    pass

func _on_gast_release_input_event(node_path: String, event_origin_id: String, x_percent: float, y_percent: float):
    pass

func _on_gast_scroll_input_event(node_path: String, event_origin_id: String, x_percent: float, y_percent: float, horizontal_delta: float, vertical_delta: float):
    pass

Collision Events Setup

A couple of requirements must be followed for the collision events handling to be properly set up.

The colliding raycast must belong to the gast_ray_caster node group. This allows to filter unwanted raycast collisions.
Since click and scroll events rely on additional input events (e.g: button press, joystick tilt), the plugin monitors a set of input action events with the characteristics listed below. It’s the responsibility of the client to declare and register the corresponding input actions in the Godot project.
- The monitored input action prefix is generated by using the colliding raycast's node name.
- There is one input action per raycast for click events. It’s generated by appending the “_click” suffix to the input action prefix generated as described above.
  - E.g: click action event for a RayCast node with name LeftRayCast would be LeftRayCast_click
- There are two input actions per raycast for horizontal scroll events. They are generated by appending “_left_scroll” for the left scroll events and “_right_scroll” for the right scroll events.
  - E.g: For a RayCast node with name LeftRayCast
  - Left scroll => LeftRayCast_left_scroll
  - Right scroll => LeftRayCast_right_scroll
- There are two input actions per raycast for vertical scroll events. They are generated by appending “_up_scroll” for the up scroll events and “_down_scroll” for the down scroll events.
  - E.g: For a RayCast node with name LeftRayCast
  - Up scroll => LeftRayCast_up_scroll
  - Down scroll => LeftRayCast_down_scroll

Performance Considerations

The GAST plugin attempts to recycle GastNodes as much as possible, only creating new ones if the pool is empty or exhausted. This allows to keep a lid on the number of generated OpenGL external textures.

m4gr3d / GAST

readme