search

thomwolf / Magic-Sand

Magic-Sand is a software for operating an augmented reality sandbox
GNU General Public License v2.0
944 stars 138 forks source link

readme

Magic Sand

Magic Sand is a software for operating an augmented reality sandbox like this one:

First View of a Magic Sandbox

An augmented reality sandbox is made of a sand box, a depth detector (like a Kinect) and a beamer coupled together. The software provided on this repository control the beamer and the kinect in order to project on the sand colors which are related to the height of the sand.

This project was inspired and adapted from the Augmented Reality Sandbox developped by UC Davis. It is a partial port of the SARndbox project of Oliver Kreylos under openframeworks and is also adapted from the ofxKinectProjectorToolkit by Gene Kogan.

It was ported to openFrameworks with multi-plateform support (linux/macos/windows), a simple user interface and a robust yet simple calibration procedure by Thomas Wolf and later modified and extended with some games by Rasmus R. Paulsen.

Animation of a Magic Sandbox

Magic-Sand was developed with the specific aim of simplifying the use of an augmented reality sandbox in a home/family environment :

View of a Magic Sandbox

Main Features

Operates on a computer connected to a home cinema projector and a kinect sensor. The software controls the projector to project colors as a function of the sand level measured by the kinect sensor and transforms a sandbox in a colorful playground.

View of the interface to use the Magic Sandbox

Getting started

The easiest way to get started is to build the physical setup provided in the guide found at the tutorial page and/or check the reddit thread

Secondly, download and install/unpack the latest ready-to-use version of the software at the release page. Follow the instructions on the release page to download and install the necessary drivers.

Setting up the system

Connect and turn on the projector and the kinect and start the software.

By default the software starts in a setup mode where the depth or color image from the Kinect can be seen in the user interface and the projector projects a completely white image. This way it is easy to check if the Kinect is running (on Windows 10 machines it can be necessary to plug and unplug the Kinect several times before it starts) and if the projector is working. The status of the kinect and the projector can be seen in the status window to the lower left in the user interface.

In setup mode the physical positions of the Kinect and projector can be optimised.

Calibration

To calibrate the system so the kinect and the projector is in correspondence a few steps are needed:

If the calibration is succesful the status window should be updated showing that all is ok.

Animation of the calibration steps to use the Magic Sandbox

Debug mode for calibration

If the calibration was not succesful a debug mode can be enabled that will place debug files in the data\DebugFiles folder. These might point you in the direction of why the calibration failed. Do this by enabling advanced|Dump Debug and run the calibration routine again.

Starting the Application

If the calibration was succesful or if a calibration was done before, the application can be started by pressing space or pushing the Run button.

Now a colored map with iso-lines should appear on the sand. The framerate should be close to 60 FPS for modern PCs.

Sandbox games

There are a few games included in Magic-Sand

Shape an Island

The background for the game is that Denmark have more than 400 Islands and we wanted to create a game that could teach people about islands. We added a few non-Danish Islands as well. The aim of the game is to shape an Island that matches a given Island. Only the outer contour of the Island is compared (not the height). The game is played like this:

It is possible to add more Islands. Instructions will be added later.

This game was used in an educational event at the Danish Island Bornholm as seen in this video.

This game was mainly developed by Rasmus R. Paulsen.

The Sandimals 2-player game

In this game the box is divided into two halfes where each player is only allowed to move the sand in his half. The goal is collect as much food and as many skins as possible in 5 minutes. You get skins by having rabbits on your half and you get food by having fish on your half. The more rabbit the more skins per second. The more and bigger fish the more food per second.

Before starting the game you should flatten the sand and shape a big lake in the middle of the box.

The game is started by pressing f on the keyboard. After 5 minutes the game stop and the player who has the most food+skins wins the game.

You can also start the game by pressing 1 (complete beginner), 2 (novice), 3 (standard) and 4 (expert).

The behaviour of fish:

The behaviour of sharks:

The behaviour of rabbits:

Fish and sharks can be taken and moved using your hands if you shape them like a bowl.

This game was mainly developed by Rasmus R. Paulsen.

The animal and their mothers game

A mother fish and a mother rabbit can be enabled. The user can help the animals to reach their mothers by digging rivers or building mountains in the sand.

The game is started by pressing m on the keyboard.

This game was mainly developed by Thomas Wolfe.

Coding and Extending Magic Sand

Source Code

The full source code for Magic Sand is available on github.com/thomwolf/Magic-Sand.

Dependencies

Magic Sand is based on openframeworks release 0.9.3 and makes use of the following addons:

Quick start for editing the source code

Be sure to check the openframeworks documentation and forum if you don't know it yet, it is an amazing community !

How it can be used

The code was designed trying to be easily extendable so that additional games/apps can be developed on its basis.

Note that some of the below descriptions are slightly out-of-date.

The KinectProjector class handles the communication with the kinect sensor, the calibration and the coordinates conversions between kinect (2D), world (3D) and projector (2D) coordinate systems.

You can create a KinectProjector object as a shared_ptr in the setup() function of your openframeworks app. It requires a pointer to the projector window (see provided main.cpp on how to properly setup two windows in openframeworks and get a pointer to the projector window).

The kinectProjector object can be shared among the various objects that need access to depth and conversions functions (not multi-thread proof of course).

For instance, a SandSurfaceRenderer object can be constructed with a pointer to the kinectProjector shared object. (the SandSurfaceRenderer class convert depth information in color using a editable colormap and display these colors on the sand).

A typical setup() function of a openframeworks app can thus reads:

std::shared_ptr<ofAppBaseWindow> projWindow;
std::shared_ptr<KinectProjector> kinectProjector;
SandSurfaceRenderer* sandSurfaceRenderer;

void ofApp::setup() {
    kinectProjector = std::make_shared<KinectProjector>(projWindow);
    kinectProjector->setup(true);

    sandSurfaceRenderer = new SandSurfaceRenderer(kinectProjector, projWindow);
    sandSurfaceRenderer->setup(true);
}

setup(true) indicates that the GUI of the kinectProjector and the sandSurfaceRenderer will be displayed.

The kinectProjector object then needs to be updated in the update() function of the openframeworks app (preferably before the objects that use its functions) and drawn in the projector draw() function.

The kinectProjector object needs full control on the projector window during the calibration process so you should be careful not to draw things on the projector window after the call to kinectProjector->drawProjectorWindow() if a calibration is currently performed (you can check kinectProjector->isCalibrating()).

The following example illustrates the update() and draw() functions to implement a simple augmented reality sandbox once the kinectProjector and sandSurfaceRenderer objects have been initiated as detailed above and provided that the projector window has a listener callback setup to the drawProjWindow(ofEventArgs &args) function (see main.cpp).

void ofApp::update(){
  kinectProjector->update();
  sandSurfaceRenderer->update();
}
void ofApp::drawProjWindow(ofEventArgs &args){
  kinectProjector->drawProjectorWindow();

  if (!kinectProjector->isCalibrating()){
      sandSurfaceRenderer->drawProjectorWindow();
      fboVehicles.draw(0,0);
  }
}

The source code of Magic Sand itself is a simple example on how to use the main KinectProjector class to make a simple game.

kinectProjector Functions

Shader functions

The sandSurfaceRenderer class shows example of shaders that can be used to compute color and how to set uniforms.

The following function of KinectProjector are of special interest to setup a uniform.

void bind();
void unbind();
ofMatrix4x4 getTransposedKinectWorldMatrix();
ofMatrix4x4 getTransposedKinectProjMatrix();

The sampler2DRect received in the shader is normalized between 0 and 1, a conversion scale thus has to be also sent.

Coordinate conversion / elevation functions

Three coordinate systems can be used:

The most straighforward conversion goes from kinect coordinates to world coordinate system and projector coordinate system. If you want to animate or display objects, a natural choice would thus be to store then in kinect coordinate and to perform the conversion on display.

The following functions provide conversions between the coordinate systems:

ofVec2f worldCoordToProjCoord(ofVec3f vin);
ofVec3f projCoordAndWorldZToWorldCoord(float projX, float projY, float worldZ);
ofVec2f kinectCoordToProjCoord(float x, float y);
ofVec3f kinectCoordToWorldCoord(float x, float y);
ofVec2f worldCoordTokinectCoord(ofVec3f wc);

Another value that can be used is the elevation which is the distance from a point in world coordinate to a 3D base plane of that is defined by:

elevation can be converted/accessed by the following functions:

float elevationAtKinectCoord(float x, float y);
float elevationToKinectDepth(float elevation, float x, float y);

KinectProjector also store a matrix of gradients of the kinect depth in the world coordinate system (slope of the sand) computed with a given resolution (with a 10 pixels bin by default). The gradient at a given location can be accessed by:

ofVec2f gradientAtKinectCoord(float x, float y);

Setup & calibration functions

startFullCalibration() perfoms an automatic calibration of the kinect and the projector. An automatic calibration comprises:

The following functions can be called to change some internal values of kinectProjector:

Kinect projector state functions

The following functions give information of the state of the kinectprojector object:

Kinect projector other getters

The following functions give additional information :

Main differences with SARndbox

Magic Sand is a cross-platform project while SARndbox currently is only Linux. SARndbox is inherited from a larger VR toolbox that makes is somewhat daunting to start modifying. We hope that Magic Sand is slightly easier to start with.

Magic Sand uses the build-in registration feature of the kinect to perform an automatic calibration between the projector and the kinect sensor and does not use a pixel based depth calibration.

It is thus probably less acurate than SARndbox.

Magic Sand does not provide dynamic rain features (typically require a stronger GPU than the graphic card provided on a laptop).

Changelog

1.5.4.1 - 10-10-2017

Bug fix release

Bug fixes

Added

1.5.4 - 23-09-2017

Minor release of Magic-Sand-with-Games

Added

Changed

Bug fixes

1.5.0 - 08-08-2017

Initial release of Magic-Sand with Games