search

doughtmw / display-calibration-hololens

Point correspondence-based display calibration and ArUco marker tracking for the HoloLens 2.
46 stars 9 forks source link
aruco calibration hololens2 opencv tracking

readme

Display Calibration HoloLens

Point correspondence-based display calibration and ArUco marker tracking for the HoloLens 2.

This is the accompanying GitHub repo for our paper investigating the perceptual accuracy of manual tasks using an optical see-through head-mounted display (OST-HMD) for guidance:

Table of contents

Project setup

Folder setup for the project sample.

data/
... GroundTruths/
... MarkerConfig/
... SampleResults/
... ... Figures/
... ... Matlab/
... ... Traces/
... ... Truths/
... TraceTargets/
... TraceTemplate/

unity-sandbox/
... CustomArUcoBoards/
... HoloLens2-Display-Calibration/
... OpenCVRuntimeComponent/

Data

We have included the ArUco board tracking configuration that we used in our study, along with the calibration marker used for per-user display calibration. There are several test cases of scanned user-trace results available in the data/SampleResults/Traces folder. Additionally, we have included the relevant Matlab scripts used for processing the user-trace data and computing relevant metrics of accuracy.

Printing the required calibration object and tracking board

To be able to use this repo for tracking and display calibration, we need to first print off the required materials.

  1. In the data/TraceTemplate/ folder, open the trace_template.pdf document (or data/MarkerConfig/ and the board_config.pdf document) and print at 100% scale (actual document size). Attach/tape down the trace template onto a flat object to hold it in place during tracking.
  1. In the data/MarkerConfig/ folder, open the calibration_marker_config.pdf document and print at 100% scale. Create a calibration marker as shown below using cardboard and a handle.

You now have the materials prepared for calibration and tracking.

Optional: Confirming the board object for tracking

Running on the HoloLens 2

About

Optional: Camera calibration for the HoloLens 2

Optional: Build runtimes from source for ARM64

Deploy and run the sample on the HoloLens 2

Open HoloLens2-Display-Calibration in Unity. There are several different scenes available for selection in the Assets/Scenes/ folder, as detailed in our paper, these will determine which tracing paradigm (adjacent, direct, or calibrated) will be used for the experiment.

hmd adjacent: There is no calibration procedure required, when looking at the printed trace template, you will see a virtual model augmented adjacent to the tracking target.

hmd calib: There is an additional calibration procedure required, with 10 point correspondences collected per eye. For the point collection process you need to align the tracked marker corners with the virtual on-screen reticle and press spacebar on a bluetooth or USB keyboard connected to the HoloLens 2 (or use the double-tap gesture) to collect that point correspondence. Virtual marker positions will change slightly across each calibration point correspondence. Begin calibration with your right eye (left eye closed), next calibrate left eye (right eye closed).

After completing calibration, set down the calibration object and look at the printed trace template. You should now see a virtual model augmented directly on the tracking target.

hmd: There is no additional calibration required, when looking at the printed trace template, you will see a virtual model augmented directly on the tracking target.

There are a number of settings which can be adjusted on the ScriptHolder gameobject in Unity, including:

After selecting the desired tracking scene, we can now build the scene to the device.

Evaluation

Evaluation of the user trace performance relative to ground truth points and contours was performed in Matlab.

Citation

If you found this code repo useful, please consider citing the associated publication:

@article{jimaging8020033,
    author={Doughty, Mitchell and Ghugre, Nilesh R.},
    title={Head-Mounted Display-Based Augmented Reality for Image-Guided Media Delivery to the Heart: A Preliminary Investigation of Perceptual Accuracy},
    journal={Journal of Imaging},
    volume={8},
    year={2022},
    number={2},
    article-number={33},
    url={https://www.mdpi.com/2313-433X/8/2/33},
    issn={2313-433X},
    doi = {10.3390/jimaging8020033}
}