openFrameworks + ARKit
This is an openFrameworks addon that provides some helper classes for working within ARKit.
First
then
After you've opened up the project file
ofxIOS-Info.plist
file. See Permissions below.Shader.metal
file, see the next sectionNote that you may have to repeat these steps if you make any changes to your project via the generator.
See the Wiki for general tips and how to get started.
This addon involves use of Apple's Metal framework to handle camera imaging and processing as the raw feed from your phone/tablet's camera comes in unprocessed. In the src files there is a Shader.metal
file included which handles turning that unprocessed image into something more recognizeable. There is also Metal code that takes care of translating the resulting Metal textures to OpenGL compatible ones. At the time of this writing, for some reason, projectGenerator
does not treat .metal
files correctly and labels it as a Data file instead of a Metal file. You will likely have to change it to it's correct designation by using the file inspector in XCode.
projectGenerator
also does not automatically add it to the list of compiled sources for a project so you will have to do that as well.
On a related note - you will likely see what appear to be OpenGL ES related warnings - that is expected and should, for the moment at least, not cause your application to fail to compile or run.
While most features are generally supported across all devices that support ARKit, there may be some features that require specific hardware. See the ARKit website for more details. You should see any limitations listed at the bottom of the page in the footer.
For ARKit and iOS in general,a there are several permissions you may need to request from the user depending on the kinds of features you're looking to utilize. At a minimum you'll have to enable the Privacy - Camera Usage Description
in your ofxiOS-Info.plist
file. The value for this field is just the string you want to show users when you ask for camera permissions. If you've never touched a plist file before, no worries! Its very easy to change.
For example in the screenshot below
You'll see I added it to the very end. If the permission isn't there, all you need to do is over over one of the items already in the list and click on the plus sign. This will add a new field and you can just start typing Privacy - Camera Usage Description
. Xcode will attempt to autocomplete as well.
If you are trying to use the AirPod Motion with it, you must add an other field : NSMotionUsageDescription
.
By default #AR_FACE_TRACKING
is turned on, allowing you to try out examples
such as example-face-tracking
(if you have an iPhone X). We keep this
variable on by default in order to make the plugin easy to experiment with, but if you're
not using the TrueDepth API for face tracking in your app then you'll get issues trying to
publish to th Apple App Store:
"We noticed your app contains the TrueDepth APIs but we were unable to locate these features in your app. Please provide information about how your app uses the TrueDepth APIs."
To avoid this if you're not using TrueDepth & going to publish to the App Store change the macro defined in ofxARKit.h
in your OpenFrameworks plugins directory to false
:
// Line 2 of ARFaceTrackingBool.h
- #define AR_FACE_TRACKING true
+ #define AR_FACE_TRACKING false
This will remove the code from compilation so you don't get flagged by Apple for including code you're not using.
Please feel free to open up a ticket as I don't visit the openFrameworks forum all that often and will likely miss things.
For me, time is unfortunately a luxury as creative coding is sadly not my day job; I can only hunt and peck at small things here and there. All that as well as a general lack of knowledge on topics required to work effectively with AR all contribute to making it difficult to keep this up to date as I ought to; sooo, if there's something you feel you can contribute, by all means, feel free to make PR's!
As long as it doesn't break anything I'll most likely accept it.
First and formost, please work from the develop
branch. This is usually the most up-to-date branch and is intended to be a staging ground for any new features or changes. Once you are done with the feature you would like to add, please make all PRs against the develop
branch
A big thank you to all contributors thus far!