Share-With-Images

AR iOS app for a vision-based tangible interface. This extends work by my advisor, Dr. Amanda Lazar, in supporting Digital Social Sharing for People with Dementia. The idea for the app is, as the name suggests, to use images to indicate actions, objects, and other categories of inputs that can be used to trigger apps on your iPhone through a simple communication model. I've used image-recognition functionality provided by ARKit 2.0 for this and provided a template that will allow me, or other developers, to adapt this communication model for the needs of the end users (See Issues and Pull requests for discussions, thoughts, sketches, and referred publications on topics like communication, AI assistants etc.). It needs to be mentioned that this app hasn't been tested by any users. So its a good prototype to present to a focus group and break apart in discussions of what it does well and what it fails at for a certain population (e.g. Older adults might actually hate it, I dont know).

This issue and its related pull-requests talk about the currently working functionality: Saying you want to send an email (desired action as input) and to whom (person as input). Its hardcoded for a Gmail image, and Obama image. As an example of sequential presentation of inputs, showing the image of the gamil app first followed by obama gives a different response to the reverse case. Try it out. This can also be changed to a non-sequential presentation of inputs as well.

A question you might have is why I'm even using AR specific libraries at all, why not just OpenCV? Well, they seemed to have good set of marker tracking libraries which I could easy use when developing for mobile. This is why I was initially interested in ARKit. I also had this idea of being able to create personalized AR spaces for memory-related activites (Display old photographs on the wall, preserve models of objects that hold meaning ...). These could, once again, use a lot of feedback from the community of older adults I planned to design this app for. But I definitely see potential for AR/VR in enabling shared experiences for a multigenerational userbase that includes older adults and this app is a practise exercise.

Requirements

• Tech:

  • Device with iOS 11.3 or greater
  • XCode 9.3 or greater to build the code (with ARKit)

• Knowledge: Ability to read, understand and/or write iOS app code. Don't worry if you're a novice or outright beginner. I've used and adapted apple tutorial code myself to write this app. Go through the following tutorials and links in the given order and you should be able to understand my code

  • Get started with Swift
  • Get started with iOS apps: This gives a good intro to XCode as well
  • Build your first AR Experience
  • Image recognition using ARKit and other related examples There are swift playgrounds in most of these links. Download them and try playing around with the code.

Expected Future Work:

• Work on the UI (the look and feel) (Dont want it to resemble the tutorial app too much :))
• Work on an AI component (See issue #16).
• The big one: extend to 3D objects and not just images as inputs. Maybe ARKit already has an object detection function implemented somewhere? Edit: The latest release of ARKit has 3D object scanning and detection capabilites (It currently doesn't seem too robust). We could now use, for example, a watch associated with a friend's memory to represent that friend for our tangible interface.
• Unknown ...
• Extend to android and desktop? (I've used ARKit in this one. Maybe there could be an equivalent version that uses some open source AR SDK?)