Differentiating anchors onscreen vs. offscreen

[alexrothera]

Thanks again for the killer ARKit addon :)

Question: I'm trying to build something so that visuals (either animation or audio) trigger when the view sees them on screen. For example, a room might have 10 anchors plotted around the room, but only animate when the viewer seeings them on screen (eg. think the opposite of the stalking ghost in Mario).

Is there a good way to know which anchor points a user created are visible? I'm building from the "example-anchormanager" code and I assume the solution would be something in relation to the ofCamera? (line [48])

https://github.com/sortofsleepy/ofxARKit/blob/4f07e68f74b376ff20280144bbc5fc7d0f651390/example-anchormanager/src/ofApp.mm#L48

But any ideas/solutions would be great :)

[aferriss]

I think you should be able to use the worldToScreen() function in the utils to test if a point is within the frame or not. This is untested but I think should work...

// loop through all the anchors
for(int i = 0; i<anchors.size(); i++){

   // get anchor xyz position
   ofVec3f xyz =  ARCommon::getAnchorXYZ(anchors[i].transform);

   // find anchor position in screen space
   ofVec2f point = ARCommon::worldToScreen(xyz, processor->getProjectionMatrix(), processor->getViewMatrix() );

   // check if it's on screen or not
   if( point.x > 0 && point.x < ofGetWidth() && point.y > 0 && point.y < ofGetHeight() ){
      // point is on screen
   } else {
      // point is off screen
   }

}

If using the anchor-manager example, you can just use obj.transform instead of the anchors[i].transform I wrote above.

[sortofsleepy]

Hey there Alex!

Hmmmm, there are a couple of ways you could approach this off the top of my head.

I think Adam's idea of comparing your position to the anchor is probably the easiest but if you wanna get fancy

• now that 11.3 is out - you could try taking advantage of the image recognition features. The catch is that with this route, nothing is currently built into the addon that could help with this(though I think if you look in the 11.3 branch, you might find some helpful code, Brett was working towards this a couple weeks ago)

• you could try marker tracking. this is a library I've tried at work recently. I believe there is another ofxAddon that uses this library.

• depending on the shape, you could try your hand at shape tracking too with just plain OpenCV.

[aferriss]

Oh, I might have misunderstood the question. If you are trying to detect if a specific object is within the frame (like a banana or a tea cup) that's a different problem. The Core ML / Vision frameworks might be good places to start for that, although they don't give you any 3d information.

There's also the ofxDarknet addon that does image recog quite quickly, though I haven't tried it on mobile. There's also a new version of the yolo net out that is supposed to be really fast but will probably take some work to get it gelling with ofxDarknet

[alexrothera]

@aferriss Nope, you were correct the first time. I'm gonna give your first solution a shot now. Will report back.

ofxDarknet is great, but for another project maybe.

[alexrothera]

@aferriss Great solution, thanks. And your pseudo code was super close. Just needed one extra step to convert to ofMatrix4x4.

//Loop through all Anchors
for (int i = 0; i < session.currentFrame.anchors.count; i++){ 

//Anchor index
ARAnchor * anchor = session.currentFrame.anchors[i];

// get anchor xyz position
ofMatrix4x4 temp = ARCommon::convert<matrix_float4x4, ofMatrix4x4>(anchor.transform);
ofVec3f xyz =  ARCommon::getAnchorXYZ(temp);

// find anchor position in screen space
ofVec2f point = ARCommon::worldToScreen(xyz, processor->getProjectionMatrix(), processor->getViewMatrix() );

// check if it's on screen or not
if( point.x > 0 && point.x < ofGetWidth() && point.y > 0 && point.y < ofGetHeight() ){
cout<<"on screen"<<endl; // point is on screen
} else {
cout<<"off screen"<<endl; // point is off screen
}
}

[sortofsleepy]

@alexrothera Sweet! Glad you got it working. FYI there's also ARCommon:: toMat4 which you can use instead of the convert function. Does the same thing but less to type out 😄

Closing this issue now.