Issues with monocular rendering in portrait mode

[RikHeijdens]

We are experiencing a couple issues with monocular rendering in Portrait mode. We believe that all of the issues we're experiencing may be related to each other.

In the - (void)cardboardView:(GVRCardboardView *)cardboardView drawEye:(GVREye)eye withHeadTransform:(GVRHeadTransform *)headTransform method we noticed the following bugs:

If eye == kGVRCenterEye the [headTransform viewportForEye:eye] method call reports an incorrect viewport. It reports the dimensions of the screen in points, instead of the dimensions of the GVRCardboardView in pixels. When rendering in stereo, the viewport appears to be correct.

We managed to work around that by scaling the viewport with the scaling factor returned by [UIScreen mainScreen].scale, but at that point we ran into the next issue: when the user interface orientation is in portrait mode the GVRInternalGlView does not scale to the dimensions of the GVRCardboardView. And only a portion of the GVRCardboardView's frame is used for rendering.

We also observed that the viewport returned by [headTransform viewportForEye:eye] returns the width value for the height value and vice versa when the device is in portrait mode.

Furthermore [headTransform projectionMatrixForEye:eye near:Z_NEAR far:Z_FAR] returns a projection matrix with a very wide FoV in portrait mode. I'm assuming this is because the SDK is not calculating the aspect ratio correct.

Being able to render in portrait mode is very important for our application, we would appreciate it if these issues could be fixed within a reasonable amount of time.

Please let me know if you have any questions, or if you need more information.

Thanks

[nathanmartz]

There appears to be a 90 degree rotation applied in our code. This is a bug, but may not be fixed quickly. Here's a workaround from one of our internal clients:

UIInterfaceOrientation orientation = [[UIApplication sharedApplication] statusBarOrientation];
switch (orientation) {
  case UIInterfaceOrientationPortrait:
    _cardboardView.transform = CGAffineTransformMakeRotation(-M_PI_2);
    break;
  case UIInterfaceOrientationLandscapeLeft:
    _cardboardView.transform = CGAffineTransformMakeRotation(M_PI);
    break;
  case UIInterfaceOrientationLandscapeRight:
    _cardboardView.transform = CGAffineTransformMakeRotation(0);
    break;
  case UIInterfaceOrientationPortraitUpsideDown:
  case UIInterfaceOrientationUnknown:
  default:
    _cardboardView.transform = CGAffineTransformMakeRotation(-M_PI_2);
    break;
}

[RikHeijdens]

Thank you Nathan, excellent workaround.

In order to make it work I also had to rotate the eyeFromHeadMatrix:

GLKMatrix4 projectionMatrix = [headTransform projectionMatrixForEye:eye near:Z_NEAR far:Z_FAR];
GLKMatrix4 eyeFromHeadMatrix = [headTransform eyeFromHeadMatrix:eye];

if (eye == kGVRCenterEye && UIInterfaceOrientationIsPortrait(_orientation)) {
    eyeFromHeadMatrix = GLKMatrix4RotateZ(eyeFromHeadMatrix, GLKMathDegreesToRadians(270));
}

[nathanmartz]

Hooray! Glad you got it to work.

[justsomeguy-google-com]

Is this still a problem with the latest SDK?

[RikHeijdens]

We haven't upgraded yet, but we are planning to upgrade soon. I'll let you know if I run into any issues.

[justsomeguy-google-com]

Working on a fix right now.

Can you post the exact results you expect for all 4 rotations? Don't care what device you use, posting for multiple devices might be a good idea.

I think I have the right values, but with the long turn around time I'd like to make sure I send you exactly what you expect.

[RikHeijdens]

I've added some logging to - (void)cardboardView:(GVRCardboardView *)cardboardView drawEye:(GVREye)eye withHeadTransform:(GVRHeadTransform *)headTransform.

Code used to determine the viewport and projectionMatrix:

    // Set the viewport.
    CGRect viewport = [headTransform viewportForEye:eye];
    glViewport(viewport.origin.x, viewport.origin.y, viewport.size.width, viewport.size.height);
    glScissor(viewport.origin.x, viewport.origin.y, viewport.size.width, viewport.size.height);
    NSLog(@"X: %f, Y: %f, W: %f, H: %f", viewport.origin.x, viewport.origin.y, viewport.size.width, viewport.size.height);

    // Get this eye's matrices.
    GLKMatrix4 projectionMatrix = [headTransform projectionMatrixForEye:eye near:Z_NEAR far:Z_FAR];
    GLKMatrix4 eyeFromHeadMatrix = [headTransform eyeFromHeadMatrix:eye];

Output for portrait: X: 0.0, Y: 0.0, W: 421.0 (expected: 750.0) px, H: 750.0 (expected 421.0) px

Visual result:

As you can see the FoV is very wide, much wider than I would like it to be.

Output for Landscape: X: 0.0, Y: 0.0, W: 750.0 px H: 421.0 px

Visual result:

In landscape mode the values are correct, and the projectionMatrix has a FoV of about 90 degrees, which is what we expect.

I tested this with version 1.0.1 of the GVR SDK. Does this answer your question and provide the data you need?

[justsomeguy-google-com]

OK, that makes more sense.

I want to verify, the result you get right now just needs to be reversed and the eyeFromHeadMatrix needs a rotation. The scale is right.

[RikHeijdens]

Correct. I don't remember exactly how I ran into the scaling issue but I didn't run in to it just now.

[justsomeguy-google-com]

I've tried playing with this with our demo apps and they all work fine in magic eye mode in portrait. I have seen some issues with the simulator, but on a device they all work right.

Obviously you are doing something different than we are in our examples. Can you give me a small project that reproduces your issues?

[RikHeijdens]

Apply the following patch on this repository:

From a0b8f82a18bed540908fd1657c499848ac996223 Mon Sep 17 00:00:00 2001
From: Rik Heijdens <rik@jwplayer.com>
Date: Wed, 7 Dec 2016 17:13:06 -0500
Subject: [PATCH] Bug PoC for Michael Rutman

---
 Samples/TreasureHunt/TreasureHuntRenderer.m       |  1 +
 Samples/TreasureHunt/TreasureHuntViewController.m | 16 ++++++++++------
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/Samples/TreasureHunt/TreasureHuntRenderer.m b/Samples/TreasureHunt/TreasureHuntRenderer.m
index 62b0ef1..c2c83e8 100644
--- a/Samples/TreasureHunt/TreasureHuntRenderer.m
+++ b/Samples/TreasureHunt/TreasureHuntRenderer.m
@@ -548,6 +548,7 @@ - (void)cardboardView:(GVRCardboardView *)cardboardView
   CGRect viewport = [headTransform viewportForEye:eye];
   glViewport(viewport.origin.x, viewport.origin.y, viewport.size.width, viewport.size.height);
   glScissor(viewport.origin.x, viewport.origin.y, viewport.size.width, viewport.size.height);
+  NSLog(@"X: %f, Y: %f, W: %f, H: %f", viewport.origin.x, viewport.origin.y, viewport.size.width, viewport.size.height);

   // Get the head matrix.
   const GLKMatrix4 head_from_start_matrix = [headTransform headPoseInStartSpace];
diff --git a/Samples/TreasureHunt/TreasureHuntViewController.m b/Samples/TreasureHunt/TreasureHuntViewController.m
index e219e0a..0dfd161 100644
--- a/Samples/TreasureHunt/TreasureHuntViewController.m
+++ b/Samples/TreasureHunt/TreasureHuntViewController.m
@@ -13,15 +13,18 @@ @interface TreasureHuntViewController ()<TreasureHuntRendererDelegate> {
 @implementation TreasureHuntViewController

 - (void)loadView {
+  [super loadView];
   _treasureHuntRenderer = [[TreasureHuntRenderer alloc] init];
   _treasureHuntRenderer.delegate = self;

-  _cardboardView = [[GVRCardboardView alloc] initWithFrame:CGRectZero];
+  UIScreen *mainScreen = [UIScreen mainScreen];
+  CGRect bounds = mainScreen.bounds;
+  CGRect frame = CGRectMake(0,
+                            [UIApplication sharedApplication].statusBarFrame.size.height + 10,
+                            bounds.size.width,
+                            (bounds.size.width / 16) * 9);
+  _cardboardView = [[GVRCardboardView alloc] initWithFrame:frame];
   _cardboardView.delegate = _treasureHuntRenderer;
-  _cardboardView.autoresizingMask =
-      UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight;
-
-  _cardboardView.vrModeEnabled = YES;

   // Use double-tap gesture to toggle between VR and magic window mode.
   UITapGestureRecognizer *doubleTapGesture =
@@ -29,7 +32,8 @@ - (void)loadView {
   doubleTapGesture.numberOfTapsRequired = 2;
   [_cardboardView addGestureRecognizer:doubleTapGesture];

-  self.view = _cardboardView;
+  //self.view = _cardboardView;
+  [self.view addSubview:_cardboardView];
 }

 - (void)viewWillAppear:(BOOL)animated {
-- 
2.9.3 (Apple Git-75)

Then, compile and run the app.

Notice the logging output: for me the width and height of the viewport is inverted. Rotate from portrait into landscape and see the difference in FoV.