improve synced view tracking

[incanus]

General ticket for improvement of "sticking" native, Cocoa-side views to the panning, zooming GL render view.

This relates to the user dot now, later to callouts and annotation views. Let's keep a ticket around until we are 100% satisfied with the behavior here, but we can move it along milestones as we make staged improvements.

[incanus]

I'm feeling pretty good about the current state of tracking, especially after the above commit which avoids invalid CLLocationCoordinate2D-based updates during TransformState "will change" events (and just catches the "did change").

The one issue I'm seeing which I'd like to clean up is the dot disappearing during e.g. a double-tap animated zoom. It doesn't show up again until the next CLLocationManager update. It gets hidden because the CGPoint received out of -[MGLMapView convertCoordinate:toPointToView:] is well out of range of given screen bounds. I think this is just some sort of math error in TransformState during animations.

[incanus]

Another lead that confirms the above: pan the map until the dot is >150px off screen, then hit the user locate button. The map pans back into view (meaning the map centering to the valid user location is of course working) but the dot doesn't show up until the next CL update post-animation. During the animation it's still residing at the offscreen value, so it's hidden.

[incanus]

Tracking is pretty good right now. Only outstanding issue I'd like to hit for the beta is the dot disappearing briefly during animated zooms (but coming back at least, now). I'm nearing some completion on this but don't want to hold up a build tonight so I'll work on that a bit over the weekend.

Going to squash & merge a few commits now with improvements thus far.

[incanus]

First batch of improvements in https://github.com/mapbox/mapbox-gl-native/pull/1151.

[incanus]

Main blocker here now captured in the more general https://github.com/mapbox/mapbox-gl-native/issues/1160 bug with projection conversion.

[kkaefer]

FWIW, I think we should get rid of overlaying iOS views onto the Map and instead render everything ourselves. Apple recommends against doing this as well: https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/Performance/Performance.html#//apple_ref/doc/uid/TP40008793-CH105-SW8

For the absolute best performance, your app should rely solely on OpenGL ES to render your content.

[incanus]

Right, but I direct your attention back to https://github.com/incanus/AniAnnoDemo and what needs to happen to make this work :-) The thing I'm most uncertain about is syncing frames of animation from native views into GL. Maybe it's not as terrible as I'm imagining though.

[mb12]

@incanus Perhaps, we should try posting the synchronization problem to Apple developer forums and/or use the 3 incidents that Apple includes in developer support. Apple is already doing this for Mapkit. Its a problem that they have already solved. It would make writing all kind of Map overlays super easy.

https://developer.apple.com/devforums/

[incanus]

http://stackoverflow.com/a/17096916/977220

In fact, you can't synchronize them using current APIs. MobileMaps.app and Apple Map Kit use private property on CAEAGLLayer asynchronous to workaround this issue.

I'm intending to dig deep when the time comes for this, but right now the user dot is syncing well and we just need the bug fix in https://github.com/mapbox/mapbox-gl-native/issues/1160 for a workable version right now.

From the POV of annotations, after shapes, this will be my next priority.

[incanus]

On this front, I've been thinking about looking into routes for capture a view's layer's presentationLayer at a given point in time and rasterizing it to a GL render layer. That is, all tracked views atop the GL view will actually get rasterized into a single "layer" in the GL. I have no idea how feasible this is, but is the sort of approach I've been mulling over researching.

[incanus]

We are ok here for b2 and b3 and beyond will feature much more robust point annotations.

[friedbunny]

Per WWDC and chat and Stack Overflow:

In iOS 9 CAEAGLLayer has presentsWithTransaction property that synchronizes the two.

[incanus]

As part of #1784, I'm looking into this again. While the iOS 9 API looks intriguing, we need a solid iOS 7 & 8 approach anyway.

Using what's in view-based-points works fairly well, however some lag is evident. Using the Time Profiler in Release configuration reveals our three main resource hogs:

1. -[UIView setCenter:] at 7.5%. This is called across each annotation view to update its position every frame sync.

1. CALayer animation commit internals at 2.1%. This improves the smoothness of syncing at a slight cost.

1. GLKView internal GL rendering at 1.9%.

Next steps are to learn from similar approaches for views/layers in our raster SDK and explore:

• CAScrollLayer which updates -scrollToPoint: during pans/rotates, which is where the lag is most noticeable, reframing itself on zooms. This collects all pan updates into one system-maintained position update, effectively, by using a class optimized for collective panning.
• Possible UIScrollView for a similar approach, keeping in mind the most annoying limitation to ever emerge from our raster SDK, https://github.com/mapbox/mapbox-ios-sdk/issues/72.

[incanus]

Tried a number of things in combination with reducing the number of UIView syncing calls (https://github.com/mapbox/mapbox-gl-native/commit/6987536957f5e5a3d8bf75307a6834ab891a6be6) such as:

• Directly driving -[GLKView display] instead of relying on the normal UIView-like -setNeedsDisplay and subsequent -[GLKViewDelegate glkView:drawInRect:] trigger, which only serves to move both GL and native-side views the same interval and doesn't change their relative relationship.
• Various combination of UIView and CATransaction animations wrapping both the GL render call and the UIView updates, which doesn't improve things noticeably due to the asynchronous nature that's allowed to be configured in the new iOS 9 API. This includes experimenting with the current private API -[EAGLLayer asynchronous] property (http://openradar.appspot.com/radar?id=3118401).
• Driving UIView updates in a new -[UIView drawRect:], which also suffers the same lag due to the nature of -setNeedsDisplay.
• Experimenting with calling GL renders from a non-main queue.
  • http://codereply.com/answer/4p0288/cadisplaylink-opengl-rendering-breaks-uiscrollview-behaviour.html

Because of the reduction in the number of sync calls being made, we now have no theoretical lag between when a GL frame is drawn and when the annotation views' are updated, but we still see a ghosting or stuttering effect due to the asynchronous nature of GL frame updates in relation to a normal CATransaction.

Now going to try a perhaps-crazy approach of:

• Tracking UIView annotations but hidden.
• Rasterizing the views' layers' presentationLayer throughout possible animation updates.
• Using the rasterized copy as a texture on a quad in the GL view.

Refs:

• http://stackoverflow.com/questions/10120962/combining-uiview-animation-blocks-and-opengl-es-rendering
• http://gamesfromwithin.com/wp-content/uploads/2010/03/GDC10_uikit_opengl.pdf
• http://www.tagwith.com/question_466813_how-to-track-a-zooming-uiscrollview
• http://stackoverflow.com/questions/12772723/replicating-uiview-drawrect-in-opengl-es?rq=1

Might be prohibitive, but if we could avoid constantly uploading the vertices, re-uploading unchanged textures, and doing anything with non-viewport relevant annotations, it just might work. We could also eventually improve this to handle alpha blended annotation views, intermingled z-ordering of native views and GL symbols, and the like.

Will start with a single annotation first to prove the concept.

[mb12]

Does the issue get fixed on iOS 9 beta with presentsWithTransaction?

[incanus]

The good news: I got a quad billboard drawing (within the iOS side) that updates its position based on the associated annotation. No texture yet, just a dumb colored quad.

The bad news: It still has jitter, just like a native UIView. So even drawing in GL atop the map view post-renderSync() isn't enough to prevent this. Sort of makes sense; Core Animation is also compositing at this step, so same adverse result.

The next thing I'm going to try is setting up a mechanism for communicating between GL and iOS about which annotations have iOS-side textures that will be provided (i.e. rasterized CALayer objects), changing those from normal Symbol rendering to a special quad, and communicating over the boundary what the actual texture is upon CALayer update iOS-side, uploading the texture to the GPU and applying it to the quad.

[incanus]

I think this meshes well with the MapKit-style annotation view reuse method.

1. Rasterize a view's layer and upload it as a texture, making a note of the "clean" status.
2. As long as the view/layer doesn't change appearance, don't re-upload the texture, even when instantiating new, reused annotation views with the same identifier.
3. Upon view/layer change for each annotation, mark "dirty" status, then re-upload the texture from a new rasterized snapshot as needed. This forks the number of textures across views with identical identifiers, but only when absolutely necessary. As long as the views aren't visibly changing, the same single texture can be reused across multiple annotations.

[incanus]

Encapsulating both the GL render calls and the UIView-based updates in a CATransaction block while using [(CAEAGLLayer *)self.glView.layer setPresentsWithTransaction:YES] seems to have no effect. Trying to figure out if I'm using it right.

[incanus]

I will push some code shortly to demo, but I am seeing some odd behavior.

• If in the GLKView draw routine, I call renderSync to do the normal map draw, then call my own GL commands (with a small, dumb shader) to draw a quad immediately afterwards (in the main thread), the quad jitters and lags a bit in comparison to map pans and zooms.
• If instead I take the same quad drawing code and put it into MapContext::render to call immediately after painter->render, there is no lag and the quad is perfectly in sync with the map. In this case, I am calling my custom GL code on the map thread.

Since renderSync is synchronous, I would think that in the first case it wouldn't matter that I'm drawing from two different threads, but that approach may be suspect.

Note that in the long run, I wouldn't be doing this sort of work on the main thread anyway so as to not impact UI responsiveness, so maybe I should just skip to an approach that draws the quad in the latter way anyway. I can next work on getting CALayer snapshots across the wire to use as textures for the quads.

[incanus]

It is also a possibility that GLKView is doing something smart around automatic buffer swapping based on calling thread.

[incanus]

A method to compare the two approaches:

• Jitter: Grab b02dc95b798a89d1f1a6ccc18329eb4ad1bdad6e, zoom the viewport to DC, add "100 annotations" (which will really be just one), and see a green quad on the map. Pan and zoom and observe jitter. This is drawn in GL, but the same effect is what happens with UIView.center updates, too.
• No jitter: Grab bf98dfa5659ce489ef99333c468a01272f8dc468, zoom the viewport to DC, and see the hardcoded, automatically-added quad. It tracks the map perfectly.

[incanus]

I've confirmed that -[EAGLContext presentRenderbuffer] is only ever called by GLKView internals from the main thread, and is thus not called between the call to renderSync and any external drawing is done. I did this by subclassing EAGLContext to log the method.

2015-06-29 11:43:27.529 Mapbox GL[378:21959] => presentRenderBuffer
2015-06-29 11:43:27.537 Mapbox GL[378:21959] 1. start renderSync
2015-06-29 11:43:27.557 Mapbox GL[378:21959] 2. end renderSync
2015-06-29 11:43:27.559 Mapbox GL[378:21959] 3. start gl marker
2015-06-29 11:43:27.564 Mapbox GL[378:21959] 4. end gl marker
2015-06-29 11:43:27.566 Mapbox GL[378:21959] => presentRenderBuffer
2015-06-29 11:43:27.572 Mapbox GL[378:21959] 1. start renderSync
2015-06-29 11:43:27.594 Mapbox GL[378:21959] 2. end renderSync
2015-06-29 11:43:27.595 Mapbox GL[378:21959] 3. start gl marker
2015-06-29 11:43:27.599 Mapbox GL[378:21959] 4. end gl marker
2015-06-29 11:43:27.601 Mapbox GL[378:21959] => presentRenderBuffer
2015-06-29 11:43:27.607 Mapbox GL[378:21959] 1. start renderSync
2015-06-29 11:43:27.628 Mapbox GL[378:21959] 2. end renderSync
2015-06-29 11:43:27.630 Mapbox GL[378:21959] 3. start gl marker
2015-06-29 11:43:27.632 Mapbox GL[378:21959] 4. end gl marker
2015-06-29 11:43:27.633 Mapbox GL[378:21959] => presentRenderBuffer
2015-06-29 11:43:27.638 Mapbox GL[378:21959] 1. start renderSync
2015-06-29 11:43:27.658 Mapbox GL[378:21959] 2. end renderSync
2015-06-29 11:43:27.659 Mapbox GL[378:21959] 3. start gl marker
2015-06-29 11:43:27.662 Mapbox GL[378:21959] 4. end gl marker
2015-06-29 11:43:27.664 Mapbox GL[378:21959] => presentRenderBuffer
2015-06-29 11:43:27.674 Mapbox GL[378:21959] 1. start renderSync
2015-06-29 11:43:27.699 Mapbox GL[378:21959] 2. end renderSync
2015-06-29 11:43:27.700 Mapbox GL[378:21959] 3. start gl marker
2015-06-29 11:43:27.703 Mapbox GL[378:21959] 4. end gl marker
2015-06-29 11:43:27.703 Mapbox GL[378:21959] => presentRenderBuffer

[incanus]

Next up is comparing the TransformState between the use inside of renderSync and outside as provided by the getter. Perhaps the geography/pixel conversion routines differ.

[incanus]

Transform is not a factor as the transformed screen point for a given annotation is the same between iOS and GL. Checking next the height, width, and other factors that might vary (for some reason?) between frames.

[incanus]

Upload vertex positions are the same between core-side and iOS-side GL markers, but the lag persists:

Black square is drawn inside of renderSync, green square is drawn in OpenGL immediately afterwards on the iOS side

I also checked that the vertex shaders were using the same basic procedure, as well as precision, and they are — just a simple:

attribute vec2 a_pos;
void main() {
   gl_Position = vec4(a_pos, 0.0, 1.0);
};

The core-side shader applies u_matrix, but in the case of these annotations, we pass the painter->identifyMatrix since we are doing the screen conversion ourselves.

One thing that does jump out at me in the above GIF is that finger-down drags seems to sync perfectly, but coasting/decelerated pans are where things get out of sync. Looking into that now.

[jfirebaugh]

One thing that does jump out at me in the above GIF is that finger-down drags seems to sync perfectly, but coasting/decelerated pans are where things get out of sync. Looking into that now.

That sounds telling. Please rebase on master and put the main-thread drawing here.

The Transform update for easing happens on the next line. In the code where your branch is currently, it happens on the rendering thread, so your code is racing against it.

[incanus]

:+1: Was closing out some hanging code from before the weekend; rebasing now to get up to speed.

[incanus]

Please rebase on master and put the main-thread drawing here.

FWIW this requires making the MapContext's painter public to Map so that we can use the shaders.

[incanus]

Disregard last comment. Just merged master.

Black square was custom GL drawing in MapContext, as a comparison with...

Green square, which was custom GL drawing in MGLMapView after calling Map::renderSync, which lagged.

After applying master, I just tried only the green square and the lag is gone. This is because we now drive rendering from the main thread, then do this custom GL code also in the main thread.

This should be enough for me to next:

1. Test straight UIView / CATransaction, but I'm guessing we'll still see the problem that's purportedly solved by the new iOS 9 API.
2. Regardless, move on to snapshotting CALayer and uploading as a texture in the MGLMapView custom GL code.

[incanus]

UIView tracking directly on the map view now works without jitter! Both the user dot and custom annotation views returned by the delegate track nicely.

Going to double-check that I'm not doing anything in particular in my branch by also testing master with the user dot, but if not, we can close here as solved by https://github.com/mapbox/mapbox-gl-native/pull/1624 and move back to building out the annotation view API in https://github.com/mapbox/mapbox-gl-native/issues/1784.

[incanus]

The user dot in master does behave smoothly and without jitter, but in both branches, I do notice a lag / movement of supposedly static markers during fast panning. I made sure that subtle user location changes weren't at fault by testing on my branch with a custom annotation view with a fixed coordinate.

Here's a device video capture: https://dl.dropboxusercontent.com/u/575564/floater.mov

Note how the fixed marker has varying locations, and you can see the subtle floating during a slo-mo playback of the video.

initial location

frame capture during fast panning

another frame capture

So it's not just perception; it's really moving slightly. This may be what the OpenGL / CATransaction sync-up is about, in which case, we may still want GL-side snapshotting.

More understanding / definitely a better effect here after #1624, but...

[incanus]

The drift happens in main thread GL drawing code as well as just UIView syncing, so it's not the CATransaction thing.

We could live with the drifting if we had to for now, I think, but I'll keep digging on it a bit.

[incanus]

Procedure for replicating / testing the "green square hack" — a GL quad drawn in the main thread immediately after renderSync and inside of MGLMapView:

1. Grab 1fb66068f35d5084315fdeb493191d5814773076.
2. Center so that DC is visible.
3. Add "100" annotations (adds just a single for now).
4. Observe green quad lag/floating (but no jitter).

[incanus]

I can pretty much again isolate lag (and what used to be jitter) to the pan deceleration / animation routine upon end of a pan gesture in iOS. Keeping you finger on the screen prevents the lag from appearing.

[incanus]

Currently trying to isolate whether it's specific to Transform or specific to gestures by writing up some animated transition code that tries to replicate the lag without gestures being involved (so, still eventually calling startTransition).

[incanus]

Using a patch like https://gist.github.com/incanus/41d85a875b032558a6ec proves that it's related to animated transitions and is not specific to gestures. Show the green square, long press the map, and watch it drift.

[incanus]

@jfirebaugh nailed this exactly in chat:

renderSync does two things in this order: 1. render, 2. update transition for easing if you draw after both of those, it will be out of sync you need to draw in between those two things

This is why the black square demo worked fine — that's exactly where I placed the draw code.

Refactoring things into a split between rendering and updating transitions lets this work. The green square, drawn in iOS code, doesn't lag!

Working on cleaning that up.

[mb12]

Does this explain the difference in syncing behavior between downward finger drag and upward finger drag?

[incanus]

I’m not familiar with that @mb12 — can you elaborate?

[mb12]

I was referring to this comment on the gif animation.

https://github.com/mapbox/mapbox-gl-native/issues/1125#issuecomment-116754125

[incanus]

Breaking things into two parts also fixes syncing issues with straight-up UIView annotations.

@mb12 Yes, there is now no lag.

[incanus]

While this will certainly be one of the weirder animations posted here, bear with me a moment.

I have a prototype of the ability to sample an animating UIView and upload a snapshot of it to a texture to be rendered by OpenGL each frame. This points at the future possibility of truly using reuse identifiers for annotation views and uploading them as reusable textures. It remains to be seen whether this is a performance gain, but I'm guessing that it could have benefits over simply tracking UIView objects on top of the GL view directly, plus we could reuse annotation views (animating or not) to move things from duplication in the view hierarchy to a memory cost savings in GL textures.

(Full-res video capture of this here)

What is happening here is the following:

1. There is a normal UIView (center right) with an image as its layer.contents which has a repeating rotation animation applied to it. Consider this a potential annotation view created in the expected way.
2. When frames are being redrawn, i.e. when the map is being moved, every frame we are sampling the view.layer.presentationLayer.contents into a CGImage.
3. We are also sampling the view.layer.presentationLayer.transform.
4. We use the image and the transform to draw into a CGContext, capturing the raw pixel array, which can then be uploaded as an OpenGL texture.
5. That texture is drawn on a simple quad (the weird tracking image).

In practice, the actual view would be hidden so that we only see the OpenGL one, incurring no resource usage to render the original UIView.

The weird in-and-out-of-frame bit is because the animation rotation origin and the context transform point of reference aren't the same and could be figured out with a little trial and error, causing the texture to appear to rotate in place just like the native view instead of rotate in and out of the quad. Note that the rotation is backwards, too. But the general proof is there that a UIView can be sampled as a texture into GL frames, smoothly animating.

I think we should punt on animation annotation views for now, as well as this texture upload business, but it's promising for future use.

I'm next going to focus on just a bare-bones annotation view API (no actual reuse identifiers), which I should be able to wrap pretty easily now that we've solved the syncing issues.

[incanus]

We could further optimize by passing the view.layer.presentationLayer.transform into the shader as the projection matrix, avoiding the need to re-upload the texture during animations. Then we'd only have to sample the view to a texture after view.layer.contents changes (which should be KVO observable). That would also let us reuse the same texture for all views with the same reuse identifier, since the matrix would vary for each quad being drawn (depending on animation state) but the texture wouldn't.

[incanus]

Going the basic UIView route for now in #1784, but first we need to split the renderSync call per https://github.com/mapbox/mapbox-gl-native/pull/1813.

[incanus]

Per chat, @kkaefer is going to look into this & https://github.com/mapbox/mapbox-gl-native/pull/1813 to improve our current user dot.

[incanus]

I'm grabbing this for the b3 release branch.

[incanus]

We are good for release-0.5.0 now as of https://github.com/mapbox/mapbox-gl-native/commit/b8388168dd130c67c77254565cdb576df7905915 and https://github.com/mapbox/mapbox-gl-native/commit/a67e80219f90c409dfce8bfef91d9131dabd8d4f; moving off of milestone.

[ljbade]

Making note of #2384 as it looks like iOS 9 has some fixes that can be used.

[incanus]

We started looking at that in https://github.com/mapbox/mapbox-gl-native/issues/1125#issuecomment-111636947 @ljbade. No help currently.

[1ec5]

3683 finally synchronizes the user dot with the map. Keeping this issue open until we’re able to leave callouts open while the viewport moves.