Migrate iOS OpenGL backend to Metal

[codenameone]

Since Apple is deprecating OpenGL it's time to move forward. This issue should be divided into an evaluation phase of picking a strategy:

• Direct metal access
• SKIA backend
• Something else

Then the process of implementing that strategy possibly as a duplicate port so developers can pick the old version of the port for compatibility. This will allow us to gradually phase out the old renderer in favor of the new.

[ramsestom]

Like I already said in the forum (https://www.codenameone.com/discussion-forum.html?place=msg%2Fcodenameone-discussions%2FJI5PQYPoXt4%2F3tISJSixCwAJ), I would be in favor of using Skia. Ok there is some (minimal in my point of view) drawback that is that you minimal IPA size would increase by 1 or 2 Mb (as you would have to embed the Skia lib in your app) but the advantages are completely worth it in my opinion:

• Skia is already compatible with multible platforms (Android, desktops, iOS (via OpenGL for now but the support of Metal is expected in the comming months, UWP (even if building it for this target seems a bit tricky): https://groups.google.com/forum/#!topic/skia-discuss/pGQ6hiR3rZQ) and there is WIP to support other targets (like browsers and javascript with WebGL). So you could writte a single renderer that maps CN1 graphic calls to Skia and it could directly work on multiple platforms (writte once works "everywhere", isn't it in the CN1 phylosophy? ;) )
• If ever Apple choose to change his shoulder gun in a few years by switching from Metal to Vulkan for example, it is likely that Skia will quickly support this (if it is Vulkan it already does in fact ;) ) as Skia is already used by significant cross-Platform frameworks like Flutter or Xamarin so it is likely to follow the evolution of graphics engines on platforms supported by these frameworks and you would directly benefit from the porting work of the Skia team instead of having to rewritte your own binding to the new graphic engine once again.
• On some platforms, like Android, directly binding to Skia instead of using the native java graphics API offers you more graphic capabilities than the one exposed in this java API. And having a common renderer for multiple platforms allows you to use all the possibilities offered by this engine, and not just the one that might be common to the various native APIs...

FYI bytedeco maintain a binding of java to the Skia C API (https://github.com/bytedeco/javacpp-presets/tree/master/skia). Might be usefull if you choose the Skia option

[shannah]

Thanks for the Skia information @ramsestom . Making a skia port using the bytedeco bindings could be interesting, and may result in a better foundation for cross-platform support moving forward since all of the drawing for all of the platforms could be consolidated into a single implementation.

For the specific task of adding metal support, though, it would be quite a bit easier to just write metal natively. It is quite similar to OpenGL, so it wouldn't require any architectural changes. We'd just need to change the existing OpenGL calls to their equivalents in Metal.

[shannah]

I've been playing around with Metal - going through tutorials, making simple sample projects, etc.. And I really like it. The fact that the shaders are compiled makes development much more sane than OpenGL (in which you routinely face a blank white screen, and have to stare at your shader code until you see what you did wrong - or find out that you did nothing wrong).

I recommend that we implement this issue directly in Metal. Rather than create a separate port, we can use define flags to toggle between Metal and OpenGL. There are still good reasons to keep OpenGL as a first-class citizen for at least the next 2 years. For one think Metal can't be run in the Xcode simulator. It needs a device. When Apple transitions off of Intel in 2020, I expect that you'll be able to use Metal in the simulator. Being able to run things in the Simulator is very important for development since I can't possibly have access to all of the devices and OS versions that we encounter.

I looked closely at Skia, and it looks very interesting, but I don't recommend it (for this issue) for 3 reasons:

1. Skia requires greater effort to make the change. Our OGL code can be converted to metal in a 1:1 fashion. Skia will likely require more architectural changes in the port.
2. Skia carries greater risk. We might invest a lot of time into a Skia port only to discover that the performance is not as good as native Metal (or even our existing OGL port) is, and we'll be forced to implement in Metal anyways. We don't know. Metal is almost guaranteed to provide at least the same performance of our existing OGL port.
3. Skia still doesn't fully support Metal. I expect this to change in the future, but it isn't clear right now what blocks we may hit while we wait for Metal support.

All that said, I think it would be worth while doing some R&D with Skia. It may open doors to support many things with 2D graphics that would be very difficult to implement and maintain on our own. We could share all of the graphics code between all of the different platforms; or we could simply use skia separately on each platform using the API/bindings that make the most sense for that platform. In either case it would greatly simplify both development of new features, and maintenance of old ones. It would likely result in better platform stability - fewer whack-a-mole situations.

@codenameone What do you think? I'd like to starting working on the Metal port right away.

[codenameone]

I agree with one point to notice. In OpenGL we use a special flag that retains the backing store when drawing. At least we did in the original implementation, if this was removed then no problem. If it's still there we might need to check if we can use that mode.

[ramsestom]

Did you get any chance to experiment using Skia as an alternative rendering engine for CN1 or is it still in the plans? I am asking as the support of Skia greatly improved for many platforms these last months (seems like th Flutter team is putting some pressure on the Skia team for it) and I would love to be able to render complex UI (requiering blending and color transform operations or efficient shodowing among others...) with CN1 as I can with Flutter for now.

[shannah]

I haven't experimented any more with Skia than the initial assessment. I am part way through implementing with native metal.

[shannah]

Created branch for METAL port. Not functional yet. https://github.com/codenameone/CodenameOne/tree/metal