[Feature] Display objects accelerated by GPU

[bobaoapae]

If possible when targeting render mode DIRECT make use of gpu to accelerate display objects render without the need to migrated to context 3D or use render mode GPU, as render mode gpu it's not recommended and has a lot of limitations.

[bobaoapae]

Or if not possible search if possible to remove some limitations of render mode GPU, like support for filters

[ajwfrost]

I believe the "cache as bitmap" option is what's meant to be used for acceleration here. In "direct" mode, the shape/vector rendering is done using the normal (assembler/simd based) rasterisation code, and the buffers are then uploaded to a texture for composition (along with other 3D-ready stuff like Stage3D and StageVideo etc). Having cache-as-bitmap should make these elements get uploaded once into a GPU texture and then composited from there.. but I'm not 100% sure on that (as it could lead to issues with overlaying things.. would only work if they're not obscured by other content..)

What particular elements are you seeing performance issues with?

thanks

[FliplineStudios]

Unfortunately there's the issue of cacheAsBitmap being highly unstable on Android devices (#103 ), so I wouldn't recommend using it in live apps anymore if you're releasing on Android. Using cacheAsBitmap on Android still has constant rendering issues on all Huawei devices, Amazon devices, some Samsung devices, and a good deal of others with ARM GPUs.

[bobaoapae]

Render Mode GPU: https://prnt.sc/QWI2rVXiw13T https://prnt.sc/7HaMQvOC3o4M

Render Mode Direct: https://prnt.sc/bzT3HrrNoHU- https://prnt.sc/VFg8tsaipLOh

[bobaoapae]

The performance diff it's crazy... With direct using cacheAsBitmap or not I cannot get even close of the performance of gpu mode. But gpu mode have several limitations, the bigger one it's the missing filter support.

The best update will be if display objects can be accelerated like in GPU mode but using mode with will make possible use stage3d if need and keep using old native display objects. Or fix the problems with gpu mode.

[bobaoapae]

@ajwfrost it's possível to improve something on this?

[ajwfrost]

Hi

I'm not able to see anything in those prnt.sc links..? but I guess you're seeing some speed-up in "gpu" mode vs "direct" mode..

What's the request here, I wasn't quite sure that you meant:

The best update will be if display objects can be accelerated like in GPU mode but using mode with will make possible use stage3d if need and keep using old native display objects. Or fix the problems with gpu mode.

Would you like to have display objects accelerated (like in GPU mode) whilst also in Direct mode i.e. being able to use Stage3D? So basically just a mixture of the two (it's probably simpler to enable Stage3D in the GPU rendering mode, I suspect..)?

Fixing the problems with GPU mode may have some limitations in terms of how some of this stuff works in the SWF format, but as you mention, the main thing is around filtering, and this appears to currently work by applying a filter to the display object in order to create a static bitmap (i.e. cache-as-bitmap), which is then used for rendering. So it might be possible to enable this in GPU mode via a similar two-stage approach where the filter is applied using some CPU rendering into a cached bitmap, and this is then used to render. But per the comment from @FliplineStudios above, there have been a number of issues with cached bitmaps (as textures) not having the correct contents on Mali GPUs so this may not be of such use. Although it might be that this uses a different mechanism in GPU mode anyway, we would have to check..

thanks

[FliplineStudios]

@bobaoapae Just a side note, if you want to use filters in GPU mode, you can get around the limitations a bit if you draw MovieClips into BitmapData, since the BitmapData.draw function will retain the filters applied to the MovieClip. It's a bit of a process, since you'll also have to swap out your MovieClip on the display list with a new Bitmap after rendering it, but for static content that doesn't need to update often it works pretty well. For animated content it would get trickier though, since you'd have to set up a sprite sheet or multiple BitmapData objects for each frame and handle that animation cycle yourself. I believe BitmapData.applyFilter still works in GPU mode too, if you're already dealing with bitmap assets.

[bobaoapae]

Yeah using this trick work to get the filters, it's just a little hard to do this and for some devices even doing this the filter don't work as expected.

The best update will be if display objects can be accelerated like in GPU mode but using mode with will make possible use stage3d if need and keep using old native display objects. Or fix the problems with gpu mode.

As for this let me try to explain better.

Apart of the problems with filters in GPU mode, don't know if has any other big limitation. Will be good if GPU mode be replaced complety by DIRECT mode, allowing using of STAGE3D but also having old plain DisplayObject rendered by the GPU. In this way it's possible to get the best performance and the best features without need to choose between using DisplayObjects created from Adobe Animete, Flash CS 5 etc, or using 3d frameworks like Starling. With this a lot of old games builded targeting purely the old flash player or adobe air can be easy ported to Android/IOS and also will simply for new programers in AS3, as this kind of configurations and limitations can make new programers don't like the ecossitem and go for others

[bobaoapae]

@ajwfrost Maybe it's easy to add option to change GPU/DIRECT somehow without need to repack the app... some external config file that i can change via AS3 or via ANE that after a restart will start the app with the renderMode changed, with this based in some device capability, fps count during runtime, or some advanced config available for users, we can have the control withtou need to use 2 separated build, as during runtime we already can check the renderMode and change the application behauviour

[bobaoapae]

@ajwfrost can this be done somehow?

[ajwfrost]

Hi

I think this would potentially work - there are two suggestions here, and I can't think of a technical reason why we'd not be able to implement them.

1. GPU-based rendering together with Stage3D support, the only thing I'm wondering is whether the two mechanisms might interfere with each other. So we may need to check this..
2. Dynamic switching of render mode - I think this should be possible with the caveat that it would probably result in some flickering as the window modes are changed around. We can have a play with this one..

thanks

[FliplineStudios]

@ajwfrost If you're exploring that first option, hopefully it doesn't involve changing how GPU render mode works currently, since that could severely affect all of our existing and in-progress apps! I'd much rather see any changes being moved into a separate 'hybrid' render mode if needed, if there are going to be big changes to rendering.

[ajwfrost]

Yes for sure, very good point :-)

[bobaoapae]

Hi

I think this would potentially work - there are two suggestions here, and I can't think of a technical reason why we'd not be able to implement them.

1. GPU-based rendering together with Stage3D support, the only thing I'm wondering is whether the two mechanisms might interfere with each other. So we may need to check this..
2. Dynamic switching of render mode - I think this should be possible with the caveat that it would probably result in some flickering as the window modes are changed around. We can have a play with this one..

thanks

Awesome, the first one it's the better one for sure, but i agree with @FliplineStudios, a hybrid mode will be better to avoid issues with already running projects due to changes.

And the second option for me will resolve 90% of problems i currently have, as my application have just some part running in starling and the most majority it's old plain displayobjects that will render 100% better in GPU mode.

[bobaoapae]

can we expect something in short time about any of theses changes? @ajwfrost

[bobaoapae]

@ajwfrost just for know, any of theses solutions can be added for the next release?

[ajwfrost]

Probably not a short term change I'm afraid, we did look further at the dynamic switching of the render mode but there are comments in there suggesting that this has to be set up for a specific window at creation time. So e.g. you can create a new window using https://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display/NativeWindowInitOptions.html with the render mode there set to a different value .. although there does seem to also be a limitation that "gpu" mode is stuck, once set in the main application you can't then create a window without it. That may only be on a specific platform though.. and this option wouldn't help on mobile of course without the NativeWindow support.

Would it be an acceptable (shorter term) update if we could do a kind of "application restart" with a different render mode? A lot of the code that's used here would happen during start-up so if we did a graceful shut down of the native window and display elements, and then instead of terminating the application we restarted it with a different render mode, this may be an option we can get to?

thanks

[bobaoapae]

Yeah, for me it's acceptable to restart the application. I can make a user interface to change some display settings, that will trigger that and restart the application.

[ajwfrost]

Hi

Quick update on this case: trying to get the applications to restart is proving a little tricky, and may have to be fairly platform-dependent. We're struggling a bit with too many static variables it seems..

But can I check on the priority (@bobaoapae seeing as you were the one who opened this, and the initial premise was to see whether gpu/direct modes could perhaps be combined?) -> what platform are you using here?

From some of our investigations, it looks like: 1) it looks like the desktop platforms don't actually run in "gpu" mode, they fall back to "direct" mode if you have gpu in your app descriptor file... 2) on the mobile platforms, using "gpu" should result in an OpenGL ES (2?) context being set up 3) most of the Stage3D code would actually work with this "gpu" mode, there are some features that may not (I think on iOS it looks like gpu rendering means we only go with OGLES2, whereas direct rendering goes for OGLES3) and it looks like VideoTexture was only implemented in 'direct' contexts...

But other than that, we may be able to make a fairly quick switch to allow "gpu" mode to also use Stage3D on mobile platforms..... can I check whether that would help here? I don't really see many downsides to the change we're proposing to make, but am interested more whether that would help your particular scenarios..

thanks

[bobaoapae]

Yeah, running stage 3d within gpu render mode will help me too. I only running for mobile (android)

[bobaoapae]

Latest Air release 50.2.4.1, i can create the context3d with gpu render mode, but the screen get black after the creation of the context3d, no error from adobe air or logcat.

@ajwfrost

[bobaoapae]

Update: The context3d content are rendered, but all the classic display objects are hidden after the creation of the context3d, same application running with render mode direct works normally. Update 2: error 118: element unexpected for render mode gpu. It requires "direct" render mode. (depthAndStencil are used by starling (context 3d) , this will be supported by theses thanges or no? ) Update 3: DIsplay objects are in the stage, just don't show. Update 4: Editing the stage3d to visible false show make the classic display objects visible, so it's a problem of z-index.

[ajwfrost]

Thanks for the feedback!

A few thoughts: 1) Your first and fourth updates I think are the same? Classic display objects are displayed under the Stage3D content .. which is what I would expect. You should be able (I think?) to have a backbuffer configuration in Stage3D that allows you to not use the whole of the stage. Not sure whether Starling supports that or not though! But potentially .. I never quite understood why Adobe didn't enable transparency in the Stage3D context, there appears to be support for this from the Context3D.clear() method but I'm pretty sure that doesn't work... We could change this of course, but you'd still not be able to blend aspects of the display lists, you'd always have everything from the classic display list being shown as underneath the Stage3D content. 2) Second one we should have anticipated, sorry, that's just a configuration/validation check that we need to amend. Not sure yet on the impact on Starling so we can check that part too.. 3) Is this one about a memory leak? I wasn't quite sure what you were trying to show there...

thanks

[bobaoapae]

Strange, in my application I currently use Starling (context3d) and the classic DisplayObjects and DisplayObjects are rendered on top of stage3d. So this it's what i was expecting to do now buth with render mode gpu.

update 1,3,4 are the same, just about the problem i'm having changing from direct to gpu.

In my application using render mode direct and depthAndStencil (don't know if this last affect something), my native stage objects DisplayObjects are all rendered above the Stage3D, but just changing to gpu render mode made the display objects render under the Stage3D content.

[ajwfrost]

Ah .. maybe I'm mis-remembering that then! Yes good point, as you get things like the Starling "stats" window showing up above the Stage3D content. My recollection was based on some work we'd done for a customer a while ago where they had multiple Stage3D contexts each with transparency in order to show through the stuff underneath, but maybe that "stuff" was from their native OS rather than AIR-based display lists...

So then, if it's just a case of having to flip which one displays first, it may just be a case of re-ordering a function! We'll check this further... and thinking about it, in "direct" mode of course, all the normal display list stuff is rendered to a texture, and then that's composited last. So for gpu mode where we also have Stage3D, maybe this will need some adjustments! In Stage3D, where there's a call to clear the buffers, it may also affect the gpu-rendered content .. we'll need to investigate this more I guess.

thanks

[bobaoapae]

Here the display objects are being rendered as you can see in the update 2 of my other comment, just under the Stage3D, and the .clear of the Context3d are being called by Starling.

Also i think that the scaleX and scaleY of the native DisplayObjects are not working propelly, as hidding the Stage3D made my displayobjects visible but with wrong size.

[NikosLaskaridis]

This is a huge improvement for Adobe Air. We can now combine the amazing speed of GPU rendering in bitmap handling with all the features provided by Stage3D. Congratulations and thank you @ajwfrost !

[bobaoapae]

@ajwfrost can we expect some update on that to get a working version this year?

[bobaoapae]

Ah .. maybe I'm mis-remembering that then! Yes good point, as you get things like the Starling "stats" window showing up above the Stage3D content. My recollection was based on some work we'd done for a customer a while ago where they had multiple Stage3D contexts each with transparency in order to show through the stuff underneath, but maybe that "stuff" was from their native OS rather than AIR-based display lists...

So then, if it's just a case of having to flip which one displays first, it may just be a case of re-ordering a function! We'll check this further... and thinking about it, in "direct" mode of course, all the normal display list stuff is rendered to a texture, and then that's composited last. So for gpu mode where we also have Stage3D, maybe this will need some adjustments! In Stage3D, where there's a call to clear the buffers, it may also affect the gpu-rendered content .. we'll need to investigate this more I guess.

thanks

Is that ordering issue fixed in 50.2.4.2?

[bobaoapae]

@ajwfrost test: 50.2.4.2 depthAndStencil still not allowed in GPU mode. Stage3D content sitll rendered above stage objects.

[bobaoapae]

@ajwfrost hi, any news on that?

[bobaoapae]

@ajwfrost can this be fixed with minor versions or will be delayed to be fixed later?

[bobaoapae]

@ajwfrost hello, will this receive any new update? to fix the ordering issue when rendering stage3d using GPU mode

[bobaoapae]

@ajwfrost just asking again rsrs..

If this will be dropped just tell for me and others stop asking.

Missing things: fix the ordering issue when rendering stage3d using GPU mode, depthAndStencil not allowed, and some api to that made available change render mode without need to rebuild application again, can be after a restart without problem

[itlancer]

@ajwfrost Yes, It will be great to have some API to change renderMode at run-time. Even if app need to be restarted manually after that. It could help for some use cases. Especially when some Android devices works better with some specific renderMode. Also related to: https://github.com/airsdk/Adobe-Runtime-Support/issues/3023 https://github.com/airsdk/Adobe-Runtime-Support/issues/1980 https://github.com/airsdk/Adobe-Runtime-Support/issues/1291 https://github.com/airsdk/Adobe-Runtime-Support/issues/155 https://github.com/airsdk/Adobe-Runtime-Support/issues/103

[ajwfrost]

Hi @bobaoapae, sorry I've just seen I've not responded to any of the above messages, I hadn't realised I'd missed that many!

It's not that we're planning to drop this whole idea, it's just that it's quite complex and will take a while to work through to ensure the interactions between the GPU-rendering code and the Direct3D code aren't causing conflicts. And whether that's even possible or not, we're not 100% sure yet. So, fairly low priority and we keep having higher priority things that crop up.

Re. the idea of switching from one render mode to another, that is something that could be a bit more straightforward, and we did start to experiment along those lines. But the way AIR starts up actually makes it a little trickier than we had thought initially: the ActionScript virtual machine is initialised with internal AS3 code which does the application bootstrapping, which means that if we tried restarting from this bootstrap layer, all the existing AS3 environment would be retained. So we'd need to take it a step further back to where the native application code is first initialising the AS3 runtime. I think it's possible - we did a kind of 'auto-restart' mechanism for a customer a while back which cause any sigsegv or similar exceptions, and restarted the app again... So we should be able to make it work, but getting the links out from the AS3 code into the native apps to trigger all this are fairly complex and very platform-specific.

Maybe we could have an option to restart the app, passing a clone of the app descriptor (so that you could change different elements of it such as the rendermode), which would mean we drop out of the ActionScript VM, close the application windows, but rather than quitting the process we restart it as if the provided app descriptor had been passed in... (would have to check on the security aspects of that! or we could just allow specific overrides to various app descriptor elements, etc..)

Anyway - do keep chasing up on it, and I'll see if we can get some attention on this area again soon.

thanks

[bobaoapae]

Thanks for the feedback, for my specific case, the change on render mode can be complety manual, air api called, render mode changed and others need parameters to correspond the render mode like depthAndStencil to stage3d works as expected, and the application just close gracefully as a normal user action to exit application, this exit also don't need to be automatic, the api just need to ensure that in the new openning the render mode wil be changed, the whole process of restart can be manually.

As for the stage3d allowed on GPU render mode i have low priority on that, as this it's more to compensate to the missing way of change render mode for now.

The better solution of course will be as i already mentioned that GPU and DIRECT can be merged on some new render mode, that will allow all the things DIRECT allow but with the performance of GPU mode for native display objects, and with filters and others things working.

Also talking about filters for GPU render mode, if in harman has any easy way of implement this native, like some attribute to automatic convert things that have filter to bitmap and display the bitmap converted instead of the display object.

[bobaoapae]

@ajwfrost any progress on that?

[bobaoapae]

@ajwfrost Hello, do you have any progress on anything related to this? Improve DisplayObject render on direct mode, stage3d on GPU mode, ability to change from GPU to DIRECT mode from some AS3 or ANE call (even if the application need to be manually restarted to apply)