Rendering images in background thread

[GoogleCodeExporter]

Currently rendering to a buffered image needs to be done on the EDT.  It would 
be really handy to be able to render images on a background thread so that the 
UI doesn't lock up while you're rendering something very complex.  

I'm pretty sure that this would be an easy change on iOS and Android.  I don't 
know about other platforms.  Do you have any strong feelings on this either 
way?  Are there any road blocks you can think of that I might not be 
considering?

Original issue reported on code.google.com by st...@weblite.ca on 21 Jan 2014 at 5:34

[GoogleCodeExporter]

We don't block this, we might produce EDT warnings in the EDT debug mode but we 
don't explicitly block it.
Since its literally impossible to guarantee thread safety on a large API we 
probably won't "officially" support rendering from a separate thread. But as 
far as I know we don't block it and it should work.

Notice that on iOS you might have some race conditions if you do that and those 
aren't simple to fix since the mutable image implementation relies on a global 
state within the OS.

Original comment by shai.almog on 21 Jan 2014 at 6:47

[GoogleCodeExporter]

On iOS it doesn't seem to work.  I get a black image.  This is likely due to 
the following reasons and possibly more:

1. Only one mutable image can be edited at a time (i.e. it stores the context 
in the variable:
[CodenameOne_GLViewController instance].currentMutableImage

This creates race conditions.

2. On iOS each thread has its own CGContext stack.  This means that if we 
render to an image context in a background thread, this context won't be 
available if we try to close it on the EDT.  I.e. the finishDrawingOnImageImp 
needs to be called on the same thread where startDrawingOnImageImpl was called. 

I have tried to work around these problems by:

1. Store the currentMutableImage in the current thread's thread dictionary 
instead of as an instance variable of the view controller.
2. Creating an RGBImage with the mutable image in the background thread and 
trying to pass that back to the EDT to then render with drawImage.

I'm still getting black screen so I'm digging around trying to see what I can 
do.

Personally I think this is quite an important use case for usability.  E.g. I 
have ported over achartengine, and it works great!  But most charts take 
between 200ms to 1.5s to render on an iPhone4S.  If rendered on the EDT it will 
lock up transitions and it looks bad.  The same situation will come up when I 
try to port over libraries like MapForge that need to render map tiles on the 
fly.

Hence, it would be nice if this could be officially supported - even if that is 
in a limited way.

On a side note, in researching this about this, I have read that UIKit has been 
thread safe since iOS 4.  Each thread has its own context stack.  Therefore I 
think we could probably safely remove a lot of dispatch_sync() blocks, and just 
interact with UIKit directly on the EDT.  This could potentially yield some 
performance gains.  Just a thought.

Original comment by st...@weblite.ca on 21 Jan 2014 at 9:57

[GoogleCodeExporter]

UIKit isn't threadsafe, I added all of those dispatch calls to address 
failures. I would assume the low level graphics (cg stuff) is threadsafe but 
the UI kit is very sensitive to threading issues.
I see the use case for this but as you found out, its not trivial to fix. 
Mutable images are pretty painful.

What I don't understand is why do you use mutable images?
You create RGB data from picses right?
You can do that on a separate thread since there is no rendering code involved, 
then its just a matter of creating an image from an array of integers which 
isn't a mutable image.

Original comment by shai.almog on 22 Jan 2014 at 5:52

[GoogleCodeExporter]

I'm using cn1 graphics as my primary pipeline and just relying on Pisces to
fill in gaps (eg paths and transforms).  For the things that cn1 graphics
can do I find it to be faster than Pisces although I haven't done any
rigorous head to head comparisons.  Perhaps I'll modify it to only use
Pisces to see how it compares.  That would at least solve the background
rendering.

In this project I've hit quite a few little graphics bugs in iOS and
javase.  I'll post each of them once I have done some more experimenting.

Original comment by st...@weblite.ca on 22 Jan 2014 at 6:08

[GoogleCodeExporter]

We do need serious overhaul of the native graphics code, I'm optimistic we will 
find the budget for that ;-)

Original comment by shai.almog on 22 Jan 2014 at 6:22

[GoogleCodeExporter]

Yesterday I spent some time experimenting with changing the pipeline to es2
using a compatibility layer.  I'm done except I just get a black screen ..
Lol.  I could just be a few tweaks away.  Or it could be the beginning of
some lengthy painful debugging.  If I get it working I'll let you know.

I'm really trying to make sure the tool kit feels performant.  For
rendering, the ideal is just to have a very fast pipeline and do everything
on the edt...  (hence the es2 experimenting).  Failing that (or in
addition) background rendering will at least allow me to fool users into
thing it's a fast pipeline.

Original comment by st...@weblite.ca on 22 Jan 2014 at 6:37

[GoogleCodeExporter]

I agree, I hope we can make that happen.

Original comment by shai.almog on 22 Jan 2014 at 8:34