Vertex cache threading issues

I came across this bug while playing Valkyria Chronicles 2, but I am fairly 
certain that the issue is internal to JPCSP.  After a short period of time, 
usually coinciding with the introduction of a new 3D scene or modification to 
the number of components in the 3D scene, a variety of graphical glitches 
occur, including missing textures, gradually (and not so gradually) 
accumulating saturation to fully white screen (complete with strobing), 
gibberish in 2D overlays and fonts, and inappropriately translated blocks 
(otherwise untransformed) of the 3D display.  The problem is temporarily (a few 
seconds at best) ameliorated by invoking the "Clear Texture Cache" menu item.

If the "Enable saving GE screen to texture instead of memory" option is NOT 
checked, and the "Use vertex cache" option IS checked, the accumulation of 
glitches is fully halted by invoking the "Clear Vertex cache" menu item, and 
subsequently clearing the texture cache resolves the issue for a longer period 
of time.  However, leaving the "Enable saving GE screen to texture instead of 
memory" unchecked causes the game to run at 4 fps as long as any 3D scene is 
displayed.

Here's the weird part that makes this an internal issue with JPCSP.  One might 
reason that if clearing the vertex cache stops the accumulation of graphical 
glitches related to the texture cache, the problem would go away if only the 
vertex cache was always clear - that is, by un-checking the "Use vertex cache" 
option.  Not so.  Thus, I deduce that some *side effect* of clearing the vertex 
cache is resolving the texture issues, which may not involve corruption of the 
vertex cache at all.  This gives you a pretty good starting point - investigate 
the side effects of the procedure that clears the vertex cache.

A few additional facts lead me to believe there is a threading issue involved 
with the same procedure.  With the same settings described above, in one 
particular context, clearing the vertex cache has a roughly 50% probability of 
causing a Java exception (shown below).  Probability without variation of 
circumstance suggests an edge condition in thread interactions.  This is 
supported by the nature of the exception thrown.

Enabling "save GE to textures instead of memory" allows the game to run at 30+ 
fps, even with full anti-aliasing, and everything looks perfect.  In fact, with 
the vertex cache disabled, in roughly 20% of runs (determined apparently at or 
before loading the UMD), clearing the texture cache between changes of 3D 
scenes resolves the above problems (which still occur intermittently) for long 
periods.  However, in the remaining 80% of runs, the abnormalities are 
continuous, identical to the case above without "save GE to textures instead of 
memory" or "use vertex cache".  If both "save GE to textures instead of memory" 
and "use vertex cache" are checked, 30+ fps is maintained, but clearing the 
vertex cache has a ~90% probability of causing the same exception below 
regardless of context.

Relevant messages from the log file:

I didn't see anything abnormal that corresponded to the issue's occurrence in 
the log file, but the console stack trace was as follows:

Exception in thread "GUI" java.util.ConcurrentModificationException
        at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(Unknown Source)
        at java.util.LinkedHashMap$ValueIterator.next(Unknown Source)
        at jpcsp.graphics.VertexCache.reset(VertexCache.java:135)
        at jpcsp.graphics.VideoEngine.startUpdate(VideoEngine.java:768)
        at jpcsp.graphics.VideoEngine.update(VideoEngine.java:630)
        at jpcsp.HLE.modules150.sceDisplay$AWTGLCanvas_sceDisplay.paintGL(sceDis
play.java:226)
        at org.lwjgl.opengl.AWTGLCanvas.paint(AWTGLCanvas.java:314)
        at org.lwjgl.opengl.AWTGLCanvas.update(AWTGLCanvas.java:343)
        at sun.awt.RepaintArea.updateComponent(Unknown Source)
        at sun.awt.RepaintArea.paint(Unknown Source)
        at sun.awt.windows.WComponentPeer.handleEvent(Unknown Source)
        at java.awt.Component.dispatchEventImpl(Unknown Source)
        at java.awt.Component.dispatchEvent(Unknown Source)
        at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
        at java.awt.EventQueue.access$400(Unknown Source)
        at java.awt.EventQueue$2.run(Unknown Source)
        at java.awt.EventQueue$2.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown
Source)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown
Source)
        at java.awt.EventQueue$3.run(Unknown Source)
        at java.awt.EventQueue$3.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.security.AccessControlContext$1.doIntersectionPrivilege(Unknown
Source)
        at java.awt.EventQueue.dispatchEvent(Unknown Source)
        at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
        at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
        at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
        at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
        at java.awt.EventDispatchThread.run(Unknown Source)

Other important details:
I'm running JPCSP 2958:2964 32 on Win7 64 (for whatever reason, 64 bit JPCSP 
doesn't work at all for me).  Graphics card is integrated Intel, but runs 
modern games e.g. Mass Effect 3 just fine, so assume general compatibility.

Original issue reported on code.google.com by sqy...@gmail.com on 25 Feb 2013 at 7:51

I humbly suggest as a kluge-y start, in 
svn/trunk/src/jpcsp/graphics/VertexCache.java, line 134:

public synchronized void reset(IRenderingEngine re) {

Honestly, I'm not a java wizard, so it's possible that "synchronized" has 
different implications than I think it does.  But from the exception, issue 
conditions, and code, it looks like VertexCache::reset is for-each-ing over its 
values and deleting them without making sure some other component isn't trying 
to draw or write them at the same time in another thread.  Although, I don't 
see any non-synchronized methods (by a quick visual scan) messing with the 
cache property, and it's private, and it isn't returned by any public method, 
so I don't know how it's being accessed from some other thread.  

If that doesn't work, maybe it's returning references to the vertices and 
something else is carelessly messing with them asynchronously, in which case 
you could modify the VertexCache class to return copies, mutexed decorators 
that implement the same interface, or force synchronization in whatever class 
is already being returned.

Hope that helps!

Original comment by sqy...@gmail.com on 26 Feb 2013 at 2:53

One more thing!  In all cases in all games I've tried, checking the "Use vertex 
cache" option after the game has started running causes an immediate crash.

And another!  The above comment is intended as a temporary measure that could 
allow my cache-clearing in-game kluge to be effective while maintaining a 
playable fps (i.e. by using "enable save GE screen to texture instead of 
memory" and not throwing an exception).  This does not address the actual 
problem, which is (in my uninformed estimation) gradual, snowballing corruption 
of some GE data that affects the texture cache, whose accumulation is halted by 
a side effect of clearing the vertex cache.

Sorry, one more!  I have way too much free time!  I'm peeling through 
VideoEngine.java right now to ascertain the side effects of clearing the vertex 
cache.  Is there a way to set VideoEngine::useAsyncVertexCache = false without 
setting VideoEngine::useVertexCache = false through the JPCSP GUI?  This would 
be a pretty good test of my theory.

Original comment by sqy...@gmail.com on 26 Feb 2013 at 3:35

Thanks for the suggestion. The ConcurrentModificationException should now be 
fixed in r2987.
Can someone else reproduce the basic problem you are reporting? E.g. with 
another graphic card (to exclude driver bugs).

Original comment by gi...@web.de on 26 Feb 2013 at 8:07

Changed state: Accepted

My kluge now restores the screen to normal nearly 100% of the time and never 
crashes the game.  When it doesn't, the clear texture cache step is what screws 
it up, causing strobing that is an epilepsy hazard, or just permanently whiting 
out the screen.  This is really rare, though.

Regrettably, this update also broke a lot of the game that was working before.  
This could easily be game specific, but I mention it here for diagnostic 
purposes because there's a very specific set of changes that were applied.  A 
large number of scenes involving 2d stills with overlaid text & portraits with 
no 3d present that previously worked now halt at a predetermined point in the 
scene.  In most cases, the halt can be terminated by skipping the scene - in 
other words, the game is still processing input and can recover normally, it is 
just waiting on something that JPCSP now cannot complete.  Re-viewing the same 
scenes, where possible, does not change the point at which the halt occurs.  
This point coincides with the display of a new portrait and playing a sound 
effect (unfortunately these are inseparable) - fade effects and translations of 
graphic elements already on screen and sounds that have already started 
complete as usual.  The determinism and timing of these halts leads me to a 
hunch that JPCSP now chomps on loading or decoding some image or audio file, 
and the wait function on the IO or thread handle (assuming sync on PSP is like 
on Windows) is refusing to time out, or isn't complaining loudly about the 
failure.  I'm about to roll back to an old version to verify that the scenes 
that actually crash (silently) were working.

Original comment by sqy...@gmail.com on 10 Mar 2013 at 4:51

Rolling back the version to the one I mentioned in the OP fixes the problem, so 
one of the updates since then broke it.

In addition, I've found two cases of "now loading..." screens continuing 
indefinitely.  Both are fixed by rolling back the version.  A debug-level log 
is attached; it looks like all it's doing is managing audio sync, and there's 
no audio playing, so I'm guessing one of the updates had something to do with 
audio and is broken (at least situationally).

Original comment by sqy...@gmail.com on 12 Mar 2013 at 10:48

Attachments:

jpcsp_now_and_forever_loading.txt

EmulatorArchive / jpcsp

Vertex cache threading issues #342