Callbacks not implemented at all correctly

[unknownbrackets]

Patapon 2 is currently broken because of this. Any game that shows "UNTESTED - waiting within callback - probably bad mojo?" is probably affected.

Currently, we have "callbacks" which are PSP kernel objects with uids (not to be confused with mipscalls.) In PPSSPP, callbacks are pretty special:

• They may be called on a thread while it is waiting/sleeping/etc. (or while not.)
• Rescheduling is not allowed while one is running.
• Wait/resume thread actions are deferred while in "callback mode."
• There's no "callback inception", you can only be one callback deep.

It turns out, this isn't exactly how they operate. In fact:

• They are called while waiting/sleeping (and also while not.)
• While a callback is being executed, it can ..WaitCB(), and future callbacks will be executed and nest, but already pending callbacks won't it seems. PPSSPP has code that handles this, actually.
• They do reschedule normally.
• If a thread is resumed or its wait deleted/canceled, this takes effect only once the callback returns.
  • For example, suppose the thread sceKernelWaitSemaCB()'s.
  • Next, the callback (or another thread rescheduled to) signals the semaphore.
  • But then, the callback calls sceKernelWaitSemaCB() itself.
  • The callback will get the semaphore, and the thread will not wake up.
  • Note that if a second-level callback deletes the semaphore being waited on, everyone wakes.
  • While inside a callback, semaphore status etc. claims no one is waiting on the semaphore.

This is pretty different. I can kind of hack it by making it so waiting within a wait will work, but it will still cause hangs if things are deleted or in some cases.

I'm not sure exactly how to implement this correctly. I suppose on the hardware, it actually stops waiting when entering a callback, calls the callback, and then once it returns calls the wait function again.

I suppose this will require c++ callbacks per wait type, and checking them when enters/exits a callback to more correctly unwait/rewait. And timeouts will have to be specially handled too...

Edit: also, I suspect that we don't currently wake a thread if the callback occurs after it starts waiting, not sure.

-[Unknown]

[hrydgard]

Nicely investigated! When I implemented the first version of the callback system I really had no idea of any of this :-)

I think whatever you do will look like a pile of hacks, and I'm sure that it looks like that in the real PSP OS too. Schedulers and synchronization are messy businesses and these strange Callback objects don't exactly make it easier...

[unknownbrackets]

It gets even weirder. Semaphores run the callbacks before waiting (e.g. even if the sema is signaled.) In contrast (and I clearly didn't test this properly), lightweight mutexes don't even call callbacks unless they wait.

Anyway, I've got Patapon 2 working now (not sure, probably other stuff is broken at the moment...) but there's definitely a lot more inconsistency with the *CB() funcs than I expected.

-[Unknown]

[hrydgard]

Wow. And as the scheduling is cooperative and not preemptive, games are able to, intentionally or not, grow dependent on every little scheduling quirk :(

[unknownbrackets]

Causing problems in delay thread in Rockband, it seems:

https://dl.dropboxusercontent.com/u/29021761/logs.7z

Should be fixable, need to just change all the waits and rip out the defer thing.

-[Unknown]

[unknownbrackets]

Figured I'd clarify this to explain essentially how they at least seem to work.

Basically, most (not all) ...CB() function can be imagined like this:

int LockCB(SceUID uid)
{
   int result;
   while (true)
   {
      auto callbacks = GetCallbacks();
      for (auto &callback: callbacks)
         callback.call();

      result = TryLock(uid);
      if (result != NEEDS_WAIT)
         return result;

      ++uid->numWaiting;
      int result = WaitChangeOrCallbacks(uid);
      --uid->numWaiting;
   }
}

In that, for example if I trigger two callbacks, then LockCB, and inside the first callback trigger a third callback, and call LockCB again from within callback 1, only callback 3 will run (2 is still pending.) Once callback 1 returns, callback 2 will run. Annoying.

Notably, LwMutexes are an exception to this rule. So far everything else seems to fit this pattern.

-[Unknown]

[hrydgard]

I guess we are closer now, thanks to all your work on this?

[unknownbrackets]

Yes, but it's still got some problems, I have changes for it but they break other things, need to get back to it...

-[Unknown]

[unknownbrackets]

@hrydgard what if we made the CPU re-entrant?

Here's what I'm thinking:

• Always use a separate cpu and gpu thread.
• When multithreading is off, make all GE stuff synchronous/block (this should have identical behavior to current.)
• When calling into a mipscall, simply re-enter jit/interp.
• While in a mipscall, hitting CORE_NEXTFRAME/CORE_STEPPING would block and wait for cpuThreadCond in the CallMIPS or w/e func instead of in CPU_RunLoop().
• CORE_ERROR / etc. would return an error, which would bail all the way out. It'd always be at a syscall exit so this would be fairly easy.
• downcount could just trigger an Advance within the callback runloop.
• Savestates would have to apply only outside any mipscall.

I actually think this would work, but maybe I'm missing something. It's not like the jit can't run inside a syscall - it doesn't really have any troublesome state anyway.

This way mipscalls and callbacks would be a lot easier. For example sceFont's callbacks would probably be fairly easy to write, callbacks would be easy to reschedule properly without lots of complexity, etc.

The only three risks I can think of are recursion (mipscalls inside of mipscalls), threading, and savestates.

I don't think recursion is a practical worry, but it would be possible for a game to enter a callback, reschedule, enter a new callback, etc. on a bunch of threads, and we'd descend deep into the stack. I don't think this would ever happen, and anyway it would eat stack space on the PSP if this happened too (although distributed on separate threads... I think we have more stack space than even all threads combined do in most any PSP game.)

I also think multithreading works everywhere, although it currently has glitches (but only from async gpu calls.)

For savestates, we currently (incorrectly) disallow rescheduling within callbacks. So I don't think any game ever spends very long inside a mipscall/callback. Currently you can savestate while stepping inside a callback, and this would be the main loss.

-[Unknown]

[hrydgard]

I'm fine with losing the ability to savestate within a mipscall, as long as we can savestate every frame I'm mostly happy enough although it's sometimes helpful for debugging to do it while stepping around.

Anyway, I think that we really, really want to have an execution mode that's 100% deterministic in the long run - especially if we want to make the TAS crowd happy with input-state movie recording/playback and so on - which is the single core mode. Running the GPU and CPU on separate threads will always introduce some non-determinism, unless we synch until the point where there's a performance loss rather than a gain from multithreading.

So unless we can find a way to allow cpu reentrance (which I'm not really against) while still timeslicing enough to make single-core emulation possible, I don't see this working out unfortunately.

[unknownbrackets]

The only time that cpu and gpu are running concurrently is after a sceGeListEnQueue(), sceGeListUpdateStallAddr(), etc. They all call:

ScheduleEvent(GPU_EVENT_PROCESS_QUEUE);

Which tells the thread to start doing stuff. If the thread is not enabled, it then immediately runs the item from the queue:

if (!threadEnabled_) {
    RunEventsUntil(0);
}

We could, instead use SyncThread here to wait for the other thread to complete:

if (!threadEnabled_) {
    RunEventsUntil(0);
} else if (threadOnlyPartiallyEnabledOrSomething) {
    SyncThread();
}

Which uses a condition variable to wait. This would have the same behavior as current, and I'm not sure it would really have some kind of significant performance loss.

-[Unknown]

[thedax]

I wonder if this is why I can't get sleep mode to work properly. I'm sending the correct power event flags to the games (pardon the extremely messy code, I'll be making it much nicer if/when it's ever ready), here at https://github.com/thedax/ppsspp/commit/63195ba47da2828eb6ce3dfa34c341464c0b426e, but most of them ignore it, and I've found only two that wake up correctly (Diva 2nd and Outrun 2006). FF Type-0 just hangs forever once it's suspended, with the bad mojo warning. I can't get the power sample at https://github.com/hrydgard/pspautotests/blob/master/demos/power.pbp to even recognize that the power switch was pressed..

[unknownbrackets]

They will typically have a thread sleeping using sceKernelThreadSleepCB(). Theoretically, sending a callback notification should wake up said thread.

But yeah, waiting during callbacks Just Doesn't Work right now.

-[Unknown]

[thedax]

Thanks for the reply. Hopefully sleep mode will be able to work once this is fixed at some point.

[unknownbrackets]

@thedax how do the power callbacks work now? Although, I think it may be a separate issue.

-[Unknown]

[thedax]

@unknownbrackets: No clue. I'll check tomorrow or Monday. I think I still have my power branch..

[thedax]

They work a little better now. I can get Type-0 to stop its music, but the game still runs all of the 3D/game input stuff, so it's still partially ignoring it. I'll open up a separate issue later, so we can try and figure out how it works.

[ghost]

f > fs/2 & fs - f.....analiasink. To use msaa in OGL ES 2.0 Where it is the share of dark colors smaller relative noise of quantization. Allow to give on an entrance of displays the modified signal. 2x,3x,4x.....

Here, on the example of it:

{ "GPU_PROFILE": { "GPU_0": { "renderers": [{"n": "Adreno 205"}, {"n": "Adreno (TM) 205"}, {"n":"NVIDIA Tegra", "-":"0xc0f"}, {"n": "PowerVR SGX 540"}], "defaultTextureFiltering": 1, "startTextureLOD": 1, "useDof": false, "useRoadSpecular": false, "useCarSpecular": false, "useParaboloidReflection": false, "useStaticParaboloidReflection": true, "useCarQualityLighting": false, "useHighQualityCars": false, "roadTextureAnisotropy": 0, "useSkidMarks": false, "useCarParticles": false, "usePerfBoost": true, "sortSolidsFrontToBack": false, "GFXOption": "VERYLOW" }, "GPU_1": { "renderers": [{"n": "NVIDIA Tegra 2"}, {"n": "Adreno (TM) 220"}, {"n": "Adreno (TM) 220"}], "defaultTextureFiltering": 1, "startTextureLOD": 1, "useDof": false, "useParaboloidReflection": false, "useStaticParaboloidReflection": true, "useCarQualityLighting": false, "useHighQualityCars": false, "roadTextureAnisotropy": 0, "usePerfBoost": true, "useTrafficCars": false, "GFXOption": "LOW" }, "GPU_2": { "renderers": [{"n": "NVIDIA Tegra 3"}, {"n": "Adreno 225"}, {"n": "Adreno (TM) 225"}], "defaultTextureFiltering": 1, "startTextureLOD": 1, "useDof": false, "allowRoadReflectionInAP": false, "usePerfBoost": true, "GFXOption": "MEDIUM", "postFX": false }, "GPU_2.5": { "renderers": [{"n": "PowerVR SGX 544"}], "defaultTextureFiltering": 1, "startTextureLOD": 1, "useDof": false, "useParaboloidReflection": false, "useStaticParaboloidReflection": true, "allowRoadReflectionInAP": false, "roadReflectionRTFrenquentReset": 50, "usePerfBoost": true, "GFXOption": "MEDIUM" }, "GPU_3": { "renderers": [{"n": "Mali-400 MP"}, {"n": "Adreno 305"}, {"n": "Adreno (TM) 305"}], "defaultTextureFiltering": 1, "startTextureLOD": 1, "useDof": false, "usePerfBoost": true, "GFXOption": "MEDIUM" }, "GPU_3.5": { "renderers": [{"n": "Adreno 320"}, {"n": "Adreno (TM) 320"}], "defaultTextureFiltering": 1, "startTextureLOD": 1, "useDof": false, "GFXOption": "MEDIUM" }, "GPU_4": { "renderers": [{"n": "Mali-T604"}], "defaultTextureFiltering": 1, "useDof": false, "GFXOption": "MEDIUM" }, "GPU_4.5": { "renderers": [{"n": "PowerVR SGX 544MP"}], "defaultTextureFiltering": 1, "roadReflectionRTFrenquentReset": 35, "GFXOption": "HIGH" }, "GPU_5": { "renderers": [{"n": "NVIDIA Tegra 4"}, {"n":"NVIDIA Tegra", "+":"0xc0f"}, {"n": "Adreno 330"}, {"n": "Adreno (TM) 330"}, {"n": "mali-t628"}], "defaultTextureFiltering": 1, "GFXOption": "HIGH" } }, "CPU_PROFILE": { "CPU_0": { "useTrafficCars": false, "useBreakables": false, "useSimplifiedCarCollisions": true, "useAICarSounds": false, "useAICarParticles": false, "useQualityPhysics": false, "useNetworkWakeupThread": false, "useAnamorphicGlows": false, "maxPlayersWhenHosting": 2, "maxTakedownPlayersWhenHosting": 4, "disablePhysicsThread": true }, "CPU_0.5": { "processors": [{"core": 2, "min": 1.0, "max": 1.2}], "useTrafficCars": false, "useBreakables": false, "useSimplifiedCarCollisions": true, "useAICarSounds": false, "useAICarParticles": false, "useQualityPhysics": false, "useNetworkWakeupThread": false, "useAnamorphicGlows": false, "disablePhysicsThread": true }, "CPU_1": { "processors": [{"core": 2, "min": 1.2}, {"core": 4, "min": 1.0, "max": 1.2}, {"core": 4, "min": 1.5, "max": 1.6}], "useTrafficCars": true, "useBreakables": true, "useSimplifiedCarCollisions": false, "useAICarSounds": false, "useAICarParticles": false, "useQualityPhysics": false, "disablePhysicsThread": true, "useNetworkWakeupThread": false }, "CPU_2": { "processors": [{"core": 4, "min": 1.2, "max": 1.5}], "useTrafficCars": true, "useBreakables": true, "useSimplifiedCarCollisions": false, "useAICarSounds": false, "useAICarParticles": false, "useQualityPhysics": true, "disablePhysicsThread": true, "useNetworkWakeupThread": false }, "CPU_3": { "processors": [{"core": 4, "min": 1.6}], "useTrafficCars": true, "useBreakables": true, "useSimplifiedCarCollisions": false, "useAICarSounds": true, "useAICarParticles": false, "useQualityPhysics": true, "disablePhysicsThread": true, "useNetworkWakeupThread": true } }, "MEM_PROFILE": { "MEM_0": { "startTextureLOD": 1, "textureBudgetMB": 80, "freeMoreMemory": true, "dropAICarLodWhenManyCarsOnScreen": true, "useTextureStreaming": false }, "MEM_1": { "memory": { "min": 512, "max": 768 }, "startTextureLOD": 1, "textureBudgetMB": 172, "freeMoreMemory": false, "dropAICarLodWhenManyCarsOnScreen": true, "useTextureStreaming": false }, "MEM_2": { "memory": { "min": 768, "max": 1024 }, "startTextureLOD": 0, "textureBudgetMB": 256, "dropAICarLodWhenManyCarsOnScreen": true, "useTextureStreaming": false }, "MEM_3": { "memory": { "min": 1024 }, "startTextureLOD": 0, "textureBudgetMB": 384, "dropAICarLodWhenManyCarsOnScreen": true, "useTextureStreaming": false } }, "RES_PROFILE": { "RES_0": { "scaleDisplay": 100, "useAAInGameplay": false, "useAAInMenu": false }, "RES_1": { "resolutions": { "minDII": 4.0, "maxDII": 7.0, "minW": 1920, "minH": 1080 }, "scaleDisplay": 70, "useAAInGameplay": false, "useAAInMenu": false }, "RES_1.5": { "resolutions": { "minDII": 6.0, "maxDII": 7.0, "minW": 1920, "minH": 1104 }, "scaleDisplay": 100, "useAAInGameplay": true, "useAAInMenu": true, "scaleAABuffer": 0.75 }, "RES_2": { "resolutions": { "minDII": 4.0, "maxDII": 8.0, "minW": 1100, "maxW": 1920, "minH": 700, "maxH": 1080 }, "scaleDisplay": 85, "useAAInGameplay": false, "useAAInMenu": false, "scaleAABuffer": -1 }, "RES_3": { "resolutions": { "minDII": 8.0, "maxDII": 8.9, "minW": 1280, "minH": 800 }, "scaleDisplay": 80, "useAAInGameplay": false, "useAAInMenu": false }, "RES_4": { "resolutions": { "minDII": 8.9, "minW": 1280, "minH": 800 }, "scaleDisplay": 60, "useAAInGameplay": false, "useAAInMenu": false, "scaleAABuffer": -1 }, "RES_5": { "scaleDisplay": 55, "useAAInGameplay": false, "useAAInMenu": false, "scaleAABuffer": -1 }, "RES_6": { "scaleDisplay": 85, "useAAInGameplay": false, "useAAInMenu": false }, "RES_7": { "resolutions": { "minDII": 9, "minW": 1280, "minH": 800 }, "scaleDisplay": 90, "useAAInGameplay": true, "useAAInMenu": true, "scaleAABuffer": 1.0 }, "RES_8": { "scaleDisplay": 60, "useAAInGameplay": false, "useAAInMenu": false }, "RES_9": { "scaleDisplay": 50, "useAAInGameplay": false, "useAAInMenu": false }, "RES_TEGRA_4_NORMAL": { "scaleDisplay": 100, "useAAInGameplay": false, "useAAInMenu": false, "scaleAABuffer": -1 }, "RES_TEGRA_4_CONSOLE": { "scaleDisplay": 85, "useAAInGameplay": false, "useAAInMenu": false, "scaleAABuffer": -1 }

},
"OPT_PROFILE":
{
    "VERYLOW":
    {
        "useShadows": false,
        "postFX": false,
        "postFXLUTOnly": false,
        "useCheapColorCorrection": false,
        "useVertexFresnel": true,
        "cutoffDistanceOverride": 800,
        "useCarDirt": false,
        "useFog": false,
        "useRoadReflection": false,
        "useMotionBlur": false,
        "useLensflare": false,
        "useQualityRoadReflection": false,
        "useGlassCrackPFX": false
    },
    "LOW":
    {
        "useShadows": false,
        "postFX": false,
        "postFXLUTOnly": false,
        "useCheapColorCorrection": true,
        "useVertexFresnel": true,
        "cutoffDistanceOverride": 800,
        "useCarDirt": true,
        "useRoadReflection": false,
        "useMotionBlur": false,
        "useQualityRoadReflection": false,
        "useGlassCrackPFX": false
    },
    "MEDIUM":
    {
        "useShadows": true,
        "postFX": true,
        "postFXLUTOnly": true,
        "useCheapColorCorrection": true,
        "useVertexFresnel": false,
        "cutoffDistanceOverride": 0,
        "useCarDirt": true,
        "useRoadReflection": true,
        "useMotionBlur": false,
        "useQualityRoadReflection": false,
        "useGlassCrackPFX": false
    },
    "HIGH":
    {
        "useShadows": true,
        "postFX": true,
        "postFXLUTOnly": false,
        "useCheapColorCorrection": false,
        "useVertexFresnel": false,
        "cutoffDistanceOverride": 0,
        "useCarDirt": true,
        "useRoadReflection": true,
        "useQualityRoadReflection": true,
        "useGlassCrackPFX": true
    }
},
"SPECIFICS":
{
    "gt-i9220|gt-n7000|gt-i9260|sc-02d|transformer prime tf201":
    {           
        "GPU": "1"
    },
    "lenovo k900_row":
    {
        "GPU": "2",
        "CPU": "1",
        "RES": "5"      
    },
    "htc butterfly|htl21|htc6435lvw":
    {
        "GPU": "2",
        "CPU": "0",
        "RES": "5",
        "MEM": "0"
    },
    "nexus 4":
    {
        "RES": "6"
    },
    "nexus 10":
    {
        "RES": "4"
    },
    "at10le-a":
    {
        "RES": "4"
    },
    "sm-p600":
    {
        "RES": "4"
    },
    "sm-g900h":
    {
        "MEM": "2",
        "RES": "9"
    },
    "sm-n900":
    {
        "RES": "8"
    },
    "redhookbay|starxtrem":
    {
        "GPU": "2",
        "CPU": "0.5"
    },
    "byt_t_ffrd10":
    {
        "GPU": "2",
        "CPU": "0.5"
    },
    "thinkpad tablet":
    {
        "RES": "2"
    },
    "isw16sh|sht21|lt22i|shv-e160s|lg-f160s|im-a840s":
    {
        "RES": "0",
        "GPU": "1"
    },
    "htc sensation z710e|htc sensation 4g|xt910|so-02d|shv-e120s|im-a760s|lt26i|gt-i9100|im-a800s|shw-m380s|adr6425lvw|a500":
    {
        "GPU": "0",
        "MEM": "0",
        "RES": "0",
        "CPU": "0"
    },
    "lt28h":
    {
        "reduceDepthFighting": true,
        "GPU": "0",
        "MEM": "0",
        "RES": "0",
        "CPU": "0"
    },
    "gt-p5100":
    {
        "reduceDepthFighting": true,
        "GPU": "0",
        "MEM": "1",
        "RES": "3",
        "CPU": "0"
    },
    "adr6400l":
    {
        "GPU": "0",
        "MEM": "0",
        "RES": "3",
        "CPU": "0"
    },
    "b1-a71":
    {
        "GPU": "0",
        "MEM": "0",
        "RES": "0",
        "CPU": "0"
    },
    "lg-d685":
    {
        "GPU": "0",
        "MEM": "0",
        "RES": "0",
        "CPU": "0"
    },
    "me172v":
    {
        "GPU": "0",
        "MEM": "0",
        "RES": "3",
        "CPU": "0"
    },
    "me173x":
    {
        "GPU": "2",
        "MEM": "1",
        "RES": "6",
        "CPU": "1"
    },
    "sc-05d":
    {
        "GPU": "1",
        "MEM": "1",
        "RES": "0",
        "CPU": "1"
    },
    "sch-i535":
    {
        "MEM": "2"
    },
    "sol21":
    {
        "GPU": "2",
        "MEM": "2",
        "RES": "0",
        "CPU": "1"
    },
    "mediapad 10 fhd":
    {
        "GPU": "1",
        "MEM": "2",
        "RES": "4",
        "CPU": "1"
    },
    "gt-p3200|gt-p3210|sm-t210|sm-t210r|sm-t211|sm-t2105|gt-p5210|gt-p5200|sm-t311|gt-n5100|gt-n5110|sgh-i467m|sm-t310|sm-t312":
    {
        "reduceDepthFighting": true,
        "GPU": "1",
        "MEM": "1",
        "RES": "0",
        "CPU": "0"
    },
    "hudl ht7s3":
    {
        "GPU": "3",
        "MEM": "1",
        "RES": "0",
        "CPU": "1"
    },
    "gt-p7500":
    {
        "GPU": "0",
        "MEM": "1",
        "RES": "3",
        "CPU": "0.5"
    },
    "gt-p5110":
    {
        "GPU": "0",
        "MEM": "0",
        "RES": "3",
        "CPU": "0"
    },
    "samsung-sgh-i957":
    {
        "GPU": "1",
        "MEM": "0",
        "RES": "0",
        "CPU": "0.5"
    },
    "gt-p1000|gt-p1010|gt-i9000|n861|lg-p920|lg-p990|htc desire|st15i|droidx|walkman|st25i":
    {
        "textureBudgetMB": 85,
        "useTextureStreaming": false,
        "GPU": "0",
        "MEM": "0",
        "RES": "3",
        "CPU": "0"
    }
}

} Base 64 https://yadi.sk/d/_TQ8XsEmVnDDM

[ghost]

Generalizations, it won't turn out all the same. .

[Bigpet]

Does anyone know what @QWEmct is talking about?

If not I'd like to post this image to describe my reaction to his last few posts

[ghost]

About that it is necessary to divide into groups. Depending on video accelerators and processor.

[Bigpet]

@QWEmct Ok, and what does that have to do with "Callbacks not implemented at all correctly" ?

[ghost]

At this message a straight line influence to a question. From this point, it is necessary to begin construction.

[Bigpet]

I can not make sense of what you are writing, please, please, please use actual English sentences.

[ghost]

unknownbrackets, I begin game. Very big ruptures of musical effects, the menu is relative, it isn't possible to move the cursor, practically. Hardly I reach to function of loading of a profile. I load a profile and problems of movement of the cursor, disappear. I can quickly move the cursor in a way of the menu. It works very quickly. Musical effects in the normal state.

Dumping in an entrance to a way of a profile ,what to kill load effects in a menu mode. ..../ "(X)_Select". ..../

---

The "transition to the menu" button to return problem, in a starting position: ...../ ..../ ..../ ..../click: 2x...

..../

---

To dump activity of loading of an operating system for the program.

[unknownbrackets]

Your comments are not related to callbacks.

I recommend you explain your problems here: http://forums.ppsspp.org/forumdisplay.php?fid=20

Ask someone else to help you get these things explained. The problem is, none of us can understand you. Sorry.

-[Unknown]

[ghost]

You gave to me the reference to a dead forum. http://forums.ppsspp.org/forumdisplay.php?fid=20

[Bigpet]

@QWEmct you can't expect people to reply to you in a matter of hours, especially if you're not posting to an international audience. But what else can we do? We can't understand you and you refuse to find someone to translate, so all we can do is link you to a place where you might find someone to help.

[ghost]

I want to help you. And then to buy the working appendix for the device.

[Bigpet]

@QWEmct We cannot understand you. Let me show how little we understand by giving you the definition to "appendix":

appendix noun 1.ANATOMY a tube-shaped sac attached to and opening into the lower end of the large intestine in humans and some other mammals. In humans the appendix is small and has no known function, but in rabbits, hares, and some other herbivores it is involved in the digestion of cellulose. "I'm merely standing in while Stella is having her appendix out"

2. a section or table of subsidiary matter at the end of a book or document. "a list of such activities was published as an appendix to the report"

[ghost]

Wow... really so everything is bad. ...) http://forums.ppsspp.org/showthread.php?tid=12354 Well here to you, it will help. Screenshots are higher. "In the environment of reproduction by the emulator, a mistake games in (monitoring)". When the program will cease to (ping) actively during game, everything will be good. Good luck.