search

hrydgard / ppsspp

A PSP emulator for Android, Windows, Mac and Linux, written in C++. Want to contribute? Join us on Discord at https://discord.gg/5NJB6dD or just send pull requests / issues. For discussion use the forums at forums.ppsspp.org.
https://www.ppsspp.org
Other
11.14k stars 2.16k forks source link

HD Remasters with more than 64MB RAM crash instantly #8268

Closed MIuIddy closed 8 years ago

MIuIddy commented 8 years ago

There is two HD Remasters, that use more than 64MB RAM: -Eiyuu Densetsu: Sora no Kiseki SC Kai HD Edition -Eiyuu Densetsu: Sora no Kiseki 3rd Kai HD Edition To make them work you need to slightly modify the code. The modifications were posted by B1ackDaemon in Issue #1494:

  1. Uncomment game strings in Core\HDRemaster.h
  2. Place decrypted exdata0 contents to memstick folder and add following code in Core\HLE\sceIo.cpp pspFileSystem.Mount("exdata0:", memstickSystem); pspFileSystem.Unmount("exdata0:", memstickSystem);

The problem occurs the moment you try to start the game.

unknownbrackets commented 8 years ago

As I recall the problem with these was that they use the same game id as the non-remaster versions. I don't have any of these remasters, but is there a way they can be reliably detected?

-[Unknown]

MIuIddy commented 8 years ago

No, currently they are not detected at all. When you apply the changes by B1ackDaemon, PPSSPP will always assume you are emulating the HD Remasters, which makes the original versions unplayable. A better detection is planed.

I did a few tests:

64MB Games: -MONSTER HUNTER PORTABLE 3rd HD Ver. - Works -Eiyuu Densetsu Sora no Kiseki FC Kai HD Edition - Works

64MB Games changed to 76MB: -MONSTER HUNTER PORTABLE 3rd HD Ver. - Works -Eiyuu Densetsu Sora no Kiseki FC Kai HD Edition - Crashes

76MB Games: -Eiyuu Densetsu: Sora no Kiseki SC Kai HD Edition - Crashes -Eiyuu Densetsu: Sora no Kiseki 3rd Kai HD Edition - Crashes

76MB Games changed to 64MB: -Eiyuu Densetsu: Sora no Kiseki SC Kai HD Edition - Works -Eiyuu Densetsu: Sora no Kiseki 3rd Kai HD Edition - Works

Error Message(Translated from German):

32bit: Exception triggered at 0x01830E2C in PPSSPPDebug.exe: 0xC0000005: Access violation while writing at position [x]. [x] is always different. Two examples: 0x19BFFC00, 0x193FFC00

64bit: Exception triggered at 0x00007FF6D1A544B9 in PPSSPPDebug64.exe: 0xC0000005: Access violation while writing at position [x]. [x] is always different. Two examples: 0x000002C38BDEFC00, 0x00000129BD53FC00

The stacktrace for the crash:

32bit: (Microsoft Visual Studio 14.0\VC\crt\src\i386)memset.asm, line 86, "rep stosb ; store the values in the destination buffer" MemMap.cpp, line 492, "memset(ptr, _iValue, _iLength);" sceKernelThread.cpp, line 425, "context.r[MIPS_REG_SP] = currentStack.start + nt.stackSize;"

64bit: (Microsoft Visual Studio 14.0\VC\crt\src\amd64)memset.asm, line 80, "rep stosb ; store the bytes" MemMap.cpp, line 492, "memset(ptr, _iValue, _iLength);" sceKernelThread.cpp, line 425, "context.r[MIPS_REG_SP] = currentStack.start + nt.stackSize;"

unknownbrackets commented 8 years ago

Right. I know they are currently not detected. What I'm asking is, is there any way to detect them reliably? Is there some "THIS_IS_A_REMASTER.DAT" file on the ISO? Are the ISOs larger than 2GB? I don't have them, so I can't check myself.

If we can detect them, we can fix this.

-[Unknown]

MIuIddy commented 8 years ago

In HDRemaster.h it says: TODO: Use UMD_DATA.bin to differentiate the Eiyuu games from the regular PSP editions.

This method is already used by Jpcsp. They look for a certain string in that file:

Eiyuu Densetsu: Sora no Kiseki FC Kai HD Edition: "ULJM-05170|55C069C631B22685|0001|G |"

Eiyuu Densetsu: Sora no Kiseki SC Kai HD Edition: "ULJM-05277|0E8D71AFAA4F62D8|0001|G |"

Eiyuu Densetsu: Sora no Kiseki 3rd Kai HD Edition: "ULJM-05353|0061DA67EBD6B9C6|0001|G |"

unknownbrackets commented 8 years ago

I wonder if UMD_DATA.bin is really the best way.

I've recently added a better way to log SFO data. I'd just like to confirm what the SFO contains. In Core/PSPLoaders.cpp, can you find:

            // TODO: Check the SFO for other parameters that might be useful for identifying?
            gameID = g_paramSFO.GetValueString("DISC_ID");

And then, add after it:

            std::vector<std::string> keys = g_paramSFO.GetKeys();
            for (std::string key : keys) {
                NOTICE_LOG(HLE, "SFO: %s = %d / %s", key.c_str(), g_paramSFO.GetValueInt(key), g_paramSFO.GetValueString(key).c_str());
            }

Just to be sure if there's anything useful in the SFO that we can tell things are a remaster without a whitelist. That's definitely be my preference if it's at all possible. I'm even wondering if there might be a flag on the ELF or something indicating the increased memory size requested.

This because a future HD remaster is probably unlikely, but not impossible at all.

I suppose we can just always map exdata0 when a remaster is detected. Should it really be the same as memstick, or would it be better to be a "exdata/" directory? I'm not sure I understand what this contains.

If we have to use the UMD_DATA.bin file, I guess we'd just add the entire content to the whitelist. I'll be honest: I don't know what any of those fields mean. I feel like the second field is probably some sort of hash of the ISO.

-[Unknown]

MIuIddy commented 8 years ago

I used Eiyuu Densetsu: Sora no Kiseki FC for this:

Normal version:

56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: BOOTABLE = 1 / 
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: CATEGORY = 0 / UG
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_ID = 0 / ULJM05170
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_NUMBER = 1 / 
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_TOTAL = 1 / 
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_VERSION = 0 / 1.01
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: PARENTAL_LEVEL = 1 / 
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: PSP_SYSTEM_VER = 0 / 2.71
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: REGION = 32768 / 
56:50:579 PSPLoaders.cpp:115 N[HLE]: SFO: TITLE = 0 / 英雄伝説 空の軌跡ＦＣ

HD version:

58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: APP_VER = 0 / 01.00
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: BOOTABLE = 1 / 
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: CATEGORY = 0 / UG
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_ID = 0 / ULJM05170
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_NUMBER = 1 / 
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_TOTAL = 1 / 
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: DISC_VERSION = 0 / 1.02
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: HRKGMP_VER = 0 / 
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: PARENTAL_LEVEL = 1 / 
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: PSP_SYSTEM_VER = 0 / 6.60
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: REGION = 32768 / 
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: TITLE = 0 / 英雄伝説　空の軌跡ＦＣ
58:33:237 PSPLoaders.cpp:115 N[HLE]: SFO: USE_USB = 0 /

There doesn't seem to be any distinct indicator for HD Remasters. The .pkg for the PS3 contains it's own PARAM.SFO, which has a distinct CATEGORY for HD Remasters: "PE" I ran that file through your code and got this:

26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: APP_VER = 0 / 01.00
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: ATTRIBUTE = 4096 / 
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: BOOTABLE = 1 / 
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: CATEGORY = 0 / PE
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: PARENTAL_LEVEL = 1 / 
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: PS3_SYSTEM_VER = 0 / 03.6500
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: RESOLUTION = 63 / 
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: SOUND_FORMAT = 1 / 
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: TITLE = 0 / 英雄伝説　空の軌跡ＦＣ
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: TITLE_ID = 0 / ULJM05170
26:48:327 PSPLoaders.cpp:117 N[HLE]: SFO: VERSION = 0 / 01.02

There doesn't seem to be any parameter for setting the RAM size, though.

About the exdate folder: I guess it is for games the are bigger than 1.4GB.

The PS3 .pkg contains the the files: (Folder) USRDIR ICON0.PNG PARAM.SFO PIC0.PNG PIC1.PNG PIC2.PNG PS3LOGO.DAT

USRDIR contains: (Folder) CONTENT (Folder) EXDATA ISO.BIN.EDAT MINIS.EDAT MINIS2.EDAT

CONTENT contains: DOCUMENT.DAT EBOOT.PBP

In EXDATA there is one folder, that is called "data" in FC and SC, but "data_3rd" for 3rd. This folder can also be found in the USRDIR folder of the original version of the games and probably just contains files that were moved there to reduce the size of the EBOOT.PBP.

It would be very useful if PPSSPP would use different exdate0 folder for each game, because there are some overlaps with the files from FC and SC.

unknownbrackets commented 8 years ago

So perhaps we could use exdata/ULJM05170/data/... for the path? That doesn't seem unreasonable. Alternatively, we could expect that EBOOT.PBP structure - and try the EBOOT directory + "/../EXDATA/". Although, there might be some minor trickiness to that.

Sounds like there's no forward-looking method. I guess we can just add the whitelisting...

-[Unknown]

unknownbrackets commented 8 years ago

8314 should do that, if you'd like to try merging it locally.

-[Unknown]

MIuIddy commented 8 years ago

Your detection seems to be working fine. The current build, even without your changes, crashes with all three games now, though.

unknownbrackets commented 8 years ago

Have a log or stack trace of the crash? Might still be that sceMp4 crash or something from #1494. If so, might close this since the detection is fixed and keep it there, or else create a new issue for what remains.

-[Unknown]

MIuIddy commented 8 years ago

The original issue was, that the Kiseki games crashed with an access violation, when they had more than 64MB RAM. Now they also crash with the same error with just 64MB. I'd keep this issue open until that is solved.

log:

50:12:631 EmuThread.cpp:160 I[BOOT]: Done.
50:15:558 Config.cpp:1250 I[LOAD]: Failed to read D:\prog\ppsspp\memstick/PSP/SYSTEM/ULJM05170_ppsspp.ini. No game-specific settings found, using global defaults.
50:15:558 System.cpp:376 I[BOOT]: PPSSPP v0.9.9.1-5083-g5a702e4 Windows 64 bit
50:15:558 MemMap.cpp:338 I[MM]: Memory system initialized. RAM at 0000021A733A0000 (mirror at 0 @ 0000021A7E370000, uncached @ 0000021ABE370000)
50:15:558 Loaders.cpp:208 I[LOAD]: File is a PBP in a directory!
50:15:572 PSPLoaders.cpp:135 I[LOAD]: HDRemaster found, using increased memory
50:15:573 PSPLoaders.cpp:195 I[LOAD]: ULJM05170 : 英雄伝説　空の軌跡ＦＣ
50:15:574 Config.cpp:1250 I[LOAD]: Failed to read D:\prog\ppsspp\memstick/PSP/SYSTEM/ULJM05170_ppsspp.ini. No game-specific settings found, using global defaults.
50:15:574 PSPLoaders.cpp:237 I[LOAD]: Loading disc0:/PSP_GAME/SYSDIR/EBOOT.BIN...
50:15:574 HLE\sceKernelMemory.cpp:436 I[KERNEL]: Kernel and user memory pools initialized
50:15:574 HLE\sceIo.cpp:519 I[IO]: Mounted exdata/ULJM05170/ under memstick for exdata0:/
50:15:594 HLE\sceKernel.cpp:152 I[KERNEL]: Kernel initialized.
50:15:632 Util\BlockAllocator.cpp:391 I[HLE]: -----------
50:15:632 Util\BlockAllocator.cpp:395 I[HLE]: Block: 08800000 - 08804000 size 00004000 taken=1 tag=usersystemlib
50:15:632 Util\BlockAllocator.cpp:395 I[HLE]: Block: 08804000 - 08a33000 size 0022f000 taken=1 tag=ELF/kernel
50:15:632 Util\BlockAllocator.cpp:395 I[HLE]: Block: 08a33000 - 0c000000 size 035cd000 taken=0 tag=(untitled)
50:15:632 Util\BlockAllocator.cpp:397 I[HLE]: -----------
50:15:657 HLE\ReplaceTables.cpp:1198 I[HLE]: Replaced memcpy at 089bfa80 with hash 1a7564fa3e25c992
50:15:657 HLE\sceKernelModule.cpp:1035 I[LOAD]: Module kernel: 08a16fc0 089d6964 089d697c
50:15:658 HLE\sceKernelModule.cpp:1236 I[LOAD]: Exporting ent 0 named kernel, 2 funcs, 4 vars, resident 089d6d44
50:15:658 HLE\sceKernelModule.cpp:1558 I[LOAD]: Module entry: 08804108

stacktrace:

PPSSPPDebug64.exe!memset() Line 80  Unknown
PPSSPPDebug64.exe!Memory::Memset(const unsigned int _Address, const unsigned char _iValue, const unsigned int _iLength) Line 506    C++
PPSSPPDebug64.exe!Thread::FillStack() Line 425  C++
PPSSPPDebug64.exe!__KernelResetThread(Thread * t, int lowestPriority) Line 1809 C++
PPSSPPDebug64.exe!__KernelSetupRootThread(int moduleID, int args, const char * argp, int prio, int stacksize, int attr) Line 1867   C++
PPSSPPDebug64.exe!__KernelStartModule(Module * m, int args, const char * argp, SceKernelSMOption * options) Line 1452   C++
PPSSPPDebug64.exe!__KernelLoadExec(const char * filename, unsigned int paramPtr, std::basic_string<char,std::char_traits<char>,std::allocator<char> > * error_string) Line 1584 C++
PPSSPPDebug64.exe!Load_PSP_ISO(FileLoader * fileLoader, std::basic_string<char,std::char_traits<char>,std::allocator<char> > * error_string) Line 238   C++
PPSSPPDebug64.exe!LoadFile(FileLoader * * fileLoaderPtr, std::basic_string<char,std::char_traits<char>,std::allocator<char> > * error_string) Line 213  C++
PPSSPPDebug64.exe!CPU_Init() Line 249   C++
PPSSPPDebug64.exe!PSP_InitStart(const CoreParameter & coreParam, std::basic_string<char,std::char_traits<char>,std::allocator<char> > * error_string) Line 396  C++
PPSSPPDebug64.exe!EmuScreen::bootGame(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & filename) Line 125   C++
PPSSPPDebug64.exe!EmuScreen::update(InputState & input) Line 682    C++
PPSSPPDebug64.exe!ScreenManager::update(InputState & input) Line 47 C++
PPSSPPDebug64.exe!NativeUpdate(InputState & input) Line 766 C++
PPSSPPDebug64.exe!UpdateRunLoop() Line 161  C++
PPSSPPDebug64.exe!Core_RunLoop() Line 193   C++
PPSSPPDebug64.exe!Core_Run() Line 268   C++
PPSSPPDebug64.exe!TheThread(void * __formal) Line 183   C++
PPSSPPDebug64.exe!invoke_thread_procedure(unsigned int(*)(void *) procedure, void * const context) Line 92  C++
PPSSPPDebug64.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>(void * const parameter) Line 115    C++
[Externer Code]
unknownbrackets commented 8 years ago

Hmm, that seems like it's not mapping the memory correctly. I see the problem, it's because it's recognizing the PBP first. Probably the other was like an extracted ISO from the PBP... okay.

If you take this function:

void InitMemoryForGamePBP(FileLoader *fileLoader) {

And change the entire function to:

void InitMemoryForGamePBP(FileLoader *fileLoader) {
    if (!fileLoader->Exists()) {
        return;
    }

    PBPReader pbp(fileLoader);
    if (pbp.IsValid() && !pbp.IsELF()) {
        std::vector<u8> sfoData;
        if (pbp.GetSubFile(PBP_PARAM_SFO, &sfoData)) {
            ParamSFOData paramSFO;
            if (paramSFO.ReadSFO(sfoData)) {
                std::vector<std::string> keys = paramSFO.GetKeys();
                for (std::string key : keys) {
                    NOTICE_LOG(HLE, "SFO: %s = %d / %s", key.c_str(), paramSFO.GetValueInt(key), paramSFO.GetValueString(key).c_str());
                }
                // This is the parameter CFW uses to determine homebrew wants the full 64MB.
                UseLargeMem(paramSFO.GetValueInt("MEMSIZE"));
            }
        }
    }
}

Does it show anything useful there? The trouble is I don't directly have access to the ISO there. Maybe I can change it to dereference the PBP earlier.

Oh, new idea. In Core/Loaders.cpp, try replacing this:

                if (ebootType == FILETYPE_PSP_ISO_NP) {
                    InitMemoryForGameISO(fileLoader);

With:

                if (ebootType == FILETYPE_PSP_ISO_NP) {
                    Memory::Shutdown();
                    InitMemoryForGameISO(fileLoader);
                    Memory::Init();

But that's a bit ugly. Just want to know if it works to do that.

-[Unknown]

MIuIddy commented 8 years ago

The first change doesn't add anything useful to the log. The second change fixes the crash.

unknownbrackets commented 8 years ago

I've updated the pull request with a change that should, I hope, properly read the SFO before initializing memory, in the case of a PBP.

-[Unknown]

MIuIddy commented 8 years ago

I tested the change and it seems to run fine, too.