MemoryCard saves per-game

[SeventySixx]

Panic! My memory card is almost full! ;-)

I think the next step to improve this core is to create a different save for each game. So it will not needed anymore to bother about memcard managing.

I'm going to look about it, the idea is that we will have in Retroarch/saves/pcsx2 folder a memory card for each game. Saves then will take 8Mb of space, a little waste, but nothing compared to the space taken by saves of modern games.

[SeventySixx]

I think at the beginning I will do also an option to disable per-game saves, because I don't know how behave multi-disc game (but there are any for PS2?)

[SeventySixx]

I can get the RetroArch\saves directory with:

environ_cb(RETRO_ENVIRONMENT_GET_SAVE_DIRECTORY, &save_dir);

but to create the pcsx2 subfolder, is a thing that I have to take care at the core init (check if the folder exists, otherwise create it), or it's a task delegated to RetroArch, maybe when it install the core?

[covey-j]

I think at the beginning I will do also an option to disable per-game saves, because I don't know how behave multi-disc game (but there are any for PS2?)

Yes, there are a handful of multi-disc games for PS2, e.g. Devil May Cry 2 and Star Ocean: Till the End of Time. That's not a huge concern, though, since the core already supports .m3u playlists, so you'd use the same memory card for both discs.

The bigger concern would be games that read your memory card for save data from other games, e.g. there's a vendor in Ratchet & Clank: Going Commando that gives you free weapons from Ratchet & Clank if you have a Ratchet & Clank save file.

I see a couple of options here, with pros and cons for each:

Option 1: Create per-game memory cards. These go in slot 1. Slot 2 will have a shared memory card. Pro:

• The typical user won't have to worry about memory card management as it's managed for them
• Closer to how other libretro cores handle save games

Con:

• If the user wants to handle the situation I mentioned with Ratchet & Clank: Going Commando without using an external program, they'd have to make sure they used the shared second memory card or else they'd have to rely on either in-game saving to switch slot or the built-in memory card tool of the PS2 bios to transfer things over. The latter option is very cumbersome since the process would be something like loading content fast boot disabled, going into disc control, ejecting the disc, and restarting the content (currently broken)
• Current users of the core will need to migrate their current save games, which might cause confusion
• Wastes a bit of storage space
• Users who run out of storage in the shared memory card might have to resort to using the PS2 bios to clear out space

Option 2: Create something like a Manage Memory Cards option that lets you create/delete/copy/rename memory cards and select which memory card files get loaded into slots 1 and 2 Pro:

• Comprehensively addresses edge cases like the one I mentioned above
• Consistent with the current system, so current users won't have to do any save migration
• Less storage waste compared to option 1
• If the user needs to use the memory card utility of the PS2 bios for some reason, they just need to load core, disable fast boot, and start core (much less complicated than the process for option 1)

Con:

• Users that play enough games to fill their default memory cards will have to either use the PS2 bios to delete saves or navigate to Manage Memory Cards to create more memory cards

[covey-j]

Personally, I'm partial to option 2, having been a user for similar approaches in PS1 emulators. Option 2 is similar to what Sony did with PS1 emulation on PS3. Option 1 is similar to what the pcsx-rearmed core does IIRC, and it was nice until I had to figure out a complicated disc control workaround to move save files around so that I could play Yu-Gi-Oh! Forbidden Memories in two player.

[covey-j]

Update: The libretro API doesn't support nesting of core options, so option 2 wouldn't work too well. Here's what I propose instead as a sort of "best of both worlds":

1. Create two new core options: Memory Card Slot 1 and Memory Card Slot 2.
2. The first two option values for these will be Empty, which corresponds to having no memory card inserted in the slot and Game-Specific Memory Card, which uses a memory card from the saves/pcsx2 directory with the same name as the loaded content. If no such memory card exists, create it automatically.
3. Designate a directory for shared memory cards. I'd recommend something like saves/pcsx2/shared_memcards. For convenience, we can generate two (or more) of these automatically during first run (for clarity, we should call them something like Shared Memory Card 1.ps2 and Shared Memory Card 2.ps2) if the shared_memcards directory doesn't already contain files of the same name.
4. During retro_init, scan the shared_memcards directory for valid memory cards and append their file names (without extension) to the list of option values for Memory Card Slot 1 and Memory Card Slot 2. Note: we need to impose checks on this so that we don't exceed the limit on the number of option values the API can handle.
5. Impose handling to ensure that the user can't put the same memory card in two different slots. Suppose Card A is in Slot 1, Card B is in Slot 2, and the user attempts to select Card A for Slot 2. I propose we handle this by setting Memory Card Slot 2 to Card A and Memory Card Slot 1 to Empty.

Thoughts on this proposal?

[SeventySixx]

Thanks for your feedback!

Yes, that could works, I have already done and tested the "Save per content" and the memcard creation if still not existing, and it works well. I will look to follow your suggestions.

For the point 5, I have a question. How can I handle the options at run time? If user selects the same MemoryCard for slot 2, how can I change the option of slot 1? Because I had looked for this in when implementing other options in the past but I couldn't figure how to do it. Maybe you know a core which uses this trick which I could look at?

[covey-j]

Not sure off the top of my head. I'll look into it later today and get back to you.

[covey-j]

Hey @twinaphex, what are your thoughts on this? Is something like what I laid out in the above proposal feasible within the API?

[SeventySixx]

Not sure off the top of my head. I'll look into it later today and get back to you.

At the moment for this case I'll go with unloading at runtime the same memcard on the slot 2 and sending also a notification to the frontend, so the user is informed about the 'mistake'

[alondero]

Would a solution like the native 'Memory Card Folder' feature of PCSX2 standalone help here? I'm assuming that, at runtime, creates a virtual memory card with the files from the matching game's folder, thus it never gets full. Potentially there could be some configuration somewhere to include other games memory card folders on load for the Ratchet & Clank scenario above?

[SeventySixx]

Ok, I did the job, before making a PR maybe could be tested from my fork There are 3 MemCard modes:

• Per-game: if shared memcard are both on "Empty", this mode is used automatically , a memcard (named like the content filename) is created under saves/pcsx2. The content then will use that.
• Shared memcards: 9 shared memcards (maybe are too much?) are automatically created under saves/pcsx2/shared_memcards, and will be used if selected in slot1 and/or slot2. If same memcards are selected on both slots, the slot2 is disabled.
• Legacy: this option can be used to access the 'original' memory cards under system/pcsx2/memcards, useful if someone has old savegames made before this update.

I added also an option "Boot to Bios", so memcards associated to the current content can be easily managed.

When starting the content the user will receive always a notification about the current memcard mode, and for the mode "shared memcard" a notification is sent when the slot 2 has been disabled due to memcard conflict.

[covey-j]

Ok, I did the job, before making a PR maybe could be tested from my fork

Yeah, that or you could open a draft pull request. I'm interested to look at it either way!

[SeventySixx]

Ok, draft PR done :-)

[covey-j]

Would a solution like the native 'Memory Card Folder' feature of PCSX2 standalone help here? I'm assuming that, at runtime, creates a virtual memory card with the files from the matching game's folder, thus it never gets full. Potentially there could be some configuration somewhere to include other games memory card folders on load for the Ratchet & Clank scenario above?

I'm not aware of such a feature in the standalone and am having difficulty finding info on this. This is an interesting idea, though. One could theoretically store just the save data rather than a whole memory card and construct the memory card at runtime. There are probably some situations where having a persistent memory card would be more appropriate, though.

Edit: Nvm, found it! Thanks for pointing this out!

[SeventySixx]

So, actually the new options system about memcards, based on 2 different slots, has been pushed on master - thanks to @covey-j for feedbacks, suggestions and his big contibute to this update 👍

Unfortunately the 'Per-Game Saves' option, based on the MemoryCardFolder feature of the standalone, has been disabled because it made hanging the core while saving, more precisely when writing on disk. Not really a blocking problem for games, but there was choppy performances. Also on some games seems not working well, corrupting saves on these particular games. So was better to put this feature aside for the moment.

The memory card management has been also updated to the latest version of the standalone, and "seems" sightly quick to load and save on memcard.... but maybe that's only psychological to me :-)

The new update introduces a new memory card management:

Added options to manage memcards on slots 1 and 2.

the options are:

Slot 1

• Empty - No memory card loaded
• Legacy - points to original location of pcsx2 memcard Mcd001, if available (system/pcsx2/memcards)
• Shared Memory Card (8 MB) - a classic 8MB memory card
• Shared Memory Card (32 MB) - a classic 32MB memory card

Slot 1

• Empty - No memory card loaded
• Legacy - points to original location of pcsx2 memcard Mcd002, if available
• Shared Memory Card (8 MB) - classic 8MB memory card
• Shared Memory Card (32 MB) - classic 32MB memory card

Shared Memcards are created only when the relative option is selected, so space is saved until are really needed.

Has been added also a "Boot to Bios" option, so memcards associated to the current content can be easily managed.

The core now creates automatically a new folder structure under retrtoarch/saves, if not already present, where it stores all memory cards generated by the options.

The files and folder struture on the filesystem is the following:

. ├── ... ├── saves │ ├── pcsx2 │ │ ├── Slot 1 │ │ │ ├── Shared Memory Card (8 MB).ps2 │ │ │ └── Shared Memory Card (32 MB).ps2 │ │ │ └── .ps2 │ │ │ └── .... │ │ └── Slot 2 │ │ │ ├── Shared Memory Card (8 MB).ps2 │ │ │ └── Shared Memory Card (32 MB).ps2 │ │ │ └── <User Memcard 2>.ps2 │ │ │ └── <User Memcard 3>.ps2 │ │ │ └── .... │ └── ...
└── ...

If the user put another memory card file in the slot1 or slot2 folder, it will be visible in the options. This allows for example to put there old memory cards from the standalone, making easier to tranfer files. Or when a shared memory card is full, the user can rename the file. In this way the new renamed card will be selectable in the option list, and the core can eventually generate a new shared memory card. In each slot folder can be put up until 20 custom memcards files.

about this issue ,now it's half resolved, it still missing:

• switch of memcards at runtime: , I looked about it, I spent some hours but is not so easy as changing simply the path to the file: the changed memcard slot must be closed and reopened, and of course when doing it the memcard should not be written, if in that moment something is writing on the memcards, it's necessary waiting until finishes. This is relatively not so difficult to implement, the difficult thing it's to access from the libretro side to the instances of the memory cards objects. The intricated code of pcsx2 won't help in this :-) I will make some other tries in the next times.

• Implementation of the Folder saves, I suppose this time based on retro_get_memory_data/retro_set_memory_data, or understand why they are so choppy on the core. On the standalone this feature work well, and the code is the same.

[SeventySixx]

@covey-j I had a look on the Per-Game Saves problem, by disabling the FolderAutoManager option seems a little bit better in terms of save times, but doesn't solve the core hanging problem. Interesting thing I found from the tests is that isn't the I/O on the disk which generates the problem, but it seems to happens when the core prepares the save data in memory. This is quite strange because the folder memcards managing code is exactly the same of the standalone....

[covey-j]

@SeventySixx The difference here between the core and standalone is the threading. The bottleneck isn't the disk. It's synchronization. Here's a quick rundown:

The normal memory card files are just binary blobs, same as if you did a raw dump of an actual PS2 memory card. The virtual machine reads in the blob and says, "Hey, this is a PS2 memory card. I'm a PS2, so I know how to read from this and write to this!" When a game is saved, the VM writes the files to the memory card (including timestamp data), same as a PS2 does. The host machine doesn't interact at all with the guest filesystem; it just writes overwrites the old binary blob with a new binary blob.

The folder-style memory cards are not binary blobs; they store the actual save files (the ones a PS2 sees when it reads a memory card) in the host machine's filesystem. One repercussion of this is that the timestamps in the host filesystem are the same timestamps that the VM has to read. Things can go wrong here if they're not the same as what the VM would have made using the standard memory card file approach. PCSX2 handles this by using wxDateTime to correctly timestamp the files.

IIRC, if you attach a debugger and look at a parallel stack during the save, you'll see the main thread fighting with the EE Core thread over something related to this wxDateTime (I don't recall what exactly). You see a hang because, when the main thread is stuck waiting for a resource to be unlocked (in this case, one of the resources related to wxDateTime), the program becomes unresponsive until the resource is freed. This, btw, is roughly the same thing that goes on when the libretro core tries to read in widescreen patches from the zip file.

[SeventySixx]

Yeah, you're right, it's wxDateTime. So for the moment nothing to do, until the thread problem will be fixed. From what I read around also the savestate has been disabled because this threading issue.

[blackwind]

Are the blockers that prevented per-game saves still present? Getting back into PS2 emulation and all my saves are in the folder format. 😕

[Rolen47]

Are the blockers that prevented per-game saves still present? Getting back into PS2 emulation and all my saves are in the folder format. 😕

Currently the core only uses *.ps2 files for game saves. You'll have to convert them.

[blackwind]

I'm aware, but was hoping @SeventySixx or @covey-j might take another look given the current state of the project.