Slow Saving With Lots of Files

[neogohan324]

When saving in-game, the PicoMemCard takes up to 30 seconds to fully commit the save file to the card.

Here are the current contents of my memory card, which has 1300+ memory cards pre-built, named, and indexed for all NTSC-U games: Memcards.zip

For example, I'm playing Wild Arms. When I save the file, the game saves like normal and the PicoMemcard's LED turns a light yellow-green. It stays like this for 30 seconds before turning green, meaning the save file was fully saved. If the PSX is turned off before the LED turns green, the save slot is corrupted (though the other save slots on the card are fine). Contrast this with its normal behavior, and the LED switches to yellow-green for less than a second before turning green.

I also tested in FF7, and while a lot better, it still takes a few seconds to save.

At first I thought this was a slow SD card issue as I'm using the 512MB card that came with the device. I moved everything to a known good 8GB Sandisk card, and the results were the same. I tried reformatting both cards from FAT32 to exFAT, and there was no change. I tried editing index.txt so it only contained the one game (ex: Wild Arms), and this also did not work. However, if it only has the one Wild Arms memory card (in its folder) and the one entry in index.txt, all works very quickly. Saving nearly instantly commits to the card. The PicoMemcard also instantly 'finds' the card when the xStation reports the GameID and loading is flawlessly quick. It's only saving that seems to slow down as the number of memory cards on the device increases.

It seems that the excessive number of the memory card files cause some sort of delay in saving. I've tried different filesystems (FAT32, exFAT) and different cluster sizes (32kb, 128kb) to see if it makes a difference, but it doesn't seem to. If I had to guess, there seems to be some inefficiency that perhaps makes the file save routine reassess all files during the save.

Let me know if I can provide more information. Also, feel free to call me an idiot for having 1,300 memory cards on one device. :)

[MrPalloncini]

woah, that's a lot of mcr files, unfortunately i don't think i can fix this, the code checks if the game is present in the index file, so the more games you have there, the more time it will require. Also at the moment the sorting algorithm needs to be redone from the ground up, but the more files you have, the more time it will require to sort things.

[neogohan324]

Yeah, that makes sense. What perplexed me was how it does quickly find and load the right memory card, which is what I assumed would have been the slowest operation. Once the card is found and loaded, I thought it would be fairly agnostic as to what other files are on the device since it should already be scoped to the correct card.

Thanks for taking a look! I'll pare my card down to a more reasonable number of cards and all should be well.

[MrPalloncini]

I'm currently rewriting the ordering function (more like eliminating that) so that part would be even much quicker. However i do not know why it takes so long for the sync part. My guess should probably be the FatFS standard this project uses

[neogohan324]

I've been tinkering with it for a few hours, and here's what I found.

First, I trimmed the number of cards down to 409 (the number of games actually on my xStation). No change.

I tried trimming index.txt to 200 lines, 50 lines, and then to only a single entry for Wild Arms. No change. I then deleted the number of MCR cards down to 200, then 50, then only Wild Arms. No change.

I then deleted index.txt and let it create a new card and index file from scratch, thinking it had something to do with the encoding of the index.txt file or the memory card itself. Save time was now instant.

I replaced the created memory card with a backup of my old Wild Arms memory card. Still saves instantly.

I moved the memory card into the "Wild Arms (USA)" folder and pointed index.txt at that card. Now it took 32 seconds to save again.

I moved the card from the folder to the root and renamed it "Wild Arms (USA).0.MCR", then updated index.txt to point to it. Saving was instant again.

I repopulated the PSXMemCard with my ~400 memory cards, now no longer using folders and instead [GAME NAME].0.MCR (ex: "Wild Arms (USA).0.MCR" instead of "Wild Arms (USA)\0.MCR"). Saving now takes 6-8 seconds for Wild Arms, a 4-5x improvement in speed.

Sooooo, it seems the slowness was mostly related to the use of folders. Loading from folders is snappy. Saving to them has some serious problems. But folders are mostly unnecessary, so that's OK. It does appear there's some slowness related to the number of games in index.txt, but it's nowhere near as bad if folders aren't being used.

I hope those findings help if you decide to rewrite the sorting. I do still think it's a bit strange that it can find and load an MCR file instantly but it can take seconds to save. It's totally worth a few seconds of lag to have all the functionality you added, though.

Thanks again!

[MrPalloncini]

Oh ok, actually the application doesn't support folders (i'm actually surprised you can use them), the index.txt is used by the application to find the correct memcard for that game on the root of the sd card, so... Basecally the application require you to have all the .mcr files on the root of the device, implementig subfolder would be tedious and slower

[neogohan324]

Yeah, it's strange how the folders did work, just not well. And despite the code having check to make sure that the MCR filename is entirely numeric, it accepts card names like "Wild Arms (USA).0.MCR"

Unfortunately, I've never touched C before, so I'm not much actual help in the code.

[MrPalloncini]

I guess the check was for ordering, nothing stops you to load an arbitrary memcard file by giving the name, still awesome and detailed find!