Auto SRAM saving

[Nebuleon]

This is an enhancement request I'm posting on my own behalf, but it was also posted on GBATemp starting at this post: http://gbatemp.net/threads/gameyob-a-gameboy-emulator-for-ds.343407/page-4#post-4564584

-- Quote: Drenn, 2013-02-24

I believe snemulDS saves automatically, though I'm not sure how it does this. Sram is used for more than just saving, for instance Wario Land 3 stores the level layout in there. So saving whenever something writes to sram would cause annoying pauses whenever a game writes to it for any purpose. I know pressing X is annoying, but I can't think of a better way.

-- Quote: Nebuleon, 2013-02-24

[In SNEmulDS, a]t the end of a frame, if the SRAM has been modified, an autosave is triggered. In your case, if the SRAM is slow and isn't written fully in one frame, maybe have it so that: At the end of a frame, if the SRAM has been modified, defer autosaving until the first subsequent frame during which the SRAM is not modified.

Sram is used for more than just saving, for instance Wario Land 3 stores the level layout in there. So saving whenever something writes to sram would cause annoying pauses whenever a game writes to it for any purpose. I know pressing X is annoying, but I can't think of a better way.

To alleviate all those "microSD saving timing issues" which would cause corruption, maybe have a little microSD access icon on the Touch Screen whenever a microSD write is ongoing? It would also help by telling the user that a certain pause is caused by microSD access.

Also an autosave option, which would be on by default, but the user could see how often the microSD icon is brought up and disable autosaving for a game if s/he feels it's brought up too often.

-- Quote: End

[etking]

The huge annoying delay when pressing X for the first time after startup could be avoided by doing this once automatically at the very beginning.

And is there really a need to wait for the SD card to finish saving during emulation? Even a constant stream of ongoing autosaves every one or two seconds should not slow the emulator down, despite being not very SD write cycle friendly.

[Stewmath]

I would love to get this working properly, but I haven't had much success.

I open the file with "r+b". I write the bytes using fseek and fputc. At the end of the frame, I'd think I should use fflush() to save it, but that doesn't work properly. The only way to get it to actually save, is to fclose() and fopen() the file again, every time sram is written to. This makes a significant delay, unacceptable in the majority of games. Sometimes it's just writing a single byte, and the annoying delay persists!

[Stewmath]

I kinda got this working, starting around 3fba3965d755d061eb008f285367545a0abe939f. A lot of games use sram as temporary storage, so this can create spontaneous lag. There could be more I can do with this to try getting rid of that.

[FI87]

If the SRAM gets updated more frequently than the disk write can keep up, might I recommend trying to shadow copy the memory with perhaps snapshots every 1 second and only writing those down?

[Stewmath]

I'm not sure what you mean by "shadow copy". Anyway, for the record, autosaving was improved in version 0.4.1. It records the start and end ranges of writes to sram, then writes that range of sram to the sd card. Less lag this way, apparently.

[Searinox]

By shadow copying I mean making a "snapshot" of the current state of SRAM - with everything in it - one second and only every one second after it's written into, and then writing that content into the file. In the case of continuous SRAM data changes, it should only write to the disk once a second.

What I mean by "shadow" copying is that once a moment of the memory's snapshot is chosen, you don't instantly write it down. You also don't instantly make a full copy of it in RAM that gets written down gradually. Both would cause a lag spike and unnecessary extra usage of RAM.

Instead you start slowly writing the SRAM contents(a few KB per cycle?) to the drive and if the SRAM gets changed during that time, you only back up those specific addresses just as they're about to be changed into an temporary buffer, and then write those into the file instead when it's time to write down that specific part of the SRAM.

What this does is it allows for instantaneous snapshotting of large amounts of data at low disk and memory usage.

It's just more tricky to code. I think this is what big emulators(VBA, SNES9X) do. Ghosting applications and Windows' System Restore do it too. And in terms of performance, it should only double the work done during SRAM writes during the time the SRAM's being modified while the data's being written to the drive.

To summarize it better, once the emulator detects a change in SRAM data:

-begin writing SRAM to disk gradually, not all at once to reduce CPU usage and let the game keep running -if any changes to SRAM are made during this time, first store the data to be changed in a buffer and then change it -if the SRAM data about to be written to the disk hasn't changed since we began writing, continue to write it down without any hesitation -if the SRAM data about to be written to the disk has changed in the meantime, write the data in the buffer down instead -set a timer of 1 second. or more if it's not enough. until this timer runs out, the emulator will not attempt to write data again, but if data is changed during this time, it will remember this fact and begin writing as soon as the timer's up

[Nebuleon]

@Searinox (#21426171) Recording the offsets of bytes that have been modified, so that only those bytes are written. That's what Drenn said.

However, I think that, inside that huge comment, there is another gem hidden: Incremental saving with 1 entire second of delay if the GB/GBC has continued writing into the SRAM 1 frame after the last save.

[Stewmath]

I'm not recording the offsets of individual bytes which have been modified, I just record the start and end positions of the modified block. So, that can be wasteful, but it worked better than just writing everything right away, and flushing the cache later. I can't say I understood all of what Searinox said, however the real problem appears to be with flushing the cache. Wario Land 3 has always been the most problematic. It writes single bytes to sram when walking around a level, and when flushing the cache after this, it can cause lag. A single byte can cause lag. Interestingly, some games like Pokemon seem to lag only the first few times they write to sram. This might have to do with the fact that it's usually writing to the same location. I'm not entirely sure.

[FI87]

Sorry I hadn't looked at the source at that time. I'm looking at it now. The file write appears to be commented out right now, or shouldn't I be looking at mmu.cpp? Would restricting save and flush to once a second help? I'll take a better look at the code when I can.

[Stewmath]

I used to do the file write immediately in mmu.cpp. Now, the writing is done later in the "gameboyUpdateVBlank" function in gameboy.cpp. The for loop which does the writing may seem confusing if you're not familiar with the gameboy. Each iteration of the loop writes the contents of a single bank of sram which has been changed.

Also are you and Searinox the same person?

[FI87]

Yes I am/we are. Sorry for the confusion of having registered 2 different accounts. I had forgotten the credentials of one of them despite cookies keeping me logged in on said computer. Couldn't be bothered to reset the password, I don't think it's even possible since I didn't verify my e-mail address, lol.

My idea was that you could write fractions of the data with each cycle(not gameboy frame) and set a threshold low enough for the amount written that it wouldn't lag the emulator. But you said the _FAT_cache_flush is the real cause of the lag, right? Is it possible to partially flush data?

[Stewmath]

Maybe, but I'd probably have to hack libfat. Still, that may not be the main problem. I mentioned earlier, Wario Land 3 has lag when it writes a single byte to sram (and then flushes it). This happens when walking around the level. I haven't investigated this yet.

[Stewmath]

The last few commits, around 9460ea9f99813eca1396ae0a97c9e8448b5281ea, have improved autosaving behavior some more. It occured to me while I was doing this, that people with different flashcards may have different results. Myself, I'm using an Acekard 2i.

After studying libfat a bit, I decided to bypass libfat's cache altogether when autosaving. When libfat writes something through the cache, it writes 8 sectors at a time (each sector = 512 bytes). What I'm doing now, is I'm recording the sectors corresponding to the save file, and writing each one individually. My Acekard, at least, responds better when writing one sector at a time, instead of 8. The random lags (like in Wario Land 3) appear to be gone. Now, it only lags when large chunks are written to sram (which is typically only happens when saving, or in transition screens). "Shadow copying" may help with this. For now, at least, the most annoying lags seem to be gone.