gba_db.bin integration and more configuration

[HTV04]

gba_db.bin integration:

• Added gba_db.bin builder Python script to the repo for building a valid gba_db.bin file from MAME's gba.xml and No-Intro's GBA DAT (with scene numbers). The generated gba_db.bin currently has 2,961 entries.
• Added full gba_db.bin support (utilizing the original debug code), allowing multiple previously problematic games to function properly.
• Tweaked saveType debug output to be more useful during testing.

config.ini changes:

• Added "Advanced" section to config.ini.
• Modified backlight option in config.ini to support a wider range of values depending on the 3DS model. The default value is now 64 as well.
• Added several options to config.ini:
  • bool useGbaDb - Use gba_db.bin to get save types
  • bool adjustGamma - Adjust screen gamma using values from inGamma, outGamma, contrast, and brightness
  • Having this option set to false uses the 3DS's default gamma setting instead of overriding it.
  • bool saveOverride - Open save type override menu after selecting a game
  • The save type override menu is based on the original save type debug menu used for gba_db.bin testing.
  • u16 defaultSave - Change save type default when save type is not in gba_db.bin and cannot be autodetected
  • The serial override list for save type autodetection has been deprecated and commented out due to this change for several reasons.
    • Its main usage, other than for the Homebrew (null bytes) and Goomba Color entries, was to include entries for problematic EEPROM games. This has been fixed with the gba_db.bin integration.
    • The default value for defaultSave is 14, SRAM 256k, which maintains compatibility with homebrew and shouldn't break games that do not have a detectable save type (only applicable to ROM hacks or games not in gba_db.bin, since games that do not have a save type but are in gba_db.bin use save type 15, which is none).
    • It expands homebrew compatibility for titles like Goomba Color that have "vanity serials."

Other changes:

• Updated Makefile to include pre-generated gba_db.bin and fresh config.ini (so new options always appear) from the resources directory with releases.
• Updated README with FAQ and other useful information.

[HTV04]

I left a few comments of things to correct/questions.

More minor nitpicks: Not sure it's a good idea to require the user to install Python just for the EEPROM save fixer.

It's either that or on Windows, they have to install Visual Studio and the entire Windows 10 SDK, or install MSYS2, and know how to compile source code. Same with Linux since I believe they would have to install build-essentials or something along those lines, instead of just installing python3 (which is usually installed by default).

[profi200]

I don't understand why you say they have to install all of that. That savegame fixer is intended for users who don't necessarily want to compile the whole project. Now the previous version with the gist and zero instructions + no builds was not any better. I give you that.

[HTV04]

I don't understand why you say they have to install all of that. That savegame fixer is intended for users who don't necessarily want to compile the whole project. Now the previous version with the gist and zero instructions + no builds was not any better. I give you that.

Which save fixer are you referring to? I was talking about the previous version written in C that I replaced with the Python script. Instead of having to install a dev environment for the previous version, they could just install Python 3 for the Python script.

[profi200]

I hate to say it but the easiest would probably be some JavaScript based converter. I hate webdev with a passion but it's the most accessible option here i think. I don't know JS at all though.

[HTV04]

I hate to say it but the easiest would probably be some JavaScript based converter. I hate webdev with a passion but it's the most accessible option here i think. I don't know JS at all though.

I agree. I don't know JavaScript enough to recreate the EEPROM fixer tool in it, though.

[exelotl]

Here ya go: https://github.com/exelotl/gba-eeprom-save-fix

(Please double check the output of this web port and let me know if it's OK)

[HTV04]

Here ya go: https://github.com/exelotl/gba-eeprom-save-fix

(Please double check the output of this web port and let me know if it's OK)

Wow, thank you! Looks like it works perfectly.

@profi200 I've adjusted the EEPROM Fixer section in the README to link to this instead if it's okay with you.

[profi200]

That's great. Thanks. @SonoSooS was also working on one which seems to work even on very old browsers.

Something i wonder about is what if a user has a whole folder full of saves? Converting each one would be a pain. I wonder if any API allows converting a whole folder full of saves.

[HTV04]

I wonder if any API allows converting a whole folder full of saves.

I think you can, actually. I know the Google Drive website has an option to upload a folder which uploads all of the files in it, so this is very likely an option (it isn't proprietary either since other browsers than Chrome support folder uploads).

Is it plausible though? Sure, it would be convenient for many saves, but couldn't saves just be converted as the user needs them? Not to mention it could be messy with other save types mixed in.

[profi200]

I can see users already requesting this and it certainly saves lots of time. To avoid converting the wrong saves we can turn the check into a more strict one only allowing 8 KiB and 512 bytes files.

[nadiaholmquist]

If the save fixer couldn't be integrated into oaf itself, maybe the most user friendly solution could be having it as an app for the 3DS itself?

[HTV04]

I think it could be integrated into oaf, the fixer just reverses every set of 8 bytes in the SAV file.

The problem here would not only be that it would break the existing saves of users who were unaware of the change, but also that it would take longer to launch EEPROM games, since the conversion would need to occur during the game launch and during power off, when the save file is saved to the SD card.

This change is only needed for emulators, so it's a bit unnecessary to have this as a default setting. Maybe this could an optional setting for people who need it though. In a perfect world, emulators would actually store saves in the format open_agb_firm expects though, since that's the format it's actually stored in on real hardware.

[nadiaholmquist]

It probably wouldn't really increase the load time by anything notable, simply byteswapping a file is trivial for the system to do.

[HTV04]

Maybe. My main concern here is the fact that such a change would break existing OAF saves, since there's no way to detect if it's in the legacy format. If it doesn't hurt launch times I wouldn't mind it too much, though.

[profi200]

Thing is we only have this problem because some popular emulators got the EEPROM saves wrong and never fixed it. They assumed data is stored in this wrong format based on the way EEPROM saves are read/written. No$GBA is probably the only one that got this right. Other, newer emulators like mGBA use the wrong format for compatibility reasons. Ideally we can push all emulators to use the correct format but i can see why they would not want that (noob complaints).

[profi200]

Before i forget to mention it: A little update on the EEPROM save issue. I have talked a bit with others about this and it seems there is no way that we can get the correct byte order format going without causing people to complain left and right about corrupted saves. So the solution is renaming correct byte order save files to .eep instead of .sav. Everdrives apparently already use this naming scheme.

[HTV04]

Just made a commit with all of my post-review changes. I don't think I have anything else to add, but do you want me to change anything?

[profi200]

Maybe squash the 3 README commits into one and then force push to your repo? There are a number of changes to driver code i have not pushed yet so it's maybe a better idea to wait for them and then you can rebase your commits.

I'm also working on a proper SDIO driver so maybe we can decrease game loading times a bit more (that was not the main reason why). And i have something else in the works that could prove useful for features i otherwise could never support ( ͡° ͜ʖ ͡°)

[HTV04]

Done.

[profi200]

@HTV04 Can you rebase this against the latest commits?

I know it's been a while but i explained this on the GodMode9 Discord/IRC channel that motivation is a major killer here and if i work on something it's usually the code behind oaf because i want to split it into a "bare metal" homebrew lib. oaf is basically the side project and the lib is the main project.

[HTV04]

@HTV04 Can you rebase this against the latest commits?

I know it's been a while but i explained this on the GodMode9 Discord/IRC channel that motivation is a major killer here and if i work on something it's usually the code behind oaf because i want to split it into a "bare metal" homebrew lib. oaf is basically the side project and the lib is the main project.

Sorry for the delay, just saw this. I rebased my commits and also re-implemented checkSaveOverride based on #40.

[profi200]

There is still one small thing that makes no sense to me. It's including the config.ini. oaf does create it automatically on first boot. Otherwise fine i guess. I want to get rid of most pull requests now to make sure they don't end up unmergable as i change things.

I will merge this now anyway. I can always throw that file out.

[HTV04]

There is still one small thing that makes no sense to me. It's including the config.ini. oaf does create it automatically on first boot. Otherwise fine i guess. I want to get rid of most pull requests now to make sure they don't end up unmergable as i change things.

I will merge this now anyway. I can always throw that file out.

I don't remember exactly why I decided to include it, but my best guess is that it allows the user to configure settings before the first boot. It isn't necessary though, so feel free to remove it.

Anyways, thanks for merging this! Honored to have my changes finally be a part of the base open_agb_firm.

[profi200]

Might as well ask here.

There is another thing i forgot to mention. I plan on relicensing the whole project. I need to ask every contributor for permissions for this step. The reason for this step is the code might be useful to others who want to develop their own bare metal FIRM but the current license is relatively restrictive. BSD and MIT have been recommended to me. I will probably go for one that still requires preserving the license and credits.

So... @derrekr @d0k3 @exelotl @HTV04 are you all ok with this?

[HTV04]

Sure, I don't mind at all. I think the MIT or Apache licenses would be good choices if you're looking for a less restrictive license.

[derrekr]

+1 I don't really care about the license, ppl just shouldn't be dicks

[d0k3]

I don't mind either, and I agree with @derrekr - that's my stance, too. Any "don't-be-a-dick-license" will do.

[exelotl]

Oh, yeah that's absolutely fine!

[profi200]

Thanks. I will decide on a license soon and switch to it. There are minor issues to resolve like GPL code from Luma3DS but i think we can get that resolved.