Restructure Web Server Directories/Paths + Wrongly Named Battery and Save State Directories

[JoeMatt]

Currently we have a mixed system of whether or not a system id for a ROM is important.

Current structure

1. /BIOS/{system id}/... bios(s)
2. /{system id}/... roms
3. /Cover Art/{md5 of URL}
4. /Save States/{filename of ROM}... saves
5. /Battery States/{filename of ROM}/... batteries
6. /roms/ <-- The import folder for new roms

and improved solution would be,

Proposal 1

1. /BIOS/{system id}/... bios(s) -- Unchanged
2. /ROMS/{system id}/... roms -- Add ROMS prefix
3. /Cover Art/{md5 of URL} -- Unchanged
4. /Save States/{system id}/{ROM MD5}/... saves
5. /Battery States/{system id}/{ROM MD5}/... batteries
5. /Import/ <-- Rename from roms to avoid confusion

Proposal 2

- /{system id}/
     - BIOS/
     - ROMS/
     - Saves States/{rom md5}/
     - Batteries/{rom md5}/
- /Cover Art/{md5 of URL} -- Unchanged
- /Import/ <-- Rename from roms to avoid confusion

Wither either structure I'm preposing changing the way battery and save states are stored for a few reasons.

• File names can collide between systems since they're all bundled together
• If the user renames, or imports under a new filename, they won't be found
• Similarly, if a user has duplicate ROMs with different filenames, they can share the saves (would come in handy if streaming ROMs from remote destinations ever happens)

How to procede

Detect somehow that the user is on the old structure and move things around on first load of new version.

Proposal 1 This requires the least amount of changes. Most things stay the same. Easy stuff The current {system id} directories would just be moved to a new parent directory 'ROMS' Rename /roms/ to /import/, obviously simple too Stuff to consider

We could either,

• just wipe out the Save & Battery directories by first presenting a warning to users that they're going to be deleted and give them an option to either a) Delete and continue b) Run the web & web dev servers to allow them to back them up, c) quit and do nothing
• Move automatically move them to a /BACKUP/ folder and notify the user that this happened and tell them to use the web server or iTunes to mange where they should go. But how would they know if we use md5 and not filename from then on?
• Iterate the paths and search the database for a match by name, then move the saves to the new appropriate path based on the system id and md5 of the game that's matched. Move anything that didn't match to /BACKUP/.

First two options are pretty much the same thing but different, and require minimal code. The latter would need some code to do that matching and some kind of waiting indicator while it's proceeding.

How attached are people to their save states that we need to go through all that effort? I don't know. It's a free app and we can give them fair warning it's going to get moved.

Proposal 2 Same concerns as above. Files could be moved or backed up.

This is a slightly more organised way of doing things I think. It depends if you consider states to belong to the system or not, which they probably do, at least for memory saves, and we don't plan on adding multiple cores for the same physical emulated system (like openemu) so I don't see the harm in putting battery states in sub directory of system.

On the proposal of changing ROM name to MD5

Now this would help with matching on the code side much better if the database gets destroy, things are moved, renamed, reimported etc. I do see how it would make it difficult for a user to just load up a file browser and find the saves for games they want. We did propose adding that data into the Realm database, but even the, if the database gets trashed, it would be nice to have a way to find them again.

To please both the code side, and the user readability, we could do 'TITLE'+'MD5'. (Stripped of special chars like we do now). That would result in some long names, but we could always do a match up if things go south by just looking at the part after the '+' or whatever separator we use. I don't think there's a performance hit, since the md5s are already calculated and stored in the DB for each rom, and to iterate and parse even hundred of folders names, assuming a user has made that many saves, shouldn't take more than a fraction of second. iOS has a max path length of 1024 and max file/folder name of 255 so I think that's fine too, but we should add code checks anyway to truncate really really long rom names to that path+'title'_'md5' is under 1024 by some amount so there's room for the filenames of the saves themselves. The title would be truncated in this scenario.

Additional notes I'm currently in the process of way simplifying the PVEmulatorConfiguration class.

I wrote extensions for PVGame to return the path of it's saves, bios's etc. Now it just uses the system to figure that out, but that could easily be changed to the updated paths in the extension that everything calls and not have to go around and change paths all over the code.

I wrote a new class for Systems that does something similar. There's a lot of code right now that try to infer one it's own where stuff should be, I'm changing that to use PVGame when it's asking about a game, or the new System struct when it's asking about system files (rom dir, BIOS dir etc). So that should make changing paths around easy for the code, we just have to deal with what the user experience is as outlined in the proposals above.

[jasarien]

I like Proposal 2 — keeps all things related to a system together.

[leolelego]

the proposal 2 could be very nice with the webdav support for backuping data!

[sevdestruct]

i'm on board with system 2, but we can drop Cover Art now, which is already implmented in UIRedesign branch along with Imports folder and rename of Conflicts, i think tucking all system paths into sub-dir is smart since people are mucking about in here, def like the ROMs sub-dir idea. then we have a nice and tidy root dir. all for this.

[sevdestruct]

Updating to task lists with some small revisions:

• [ ] /{system id}/ (why 'system id', and not just 'system'?)
  • [ ] BIOS/
  • [ ] ROMs/
  • (all the roms folders per system go here of course)
  • [ ] Saves States/{rom md5}/
  • [ ] Battery Saves/{rom md5}/
  • ~/Cover Art/{md5 of URL} -- Unchanged~ (redacted, deprecated)
  • [ ] Shaders
  • [ ] [Other, Realm, etc] (TBD)
  • [x] /Conflicts/
  • [x] /Imports/ <-- Rename from roms to avoid confusion (now supports Cover Art watcher functionality, universal dump spot

---

In addition… (considering work being done on multi-disc support)

• [ ] FIx:
  • Battery Saves/
  • [ ] "Game (Disc X)" [folder] -> "Game" [folder]
  • Save States/
  • [ ] "Game (Disc X)" [folder] -> "Game" [folder]

…based on m3u name/or disc name removal (.mcr / memcards already named correctly). The names currently are being determined by whichever disc gets imported/sorted first in the set by the importer, which as tested, isn't always the same or even Disc 1 for that matter.

[sevdestruct]

just noticed…there is the undecided component up there that @JoeMatt has regarding renaming the incoming files on import, if i am mistaken? what's the take on this? pros/cons? that could be it's own ticket outside of restructure of the dirs, if this is still a discussion point

---

It's an intresting fix for unconventional file naming conventions… but some consdierations…

1. what happens for unmatched roms (hacked roms, translations, otherwise unmatched hash roms?
2. what about manual cover art uploads? this would break it, no? or force the user to know the rewrite name instead of their incoming names (or we log that incoming name so their manual uploaded covert art matches (key to have a solution for this for tvOS especially since no paste feature…)