Embed options into RetroPie ES Menu instead of it's own ES menu?

[esmith13]

Is it possible to embed the options of this script in to the RetroPie ES menu instead of creating it's own dedicated ES menu for a cleaner look? I understand how to manually add shell scripts to the RetroPie menu but it appears to me (as a novice linux tinkerer at best) that you software is a single shell script that I'm guessing needs data passed to it at the command line to set off a particular option? Essentially I am trying to specifically use your Dropbox backup and restore features on the current version of RetroPie (which does work great still, BTW) but without integrating PiAssist into it's own dedicated ES menu. Any help would be appreciated. Also, does your script do a blind backup and restore to dropbox or does it look at each state and sav and check the date on it to ensure only older files are over written? I'm thinking I want to use this to keep sav and state files in sync between two separate RetroPie systems if I can. Thank you in advance and thanks for the great and very useful script!

[Death259]

Sorry for just now seeing this... didn't think anyone was still using this until i saw your comment on Reddit. I tried to separate out all of the save file managing portion into a separate BASH file for those that just want that functionality. I've uploaded it here. I haven't tested it out as i don't have an easily accessible RetroPie setup. If you don't mind, please give this a shot and let me know if it works.

As for how it should work, the program just looks through all of the rom directories defined in the es_systems.cfg. It grabs anything that matches the save file extensions that i defined in the script. This is currently SRM, BSV, SAV, and STATE. At the time this was all that i knew existed. It would be pretty easy to add more as needed though. Those all get zipped up and then uploaded to Dropbox using the Dropbox uploader I found. There's no comparison to do an incremental upload, it just copies everything.

As for adding menu entries to the ES menu, you can use "Backup Save Files to Dropbox" and "Restore Save Files from Backup" for your parameter that gets passed to the script file. It should read that and then just run the portion of the script you selected and then exit itself. This functionality will work with the full PiAssit. The full PiAssist currently supports the following parameters "Update PiAssist", "Backup Save Files to Dropbox", "Update Emulation Station Entries", "Restore Save Files from Backup" and "Change Splash Screen."

I hope this helps, if you do end up using the stand alone save file manager script, please let me know if everything does or doesn't work.

[esmith13]

Thank you for the reply. I tried adding the full script to the retropie menu but with no results. I think I fugured out why. The way retropie currently handles it's own ES menu is to just launch a script with no parameter passed. Basically I would need two .sh files, one that performs a backup and one that performs a restore - with no parameters required (choosing to launch a script from this menu is no different than typing in terminal './my_script.sh'). Each one, therefore, would have to be capable of asking for the api key if it's missing. Also, additional files should be backed up, they would be .fs, .nv, .rtc, and instead of .state it should be .stat or .state (does linux allow double wildcards?) since state files can be named .state, .state0, .state1, etc... Also in a perfect world you would compare what's already on dropbox with what you are currently trying to backup or restore to ensure newer files are not overwritten. To maintain the idea of uploading only a single compressed file instead of multiple loose files you would most likely need a temp space to pull down an existing backup to so you can decompress it and compare file dates. The best use case for this is cross syncing saves between multiple retropies - not just using it as a disaster recovery method for one system.

I understand if you don't feel this is used by enough people to make these changes. I appreciate you even taking the time to respond. If I knew how to write linux scripts I would fork PiAssist and do it myself but at the moment it is over my head. I barely get by with windows batch files currently.

Thanks again for your time.

[Death259]

Per the latest commit I think i've managed to do what you are needing as far as not needing to provide parameters to the scripts. Give the two uploaded script files a whirl and let me know if this accomplishes that. As for comparing save files so newer ones are not over written... that would be a doozy.... it's doable i'm sure, but would take a decent bit of work. That sounds like a fun challenge, but i'm not sure if i have enough time to do this... if i can, then i will. As for adding the other save file types, that will be pretty easy, just want to make sure that the separation of the functionality into the 2 separate script files works first.

[esmith13]

Both when run from within emulationstation and when run from the command prompt directly the scripts both do nothing. They do not even perform the wget to download PiAssist. I have of course done 'sudo chmod +x' to each of the files.

[Death259]

Whoops.... sorry about that... i had a brain fart... the latest commit should fix this. Let me know the results.

[esmith13]

No worries. It now runs but still fails to install the PiAssist directory or even perform the wget. You do see it go thru and list all the save files I guess it would backup if it was working. It never asks for the API key either but since there is no /home/pi/PiAssist directory there is no dropbox_uploader.bsh to run anyway. All results are identical when run from within ES as well as command prompt with or without sudo. If it didn't turn the screen blue with the OK box you can't acknowledge without a keyboard anyway, I could probably tell you more. It moves too quick, sorry.

[Death259]

Ahh... yeah, instead of trying to do this all blind, let me see if i can get my Pi up and running again. Then I can test everything on there. Might not be till this weekend though.

[robertybob]

Like @esmith13 The main appeal to me these days is the backing up of saves and savestates. I would love to be able to run the backup facility on shutdown (I don't play that many different games so this wouldn't take a long time to run). When I get around to it, I'll investigate how to do this. I'm a definite Linux noob so it may well take some time!

[Death259]

I found this post on running scripts on shutdown. Using that, once i get my script fixed, you should be able to just copy the backup portion to that script and then it will backup all save files on shutdown. Likewise, you could probably do the opposite when the pi turns on. That way you can have everything autobackup and then autorestore. Then if you have multiple RetroPies, you won't have to think about it.

[esmith13]

Just remember, you can't really use it on multiple Pi's unless it is modified to check dates on the files and only overwrite older ones! (wink)

[Death259]

Yeah, that would be pretty dope if i could figure that out.... so, i'm hoping to look at this in the near future and see if i can accomplish that.

I think to boil it down to how things would have to be done to do the backup is that i would check to see if a save set already exists on dropbox and if not just back up everything. If there is already a backup then i would need to download and extract that set and see what is newer and only overwrite the older ones with the newer ones. Then i would zip that all back up and save it back to dropbox.

Conversely to get the latest saves, i would download the whole save set and then only overwrite the older ones with the newer save files.

Does that sound about right?

[esmith13]

that is perfect. The only difference between what you explained and what I have used in practice via windows batch files is that I don't zip up my backups. In my dropbox I would have a folder called "RetroPie_Backup" and in that folder would be the same 1st level structure as the roms folder of retropie (folders named nes, mastersystem, pcengine, ...) and in them would be the saves/states. I was using the xcopy command which could be set to only overwrite a file after comparing it based on modification date. This way if I played a game on one pi, synced it to a second pi and played more there but then played a new game on the first pi - I could backup the first pi and not have it's older save of the first game overwrite the newer progress. In my case a single script did both tasks - perform a backup of newer and/or missing saves and copied down newer or missing saves from the cloud. More of a "cloud sync" then a separate backup and restore process.

[Death259]

gotcha.... i was thinking of doing them separately and you could have a process for starting up or shutting down.

If i could do more of a Sync, that would be great. How did you handle discrepancies, such as you creating a new save file on Pi #1 and a new one on Pi #2, but the one on Pi #1 which is older than Pi #2 is really the one you want?

I hope i explained that right...

[esmith13]

You always want the newer file (by modification date). Going by the assumption that you setup the sync on all Pi's when they are first built and start with syncing down everything in the cloud, the newest file will always be the furthest into the game. Also, while I think your script should respect the idea of cross-syncing multiple devices, I like that you plan two separate scripts so they can be done at startup and shutdown. That said - make sure you first check for a valid internet connection, else end the script. That way it can blindly run at startup and shutdown with no errors if not connected to a network. If you know windows command line at all I'd be happy to post my crappy code for my current script here if it helps you understand better.

[esmith13]

In my mind, this is the process...

RESTORE SCRIPT (could be either run at startup or from retropie ES menu manually): If internet doesn't exist - exit script. Pull down archive from dropbox and expand to temp folder. Read in a temp file. If exists in roms folder, only replace if temp file is newer than local file. If not exist, add to local roms folder. Repeat until all temp files have been checked. Cleanup temp files.

BACKUP SCRIPT (could be either run at shutdown or from retropie ES menu manually): If internet doesn't exist - exit script. Pull down archive from dropbox and expand to temp folder. Read in a save file from roms folder. If exists in temp folder, only replace if local file is newer than temp file. If not exist in temp folder then copy it there. Repeat until all local roms folder save files have been checked. Zip up temp folder and upload, replacing existing backup. Cleanup temp files and zipped files.

[Death259]

I should in theory be able to do one script file that then has the backup and restore options separate so that way you can do one or the other, or both. Then in the startup or shutdown process you will call the script file with the parameter to backup or restore. I think then using the ES menu, you can use the RUNCOMMAND that is baked into retropie and call the backup or restore process separately or together. That should make it easy to just add these commands to the existing RetroPie ES menu. Then someone can have the option of manually backing up or automatically at startup/shutdown. I may even be able to go as far as then allowing it to be scheduled so that every X minutes it backs up everything...

[esmith13]

That sounds wonderful!

[esmith13]

I eagerly await a proper solution for this, but in the meantime I have put together my own entry level scripting solution that actually works so I thought I would post it here for reference. It involves 4 files:

/home/pi/BackupScript/backup.sh /home/pi/BackupScript/dropbox_uploader.sh (from https://github.com/andreafabrizi/Dropbox-Uploader) /home/pi/BackupScript/extensions.txt /home/pi/RetroPie/retropiemenu/DropboxSync.sh

dropbox_uploader.sh is unaltered from the source. extensions.txt is a list of extensions to backup, one per line (ex: .srm, .sav) DropboxSync.sh in the retropiemenu folder simply calls the backup.sh script:

cd /home/pi/BackupScript
bash ./backup.sh

backup.sh is my beginner's attempt at cross syncing the data:

rm -rf /home/pi/BackupScript/roms
mkdir /home/pi/BackupScript/roms
rm -rf /home/pi/BackupScript/RetroPie_Save_Backup.tar.gz

bash ./dropbox_uploader.sh download /RetroPie_Save_Backup.tar.gz /home/pi/BackupScript

tar -xzvf /home/pi/BackupScript/RetroPie_Save_Backup.tar.gz -C /home/pi/BackupScript

rsync -avtWr --update --include '*/' --include-from=extensions.txt --exclude '*' /home/pi/RetroPie/roms/ /home/pi/BackupScript/roms

rsync -avtWr --update --include '*/' --include-from=extensions.txt --exclude '*' /home/pi/BackupScript/roms/ /home/pi/RetroPie/roms

tar -czvf /home/pi/BackupScript/RetroPie_Save_Backup.tar.gz -C /home/pi/BackupScript roms

bash ./dropbox_uploader.sh upload /home/pi/BackupScript/RetroPie_Save_Backup.tar.gz /

rm -rf /home/pi/BackupScript/roms
rm -rf /home/pi/BackupScript/RetroPie_Save_Backup.tar.gz

It's meant to be run on demand and cross syncs data both to the local Pi and the cloud insuring that each file in each location is the newest version and any missing files are added as well. It could be automatically run at startup and shutdown but I have no idea how to do that. Also, it does not currently verify there is an internet connection before executing. Since it is set to run manually from the retropie ES menu that's not a big deal but if I figure out how to automate it I would need to include code to skip the entire process if not connected to the internet so startup and shutdown are not slower simply due to lack of internet access.

[Death259]

Ahh, dope! I'll take a look this weekend. Should be pretty easy to test for internet connectivity and then don't do anything if there isn't any internet. I found the following code that should work... hopefully. You'll just replace "echo online" with the actual code you want to run. Then the "echo offline" can display a message letting the user know there is no internet connectivity.

wget -q --tries=10 --timeout=20 --spider http://google.com
if [[ $? -eq 0 ]]; then
        echo "Online"
else
        echo "Offline"
fi

[Death259]

I think I managed to get it all working properly... The latest commit has a new PiAssist-SaveFileManager.sh that should do everything. All it does is synchronize though. There's no restore vs backup option now.

The process is pretty much as follows:

Get backup from dropbox if it exists and extract all completely new and newer save files where they need to go. Then make a backup of the currently running Pi's save files and upload them.

Give it a whirl and let me know your results. You might want to take a manual backup of your save files just in case....

Edit: I tried to run all of the logic in my head on using two different RetroPies in the various scenarios (at the same time, one right after the other, etc.) and I think the logic in the script should accommodate for just about any scenario. Since you have two systems, please try it out in as many different scenarios that you can come up with.

Also, the script no longer requires any arguments... it just runs. Should make things easier for adding to the ES menu as well as adding it to startup and shutdown. If everything does work properly, I can probably add functionality to have it schedule itself to run on shutdown/startup.

[esmith13]

Sorry for the delay in getting back to you. Thank you very much for your time and effort. I only have one issue with the latest commit. Your script grabs save location info from the es_systems.cfg instead of just scanning the rom path in general. Because of this, I would ask that you alter your script to follow the RetroPie best practices and look for the es_systems.cfg first in '/opt/retropie/configs/all/emulationstation/' (or it's symlinked location of ~/.emulationstation) and ONLY if it doesn't exist there, read from the default location you already use at '/etc/emulationstation/es_systems.cfg'. The reason for this request is best stated in the RetroPie wiki here: https://github.com/RetroPie/RetroPie-Setup/wiki/EmulationStation Here is a quick quote from the wiki as to why you should grab es_systems.cfg from the "configs/all" directory if available there: "Because of the way the RetroPie-Setup script works custom configurations will need to be copied and edited in specific places, otherwise your custom edits would be overwritten anytime a system was updated from the RetroPie setup script." If you are willing to make this change I will be happy to test it thoroughly in a multi-Pi scenario for you. Thanks again for your work!

[Death259]

I'm so sorry for only just now getting back to you. Apparently I missed your last response. I've modified the script as you have suggested in the latest commit. Let me know if you need anything else.