Game Library Syncing

[jasarien]

Sync ROMs, saves, etc…

Possibly using dropbox? Other suggestions?

[kaikai1234]

found this on droopbox forums " Gregory September 15, 2015 20:58

Thanks for all of the information and screenshots guys, that's very helpful. I'll bring this up with the team internally.

One thing that may work with the existing infrastructure though would be how Apple TV apps of the current generation work for connecting external accounts. That is, the Apple TV app gives the user a short URL for the app's web site to browse to on an Internet connected device with a browser, and that web site processes a standard OAuth 2 app authorization flow. Upon redirected back to the app's web site, it can send down the necessary token to the TV app. That would offer a better user experience, especially if the user is already signed in to Dropbox on their phone, for example, as they'd only have to type in the short URL on their device, as opposed to typing in their full Dropbox credentials on the Apple TV (possibly including a second factor, etc.)."

https://www.dropboxforum.com/hc/en-us/community/posts/204784269-Authentication-to-dropbox-on-tvOS

[ghost]

turns out Facebook released an sdk for tv os log ins . https://developers.facebook.com/blog/post/2015/11/25/tvOS-SDK-beta/ maybe this can point to some way of getting drop box to start working or somehow signing into a Facebook thats connected to some dropbox account.

[XeresRazor]

Why don't we just add a sharing sheet to save games, so you can copy your saves to and from your iCloud Drive app. It's not perfect, but should be a pretty solid workaround for not being a store app.

[charlesvestal]

While likely not ideal, as it requires a server-side component, OAuth logins are possible on TVos without (I believe!) using private APIs, as demonstrated at https://github.com/aaronpk/TVAuth-TvOS.

[oliviercharavel]

I think this is worth taking a look at: https://blog.agilebits.com/2016/01/12/agilecloudkit-icloud-sync-gets-its-wings/

[jasarien]

Cloud kit isn't suitable for storing/syncing large files like ROMs (and before you say "ROMs are tiny!" - SegaCD ROMs are upwards of 650MB).

[oliviercharavel]

I agree about ROMs for sure but I was more thinking about saves/savestates in fact. 😊

[Recognition101]

To solve the AppleTV problem, do you think we could put the Dropbox (or Google Drive, or whatever) login on the Provenance Import Roms locally-hosted page? You use a standard client side JS OAuth flow to get a token, and then send that token over to the Objective C app for storage / re-use. I'm not experienced in ObjC, so I don't know if there's a library that can consume that token easily (specifically for Dropbox), but if there is such a library then this would at least fix AppleTV's lack of web views.

[ghost]

http://9to5mac.com/2016/02/05/apple-adds-server-api-to-cloudkit-unlocking-new-developer-possibilities-for-the-icloud-service/

something new

[jdelman]

FYI - I've created a branch with limited (but working) battery save (not state save) support using CloudKit. It allows you to copy saves to and from CloudKit manually - there's no automatic syncing. See https://github.com/jdelman/Provenance/tree/cloudkit-saves.

There is now a "folder" button on the game library view controller that takes you to the screen. It will show a list of your local and cloud saves. You can copy a save from local -> cloud by tapping on the local save, and vice versa for cloud saves.

If you're interested in building this into your own fork, you'll have to:

• add the iCloud entitlement to Provenance (and ProvenanceTV, if you're building for tvOS) and enable CloudKit
• configure a container using CloudKit Dashboard and set the proper container bundle ID on line 21 of PVCloudBatterySaves.m and in your entitlements
• create a schema called BatterySaves with the following fields:

Field	Type
dateSave	Date/Time (NSDate)
device	String (NSString)
fileExtension	String
game	String
saveBinary	Asset

Please let me know if you try it out, and of course, pull requests / improvements are welcome!

[oliviercharavel]

@jdelman This is working great. Thank you! 😊

[holincart]

@jdelman This is fantastic. Thank you. Got it working last night, and now I'm sharing save files between my iPhone, iPad, and AppleTV.

Question - is the manual saving (not automatic syncing) a deliberate design choice, or one made for the sake of releasing it sooner? Thanks!

[jdelman]

@holincart Thanks for the feedback. I suppose the answer to your question is - it was a deliberate design choice made for the sake of releasing it sooner. ;)

Also, I didn't want to deal with conflicts / sync issues. Based on what I've observed with modification dates, it seems as though Provenance touches the Battery Save file as soon as it starts emulating the ROM - meaning that if, for some reason, you have Provenance open and run the same ROM in two different spots at once, you might get undesired behavior.

But please - if you have any suggestions as to when to do the sync, or how to handle conflicts, let me know! (@jasarien, have you checked this out yet? :D)

[holincart]

@jdelman Good call - definitely appreciate you taking the simpler path to get it out sooner!

I haven't given it much thought - and it's not pretty - but one option would be to prompt (via another action sheet) to upload the battery save after a user saves a state save. Clunky, but it's a relatively logical time to do it, and the user would be consenting to it. Could even add a toggle in settings to automatically upload the battery save after saving to a "local" state save.

Far from an ideal implementation from a UX standpoint, but may bring the functionality itself closer to the ideal scenario for those toying with the branch. On that note - and pretty much just for my own curiousity - why'd you go with battery saves over state saves?

[vgz]

https://twitter.com/bobard/status/742457078229307392

.@tapbot_paul “Developer ID-signed apps can now take advantage of CloudKit, iCloud Keychain, iCloud Drive & push” https://developer.apple.com/library/prerelease/content/releasenotes/MacOSX/WhatsNewInOSX/Articles/OSXv10.html#//apple_ref/doc/uid/TP40017145-SW1

Looks like it'll be possible to use CloudKit for syncing the game library and saves later this year even without being in the AppStores.

[jasarien]

For the Mac... not for iOS or tvOS.

[vgz]

Ah, damn. Maybe they'll allow it for tvOS and iOS as well before the betas finish. It at least gives hope that something might change in that regard.

[jasarien]

I don't think so.

DeveloperID has always been for distributing signed, secure apps outside of the Mac App Store (because Apple understands that disallowing non-MAS apps would be disastrous for the Mac), and allowing CloudKit to be used with DeveloperID signed apps is a no-brainer and it's a wonder as to why they haven't allowed it until now.

iOS never had DeveloperID and never will, because it's totally restricted to running apps from the App Store - there is no reason for Apple to allow CloudKit for non App Store apps on iOS because as far as Apple care there are no non App Store apps for iOS.

[lonkelle]

So, are we still in limbo with this then? Nothing official going forward (Dropbox, Google Drive, etc) due to Apple TV incompatibilities?

[jdelman]

@JoelStickney IIRC it's not Apple TV incompatibilities. the problem is that this application is not allowed on the App Store. that means you don't get access to Apple's developer services like iCloud / CloudKit. any solution would require propping up a custom server to support syncing -- whether that server is for full blown syncing or just to relay oauth info to dropbox -- which costs money & requires volunteer effort.

[lonkelle]

Any other device (aside from the Apple TV) can connect to Dropbox directly, right? So, the intermediary server for OAuth is just for that?

[jdelman]

@JoelStickney Yes.

[hectorgarrofe]

Any updates about cloud sync? It would be awesome.

Cheers!

[marcelvoss]

I would also prefer Dropbox. Makes definitely more sense than using CloudKit.

I also think that it makes the most sense to sync game saves only for now (or at least make it into an option). A huge game library could fill up the entire Dropbox.

[blakedgordon5]

Using Firebase 3.x with this would be possible. Firebase has added storage capabilities, works great with offline/online, and is compatible across several platforms (including tvOS and iOS)

[leolelego]

syncing with dropbox can be awesome! I'm not sure it's necessary to sync games, only save state really matter. Game can be imported in function of devices easily with the web UI, and all player already have it own game library somewhere. Moreover most ROMs are available on Internet easily. But save state with auto sync can be great to never loose their.

[galip91]

Any updates on this?

[deejaywoody]

+1 for syncing of saved games across wherever you want (from inside the Files app). Could be Dropbox, iCloud Drive, whatever. Anything would be appreciated!

[JoeMatt]

I wouldn't hold your breath on this feature as far as saved states for a few reason,

1. Various cores might have version mismatching and save states won't work across devices that aren't on the exact same version. I could see a scenario where a sync overwrites a save file on you have a version mismatch, now your save is Fubar'd until you update your other device to the same version of Provenance. And you can't exactly just download an update through the app store.
2. Authneticating with a cloud service would require using OAuth most likely. A lot of 3rd party SDKs are lazy and use a website to create a token and send back to the requesting client app. This is a problem for Apple TV sine there is no webview. We have to write totally custom UI to allow passing login information. Even SDKs for cloud services that have nave iOS components to do this without a webview don't have the same on Apple TV.
3. Has to be a way to choose which save state gets overwritenn. Currently Provenence users 5 slots plus an autosave slot. How would we know which save state to overwrite when syncing across devices? We'd have to come up with a whole new way of handling multiple save states.

I could keep going but I think you get the point.

ROM sharing on the other hand I think is doable. I had an idea while ago of something like a Plex server for ROMs. We could even build this as an extension into OpenEMU since they use the same exact ROM metadata format.

This would be more like devices on the same network could announce a bonjour service with a basic REST API that describes some endpoints like /systems /roms/systemid and provide a URL for another client to download that ROM. We could make it so the ROM is copied and imported into the local library, or is saved to a temporary cache and played from there.

This would actually be helpful for Apple TV since all ROMS already live in a volatile cache directory, by design of tvOS, and they could be deleted at any point as the device runs out of space.

For iOS devices that are away from a home network, this probably wouldn't be as helpful unless you have a VPN back to your home (which I do on my iPhone but I assume most people don't)

[josejulio]

I think that for having saves on the cloud, we could have them manually, e.g. User selects "Save in cloud" (optionally a number of slots) or "Load from cloud" As @JoeMatt comments, an automatic process is more complicated because of multiple devices being used at the same time.

[amaurycatelan]

@josejulio exactly, this way would be amazing 👍

[JoeMatt]

I guess I'm having a hard time finding a compelling use case.

Is this to,

1. Save a state and continue it on another device at a later time or,
2. Back up save states and restore in case of loss of data.

If it's just #2, in the current PR for #677 I expanded the build in Web Server to support WebDav. Meaning now you can copy and overwrite entire directories in OS X finder or WebDev client which should allow for easy backup and restore.

I'm going to add an option to have it always running too so you can leave the client connected and not worry about it disconnecting every time you back out of the Upload screen.

[jasarien]

Syncing saves states specifically would be a no-no in my opinion, for the reasons @JoeMatt described about version mismatching and the fact that save states are generally volatile between core versions.

But I can see a use case for syncing a battery saves (saves made by the native, in-game save mechanism), as they are non-volatile, even between emulator implementations. Syncing these would be convenient for playing on one device and then picking up that save on another.

However, I’m not placing this as a very high priority at all, there are much bigger fish to fry around controller mappings, game importer bugs, library UI, core implementations, etc.

[XeresRazor]

Maybe a decent option would be to have manual upload/download functionality. Let the user configure a WebDAV server, or network share, or Dropbox location, and then have controls in the save state UI that let you push or pull a given save to the configured storage location.

Sent from my iPhone

On Feb 7, 2018, at 11:57 AM, James Addyman notifications@github.com wrote:

Syncing saves states specifically would be a no-no in my opinion, for the reasons @JoeMatt described about version mismatching and the fact that save states are generally volatile between core versions.

But I can see a use case for syncing a battery saves (saves made by the native, in-game save mechanism), as they are non-volatile, even between emulator implementations. Syncing these would be convenient for playing on one device and then picking up that save on another.

However, I’m not placing this as a very high priority at all, there are much bigger fish to fry around controller mappings, game importer bugs, library UI, core implementations, etc.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[deejaywoody]

I think the ability to sync battery saves would be excellent. Perhaps not means enabled by default, but there if you choose to use. To stop playing on the ATV, then pick-up where you left off on an iPad (on the plane) would be slick. That way we're not having to preface the use of save state syncing with "must be using the same version across all devices, etc".

[JoeMatt]

If you just want a 'fetch all' 'save all' for battery states we could probably use something like this,

https://github.com/amosavian/FileProvider

We would just need to make the UI for entering the service type, URL, user, pass.

The issue with the Battery saves that I see is that they're all just in a a folder called 'Battery Saves' by the filename of the ROM. They're not split into cores or renamed to the proper ROM titles in the database. Meaning even if two devices had the same ROM, identical MD5s, but the filename was different when it was imported, it will be difficult to match up the right folder.

We'd have to change the way Battery saves are saved into a more reproducible file structure, which would break existing Battery saves on upgrade (but hey it's free software things will break compatibility)

[GoldenChaos]

My two cents - from a UX perspective, syncing features like this need to be invisible and work flawlessly or they get annoying quick. Nobody wants to have to dismiss a “save state incompatible with this version of provenance” alert every time they start a game.

I also think syncing battery saves only is not what people would expect. Speaking personally, I would expect all game metadata to be synced - custom artwork, battery saves, favorite status, custom names, and save states. Plus I’d expect the UI to be updated with syncing indicators and graceful error handling so it didn’t get in the way of playing games.

I don’t think syncing is worth having unless it really can sync your entire library’s metadata and fulfill the dream of hopping from one device to another seamlessly. It’s going to be an tremendous undertaking either way so if you’re gonna sync one thing may as well figure out how to sync them all and do it right.

[jasarien]

Yeah, the whole import, database, saves and library UI needs a complete overhaul. It grew organically from when Provenance was just a Genesis front end. Now it’s a multi-emulator front end and is in dire need of refactoring.

[deejaywoody]

Either way, it’s a whole lot better than hauling around these “Classic” consoles everyone is raving about :)

On Feb 7, 2018, at 12:27 PM, James Addyman notifications@github.com wrote:

Yeah, the whole import, database, saves and library UI needs a complete overhaul. It grew organically from when Provenance was just a Genesis front end. Now it’s a multi-emulator front end and is in dire need of refactoring.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jasarien/Provenance/issues/1#issuecomment-363845344, or mute the thread https://github.com/notifications/unsubscribe-auth/AWl9SkrK0Jhyq-RNN3ZpqEFhnP8ZQy0oks5tSdz_gaJpZM4A8Vin.

[JoeMatt]

@GoldenChaos There's another ticket around here where I discussed the idea of a REST API to do just that. The meta-data is there, hell we already even have a web server built in, though there are dumb simple REST frameworks in Swift out there.

We could expose an basic URL structure like,

Example REST URL points

• cores/ {core_id, rom_types, version}

• cores/roms OR roms/core/core_id {list of ROM ids for that core in the database}

• roms/ {list of all ROM IDs}

• roms/romid {OpenPVDB name, custom name, file_size, md5, artwork_id, is_favorite}

• roms/romid?get {download rom for id}

• artwork/artwork_id {width, height, size}

• artwork/artwork_id?get download the artwork

• battery/rom/rom_id {list of battery id saves for from id}

• battery/battery_id {size, save_date, associated rom_id}

• /battery/batter_id?get download the battery

-/search/?md5=,name=,isFavorite=1,core= {return ROM id, or the whole ROM entry json, that matches the specified query} Doesn't even have to support compound queries, expect maybe for isFavorite & which should be a simple SQL query to the Realm database

I think that's about all you'd need to fully expose the ROM database that you could find everything you'd want to make a remote ROM browser by system with Artwork and downloads.

Though as @jasarien said, there would need to be some additional stuff store in the Provenance database. Right now there's no way to associated a battery with a ROM except with matching the file name, and vice-verse is worse. A battery file can only be associated with a ROM by searching all the core rom storage folder for a ROM that matches file name, would be tediously slow to match all batteries to their ROM in one shot.

Call to action for anyone looking to get involved

Here's how

Keep reading

That being said, I think it wouldn't be all that difficult to start storing that linking data. One nice thing is that right now, if you delete a ROM and re-add it, the battery will still be there. If you depend on a database, and the database gets out off sync with what's really on the filesystem, you're stick with filename matching again.

ROMs to Cores and Artwork are already basically all in the REALM db already though. You could start with just that for remote ROM streaming / downloading. If get this far I'll gladly finish the Battery save changes.

If you make it somewhat well structured to not be tightly linked to our REALM database, that is put in some abstraction layer between REST requests and SQL queries, I could probably port it over to OpenEMU so OpenEMU could become a desktop host of ROMs, Batteries and ArtWork since we we basically 95% of the same meta data and cores.

Network discovery is incredibly easy with Bonjour. There are plenty of easy examples of making and finding services by service name that you get to make up yourself. It's a small bit of code to publish and search for network services with Bonjour.

Any one with moderate Swift skills could plug in an OpenSource REST client/server framework, make some changes to how Batteries are saved to add them to the Realm database tied to the ROM that made it, and then export that data in the Realm database with each end point. The Swift based REST servers usually just have you supply and endpoint path and a function to handle calls that come into that path. A prototype could should be written in a weekend to at least get two Provenance apps talking to each other.

I encourage anyone who would like to take a crack at to just start looking at the Realm database we have now and see how it's structured do what you can from there. Link in an open source REST server, attach some calls backs to some end points to return some JSON data, hell make it up at first, and then load the URLs in a web browser and see the results. Once you see you can get that far pretty easily the rest won't be so bad.

The only difficult part is that we haven't actually used Swift in the project (yet) so Xcode will go about making changes to project to include bridging headers and such. If You want I can make a branch that has a dummy swift class that prints hello world at start, and has all the bridging stuff already tested and working.

[JoeMatt]

As an added bounty, if someone does write something that I posted above, I'll make a raspberry PI disk image that has everything setup that you need to to access Bonjour services via VPN from an iOS device.

Meaning you could buy a cheap, ultra low power, older model raspberry pi and small SD card ($25 or so), copy an image I make onto the SD card, edit a text file with a username and password you want to use, plug it into your network and then be able to stream games, batteries etc from all over the world via your own VPN, assuming you have an Apple TV with Provenance, or I add it into OpenEMU, or I could write a tiny headless app using our existing code that could be a small local rom host on any Mac in your home.

Macs and Bonjour also support WakeOnLan, so the services are continuously published by the local DNS server that would run on the raspberry pi and would be set as the DNS server when connected via VPN. Then if your MacBook for example is sleeping, it can wake up and serve the files and go back to sleep. I think this works over WiFi but the power cable might have to be plugged in is all. This is actaully way easier than it sounds. I have a lot of experience setting up Bonjour AFP / SAMBA file hosts and VPN on linux servers like Raspberry Pi. The Avahi open source project for Bonjour (also called Zero-Conf in Linux world) has been around forever.

I'd much rather do this than deal with supporting various 3rd party cloud services. And then even still you'd have to generate some kind of XML file or similar that would sit on the cloud server too that would have to be downloaded and parsed to get meta data about the files on the remote host. Might as well just build that into what we have already as a web service in the app itself (or light app that sits on a mac in the menu bar and re-use the web importing code or Finder file copying to do imports)

[QuarterSwede]

Also, don’t forget that ROMs and save states are backed up with iCloud backups. I just restored to a new device and it pulled those in. Custom ROM art doesn’t get backed up however.

I agree that syncing should be an all in behind the scenes affair if someone is willing to take it on. It would be a major benefit over using other emulators.

On Feb 7, 2018, 10:30 AM -0700, deejaywoody notifications@github.com, wrote:

Either way, it’s a whole lot better than hauling around these “Classic” consoles everyone is raving about :)

On Feb 7, 2018, at 12:27 PM, James Addyman notifications@github.com wrote:

Yeah, the whole import, database, saves and library UI needs a complete overhaul. It grew organically from when Provenance was just a Genesis front end. Now it’s a multi-emulator front end and is in dire need of refactoring.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jasarien/Provenance/issues/1#issuecomment-363845344, or mute the thread https://github.com/notifications/unsubscribe-auth/AWl9SkrK0Jhyq-RNN3ZpqEFhnP8ZQy0oks5tSdz_gaJpZM4A8Vin.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[sevdestruct]

Did Joe just describe Plex for ROMs? Badass.

[JoeMatt]

I added UIDocumentPickerViewController to the N64+Swift branch. This allow you to select documents from iCloud and other cloud services like Google Drive (though this didn't work for me for some reason, iCloud did).

I've tested it with roms, zips and bios files. Everything copies and imports smoothly. Still could use some tweaks but seems pretty good.

If this works well, might be able to add export and importing of save states

[leolelego]

As iCloud Drive not available on Apple TV. I've made some search and Cloudkit could be an easy solution for battery saves, but not Roms du to size of some roms (PSX). The integration is relatively simple, it's free (more or less), everybody with an iCloud account could use it and it's easy to setup when you build the app yourself (no need to add custom api-key in the app, just "enable" the function in the project settings). I can make some test for battery saves, but any suggestion, recommandation, opposition ?

[sevdestruct]

@leolelego: aucun progrès? beaucoup sont curieux de savoir ce qu'il y a dans votre cuisine.

[JoeMatt]

yeah i have an idea for roms anyway, i already expanded the database to have a concept of 'Library' and can have multiple, online and offline, and games have the same so in the browser we can show roms that are offline in storage, and then if you import them it'll update the rest of the meta data can sync though.

Another option is using Realm's cloud sync. They have a way to self host it, I started playing around with that a little. Not sure how user accounts are handled though.

And then a way to host the files. Maybe we could host that too, since the files are small. Maybe a paid feature...

[elaine-jackson]

@JoeMatt How would in-app purchases be handled when Apple doesn't allow emulators?

[JoeMatt]

Wouldn't use IAP. Accounts would need to be created in a web app. I haven't looked much further into how cloud based realm works, their docs for self hosting are, maybe purposefully, thin.

[elaine-jackson]

@JoeMatt If you guys need someone to develop a web application for managing accounts & payments, etc I'd be willing to help with that.