farmerbb commented 3 years ago

Overview

Over the past few years, Google has been rolling out changes in the way Android facilitates the access of shared files, called scoped storage. Basically, Google is no longer allowing blanket access to the shared filesystem and instead is requiring the usage of the Storage Access Framework to grant file/directory access to apps on a case-by-case basis.

RetroArch will be greatly affected by this change as it currently assumes that it has blanket access to the filesystem. We are currently targeting SDK 29 and opting out of scoped storage by using the android:requestLegacyExternalStorage manifest attribute, but this is not a sustainable solution in order for us to stay on the Play Store.

This bounty is for implementing support for the Storage Access Framework in order to solve for the following issues:

Users on Android 11 are currently unable to view their SD cards in RetroArch's file browser. Implementing SAF would allow these users to access their SD card again inside of RetroArch.
On November 1st, 2021, apps that do not comply with scoped storage can no longer be updated in the Google Play Store. This will be enforced by Google changing the minimum allowed targetSdkVersion to 30. We are basically required to implement SAF in order to maintain a presence on the Play Store.

Possible implementation approach

There are two main items that I can think of that will need to be addressed:

RetroArch system folder migration

RetroArch's system folder, /storage/emulated/0/RetroArch, will no longer be accessible once the app targets SDK 30. We can gain access to the folder back by making a call to the Storage Access Framework.

On app startup, if the user hasn't granted access to the system folder yet, open Android's native directory picker via the Intent.ACTION_OPEN_DOCUMENT_TREE intent and ask the user to select the "RetroArch" folder on their internal storage. From here, we can take one of two approaches:

Take persistable URI permissions and read/write to the system folder directly. This would probably require some sort of shim to make files/directories within the SAF API usable within native C code. This would also probably incur a performance penalty due to SAF being so darned slow at file operations.

or:

Migrate the system folder to the /storage/emulated/0/Android/data/com.retroarch/files directory. This negates the need to use SAF to access the system folder beyond the initial migration. It wouldn't require a shim and won't incur a performance penalty, but, due to another change in Android 11 where the Android/data folder is made inaccessible to most file managers, might cause frustration to the user when trying to add files such as BIOSes to that directory.

File Browser

RetroArch's file browser won't be able to freely traverse the file system anymore under scoped storage. One possible solution is to implement the ability to add arbitrary directories to the top level of the file browser, along with a button to add a directory to the list. Directories on the list would correspond with directories that the user has selected via Android's native picker.

The flow would go something like this:

Add a button somewhere in RetroArch's UI to add a new location to the file browser.
When the button is pressed, open Android's native directory picker via the Intent.ACTION_OPEN_DOCUMENT_TREE intent
Take persistable URI permissions, and keep track somewhere (possibly SharedPreferences) of the directories that the user has picked
Enumerate the list of document trees inside the RetroArch file picker UI

Reference implementations

The standalone Dolphin, melonDS and PPSSPP emulators are either in the process of and/or have already solved for this issue. Note that, at least in the case of Dolphin and melonDS, these apps use a Java/Kotlin based frontend instead of a pure C/C++ frontend like RetroArch does (only minimal amounts of Java code are included in the project). But this should hopefully serve as a reference point on what other retro gaming projects have done to deal with scoped storage.

Miscellaneous notes

Stack Overflow post on how to get a file descriptor from SAF in order to pass into native code: https://stackoverflow.com/a/31677287

Official scoped storage example project from Google (unfortunately, doesn't use any native code): https://github.com/android/storage-samples/tree/main/ScopedStorage

Kotlin library for making SAF usage a lot less painful, might be good for reference: https://github.com/K1rakishou/Fuck-Storage-Access-Framework (pardon the language)

plattysoft commented 3 years ago

For managing the internal files, I think relying on getExternalFilesDirs() would suffice, since it is an external directory that can be accessed with File Broweser apps. There would need to be a migration from the current external directory thou.

For the file browser part, maybe the right solution is to request access to all files: https://developer.android.com/training/data-storage/manage-all-files IMHO it. falls under the use case of "On-device file search"

inactive123 commented 3 years ago

Me and @farmerbb are going to be tackling this one.

Maingron commented 2 years ago

I just randomly saved, edited and imported the config file. I had the idea of just setting the start directory to my ROM directory on the SD card.

... It works 😳

I'm on Android 11 on a Samsung Galaxy S20+. I moved RetroArch to my SD card beforehand, so I haven't tested if that would still work if RetroArch is saved on internal storage.

My magic line, though it will be different for you: rgui_browser_directory = "/storage/6C0E-290F/ROMS"

Maybe it would be a good temporary solution to gain SD card access by just adding a virtual directory "SD-Card" which basically scans for potential SD Cards and opens them.

hizzlekizzle commented 2 years ago

Yes, this is indeed one of the workarounds.

andres-asm commented 2 years ago

RetroArch's system folder, /storage/emulated/0/RetroArch, will no longer be accessible once the app targets SDK 30. We can gain access to the folder back by making a call to the Storage Access Framework.

On app startup, if the user hasn't granted access to the system folder yet, open Android's native directory picker via the Intent.ACTION_OPEN_DOCUMENT_TREE intent and ask the user to select the "RetroArch" folder on their internal storage. From here, we can take one of two approaches:

    Take persistable URI permissions and read/write to the system folder directly. This would probably require some sort of shim to make files/directories within the SAF API usable within native C code. This would also probably incur a performance penalty due to SAF being so darned slow at file operations.

or:

    Migrate the system folder to the /storage/emulated/0/Android/data/com.retroarch/files directory. This negates the need to use SAF to access the system folder beyond the initial migration. It wouldn't require a shim and won't incur a performance penalty, but, due to another change in Android 11 where the Android/data folder is made inaccessible to most file managers, might cause frustration to the user when trying to add files such as BIOSes to that directory.

For this part I can just use media, that is world readable, or I can get permissions to /sdcard/RetroArch, that is cake

farmerbb commented 2 years ago

For this part I can just use media, that is world readable, or I can get permissions to /sdcard/RetroArch, that is cake

MediaStore only surfaces photos/videos/music/etc, we can't read arbitrary file types using it. Also granting permissions to /sdcard/RetroArch via SAF would still require the app to read that directory using SAF / DocumentFile APIs, so there would still need to be a VFS shim put in place to translate the DocumentFile API into something RetroArch can understand.

andres-asm commented 2 years ago

@farmerbb from my testing, once you get permissions you can just read the files no problem. I already achieved that even though it's supposedly impossible.

This PR has code that makes it load using an FD, but after that and going through some more hoops, I managed to translate that into the filename of the actual file. Tried to load and it bombed.

So I went back to my initial code that opens the filebrowser to grant access to that particular folder. After doing that it definitely works!

I guess I can change the API target and test this in an app targetting API level 30.

I have googled a lot, and seems NDK + SAF is a nightmare but this workaround works, even for fullpath cores.

andres-asm commented 2 years ago

@farmerbb I didn't mean the medistore API, /sdcard/Android/media is world readable, so we can switch to that location, that's shared storage.

I was the one who made it use /sdcard/RetroArch in the first place because going to /sdcard/Android/data/com.files to edit any file was a chore. I have actually always wanted to move the config outside of that but never did. If this workaround works perhaps I should do that too.

VA1DER commented 2 years ago

Maybe it would be a good temporary solution to gain SD card access by just adding a virtual directory "SD-Card"

A good solution is to just allow the user to specify directories by name. Even in Android 12 you can still access the extsdcard. You just can't navigate there due to the poor folder selection UI that doesn't have a method of typing in a name.

Vinfall commented 2 years ago

Any update on this?

The latest nightly build as of writing (2022-07-13, aka 1.10.3_GIT (1657707401)) still targets API 28. I can see that a contributor picks this up in #13538. And from #13615 I guess the app itself already gets rid of MANAGE_EXTERNAL_STORAGE?

andres-asm commented 2 years ago

Maybe it would be a good temporary solution to gain SD card access by just adding a virtual directory "SD-Card"

A good solution is to just allow the user to specify directories by name. Even in Android 12 you can still access the extsdcard. You just can't navigate there due to the poor folder selection UI that doesn't have a method of typing in a name.

After #13615 you should be able to browse the sdcard assuming you have permissions

andres-asm commented 2 years ago

Any update on this?

The latest nightly build as of writing (2022-07-13, aka 1.10.3_GIT (1657707401)) still targets API 28. I can see that a contributor picks this up in #13538. And from #13615 I guess the app itself already gets rid of MANAGE_EXTERNAL_STORAGE?

It's a nightmare situation, because it's not just RetroArch, but all the cores. I implemented some building blocks to get permissions on #13538 but I don't know if that is enough

andres-asm commented 2 years ago

@farmerbb for the system folder, I think the best solution is to switch this over from

/storage/emulated/0/RetroArch -> for user data
/storage/emulated/0/Android/data/com.retroarch/files -> for the main config

to

/storage/emulated/0/Android/media/com.retroarch/files

Of course this will break everyone's setup, but it should be fine after

LibretroAdmin commented 2 years ago

@farmerbb We really need to return to this. To date, we have not been able to push a new update to Google Play for over a year. This is concerning and we definitely need to get back on this.

farmerbb commented 2 years ago

I don't have the bandwidth to take this on, things have changed a lot for me over the past year and I no longer have the time or capacity to devote to projects like this (requires a lot of things I lack the expertise for, such as interfacing with VFS, the C / C++ / JNI side of things, etc.)

This issue was set up as a bounty, perhaps that fact could be publicized in the hopes that other potential contributors could step up and take this on.

davidhedlund commented 3 months ago

Based on my research, the functionality of RetroArch on mobile devices connected to a USB hub with Power Delivery (PD) and USB ports can vary depending on the specific device and the version of the software being used:

Android 10 smartphone

Works with all current stable versions: Versions 1.9.0 (https://buildbot.libretro.com/stable/1.9.0/android/RetroArch.apk) and later (including those released on F-Droid): When selecting "Load Content," and then selecting "/storage/", "[XXXX-YYYY]" is accessible, and I've played games from it.

Android 13 smartphone

The "Load Content" option displays the path "/mnt/media_rw/[A1B2-C3D4]," but attempting to access it results in a "Directory not found" error.

Google TV 12 (Chromecast with Google TV HD)

Works with: Versions 1.10.1 (https://buildbot.libretro.com/stable/1.10.1/android/RetroArch.apk) and later (including those released on F-Droid): When selecting "Load Content," "/storage/[XXXX-YYYY]" is accessible, and I've played games from it.
Does not work with: Versions 1.10.0 and older (for example the current release on Google Play: RetroArch 1.9.12 (2021-11-03))

libretro / RetroArch

[Bounty] [Android] Implement support for Storage Access Framework #12181

Overview

Possible implementation approach

RetroArch system folder migration

File Browser

Reference implementations

Dolphin

melonDS

PPSSPP

Miscellaneous notes

Android 10 smartphone

Android 13 smartphone

Google TV 12 (Chromecast with Google TV HD)