Android 11, Can't access /storage/emulated/0/android/data

[SudosFTW]

App isn't fully capable of the scoped storage API calls for accessing said directory as a file manager.

• Device: Pixel 3a
• OS: Android 11
• Rooted: No
• Version: 3.5.0

I can't access files saved with apps like Telegram which store downloads in such a location. They don't even show up as recent files in anything else but the built-in file manager in Android (not google files, I have that disabled)

There should be a n actual API call specifically for file managers to access that location but as I'm not completely savvy on the inner workings of modern Android APIs I don't exactly know what it's called. This should probably be looked into heavily when building for Android 11 since the scoped storage API is mandatory now, and it should be seamless to Android 10 users already using the app.

[VishalNehra]

@SudosFTW I'd like to +1 to this... I'm expecting comments from @TranceLove here as he worked on the current implementation.

[TranceLove]

Best case scenario, you would want to ditch java.io.File at all and use content:// URI to even internal/external storages, so you won't need to maintain 2 sets of APIs.

But this is too a big task (that I may want to Kotlintize and refactor the whole HybridFile system) that perhaps too big to fit in 3.5 timeline as well.

[VishalNehra]

@TranceLove I mean, no need to ditch File API at all.. just ways to fix the Android/data/ directories not being accessible right now. Even when we're explicitly giving Read / write permissions with the Uri.

[TranceLove]

As stated in https://developer.android.com/about/versions/11/privacy/storage:

Access to app-specific directories on external storage

On Android 11, apps can no longer access files in any other app's dedicated, app-specific directory within external storage.

Although, there were cases claimed that java.io.File APIs can still access these storage without problems. We probably need some kind of fallback mechanism so that when accessing files using content:// URI fails, we may try using java.io.File instead.

Meanwhile, may try to see if adding android:requestLegacyExternalStorage="true" to AndroidManifest.xml can be a band-it fix.

Will try make a case for this. Though getting Android 11 scoped storage work with Amaze is still a long battle...

[TranceLove]

Tried some little tests today with Pixel 4 emulator running stock Android 11. Even with opt-out of scoped storage with the 2 methods

• no actions at all (compile with SDK 28, no extra flag in AndroidManifest.xml), that user will see Amaze is allowed to manage all files

• adding android:requestLegacyExternalStorage="true" and upgrade SDK to either 29 or 30 (which is required by setting this flag)

contents of data directory is still not visible. Not other apps too - only Android's own DocumentUI can get through.

Though some point in the future we shall be asking Google for MANAGE_EXTERNAL_STORAGE permission via app review anyway, I'm afraid users of Android 11 not using Google Play may be left behind unless Google reaches some kind of agreements with third-party app stores, since this permission shall only benefit apps distributed by Google Play.

Also check https://developer.android.com/training/data-storage for a quick table of file accessibility in Android 11.

[VishalNehra]

Related crashes

Rename:

2020-11-16 03:34:28.529 5634-5935/com.amaze.filemanager.debug E/ActivityThread: Rename recovery failed 
    android.system.ErrnoException: rename failed: EXDEV (Cross-device link)
        at libcore.io.Linux.rename(Native Method)
        at libcore.io.ForwardingOs.rename(ForwardingOs.java:190)
        at libcore.io.BlockGuardOs.rename(BlockGuardOs.java:350)
        at libcore.io.ForwardingOs.rename(ForwardingOs.java:190)
        at android.app.ActivityThread$AndroidOs.rename(ActivityThread.java:7581)
        at java.io.UnixFileSystem.rename(UnixFileSystem.java:368)
        at java.io.File.renameTo(File.java:1412)
        at com.amaze.filemanager.asynchronous.asynctasks.MoveFiles.doInBackground(MoveFiles.java:109)
        at com.amaze.filemanager.asynchronous.asynctasks.MoveFiles.doInBackground(MoveFiles.java:59)
        at android.os.AsyncTask$3.call(AsyncTask.java:394)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:923)

Copy:

2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err: java.io.FileNotFoundException: /storage/emulated/0/Android/data/DummyFile: open failed: EACCES (Permission denied)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at libcore.io.IoBridge.open(IoBridge.java:492)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at java.io.FileOutputStream.<init>(FileOutputStream.java:236)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.filesystem.FileUtil.isWritable(FileUtil.java:754)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.utils.MainActivityHelper.checkFolder(MainActivityHelper.java:420)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.asynchronous.asynctasks.PrepareCopyTask.finishCopying(PrepareCopyTask.java:325)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.asynchronous.asynctasks.PrepareCopyTask.onEndDialog(PrepareCopyTask.java:275)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.asynchronous.asynctasks.PrepareCopyTask.onEndDialog(PrepareCopyTask.java:269)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.asynchronous.asynctasks.PrepareCopyTask.onPostExecute(PrepareCopyTask.java:186)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.amaze.filemanager.asynchronous.asynctasks.PrepareCopyTask.onPostExecute(PrepareCopyTask.java:64)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.os.AsyncTask.finish(AsyncTask.java:771)
2020-11-16 03:34:56.761 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.os.AsyncTask.access$900(AsyncTask.java:199)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.os.AsyncTask$InternalHandler.handleMessage(AsyncTask.java:788)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.os.Handler.dispatchMessage(Handler.java:106)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.os.Looper.loop(Looper.java:223)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.app.ActivityThread.main(ActivityThread.java:7656)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at java.lang.reflect.Method.invoke(Native Method)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err: Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at libcore.io.Linux.open(Native Method)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at libcore.io.ForwardingOs.open(ForwardingOs.java:166)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at libcore.io.BlockGuardOs.open(BlockGuardOs.java:254)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at libcore.io.ForwardingOs.open(ForwardingOs.java:166)
2020-11-16 03:34:56.762 5634-5634/com.amaze.filemanager.debug W/System.err:     at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7542)
2020-11-16 03:34:56.763 5634-5634/com.amaze.filemanager.debug W/System.err:     at libcore.io.IoBridge.open(IoBridge.java:478)
2020-11-16 03:34:56.763 5634-5634/com.amaze.filemanager.debug W/System.err:     ... 17 more

[danielzgtg]

On Android 11

No app on Android 11 is supposed to be able to access /sdcard/Android/data/ no matter the mechanism. If you are able to, then you should report such a security vulnerability to Google.

only Android's own DocumentUI can get through

The built-in OS one might be able to read, but not even it has write access

Rooted: No

Unfortunately, I don't think it will ever be possible to access that without root.

I am in a rooted environment, and it still doesn't work normally. However, I found that a workaround is to use /data/media/0/Android/data instead of /sdcard/Android/data. It would be nice if Amaze could detect whether the device is on Android 11 and viewing /sdcard/Android/data and automatically use root to access that

[kuzmeech]

My Samsung Note 10+ updated to Android 11 and I also lost access to Android/data, however X-plore File Manager 4.24.15 and later is able to access it somehow on non rooted device (https://www.reddit.com/r/Android/comments/j3zgmm/managing_files_in_the_androiddata_folder_on/)

It gets into some system dialog

and then here:

and after pressing "use this folder" and "allow" I'm able to browse Android/data contents.

Update 20210130-221654 EST : I guess the sentiment in the reddit topic rather votes for the freedom of user's choice as explicit permission is needed.

I need this access to be able to copy video files to my Insta360 app directly instead of using their inconvenient and only method of making a wifi bridge between the camera and phone and pumping these gigabytes for hours, camera overheats. I needed a way to place files there directly.

In addition I found other file managers are able to access and modify it - such as Total Commander. I don't want this flexibility taken away from me as long as I consciously allow this access on my device.

[danielzgtg]

Reported: https://issuetracker.google.com/issues/178910699

[kuzmeech]

Thank you but I'd rather like Amaze to support it too like Total Commander and X-plore but as I've learned your opinion - you don't seem to support my point even given my described legitimate reason to need this.

I could be wrong but some mentioned using newer SDK makes it possible.

Still the users should be warned not to open it unnecessarily - this permission is only enabled when I navigate under Android/data and not sooner. Average non advanced users would never go into this dir.

Thanks again!

[VishalNehra]

Thank you but I'd rather would like Amaze to support it too like Total Commander and X-plore but as I've learned your opinion - you don't seem to support my point even given my described legitimate reason to need this.

I could be wrong but some mentioned using newer SDK makes it possible.

Still the users should be warned not to open it unnecessarily - this permission is only enabled when I navigate under Android/data and not sooner. Average non advanced users would never go into this dir.

Thanks again!

Hey, no one said we're not going to support. If anything, this is really informative for us. Otherwise we would've merged the hack defined by #2246 I see you're using Insta360, i was planning to get one. I can feel the need, I think we'll try to fasttrack this issue and fix it asap. @TranceLove please take a look on above. I think this should be the correct way instead of using root 🤔

[TranceLove]

I suppose it's using content:// URI to work around the restriction? Can't say it's a correct way if blocking access to Android/{data,obb} was their first intention... grey area, workaround, trick, things like that.

Indeed a more welcomed path since no root is required, but expect a bigger refactoring for enabling this...

[VishalNehra]

Indeed a more welcomed path since no root is required, but expect a bigger refactoring for enabling this...

How bigger refactoring? Aren't we supporting file listing through content uri?

[danielzgtg]

[...] content:// [...] welcomed path [...]

I also support this, for the different reason that non-root privacy-minded users would become able to limit their folder selection to only the folders they want to use with Amaze.

How bigger refactoring [...] content uri?

No matter how big the refactoring is, I think it needs to be done sooner or later. This is because we don't know when Google will randomly change its mind again (as seen in Android 10 and 11) and throw another monkey wrench into non-SAF storage access

[TranceLove]

No matter how big the refactoring is, I think it needs to be done sooner or later. This is because we don't know when Google will randomly change its mind again (as seen in Android 10 and 11) and throw another monkey wrench into non-SAF storage access

Had some little attempt on making Amaze use DocumentFile instead of java.io.File even on local storages. But the scale is too big IMO, that I expect this shall be a goal for 4.0... and probably the only way to make Amaze away from Google as much as possible.

[VishalNehra]

Thanks @TranceLove for checking. Moving this to 4.0 Suggest to take down the PR #2246 for now

[EmmanuelMess]

Can we divide the refactoring into simpler steps? Maybe make separate project for it?

[TranceLove]

@EmmanuelMess In my vision the "refactoring" would be like rewriting the core in entirely DocumentFile, then putting everything back to become 4.0...

Not welcomed... but seems the cleanest approach🤔

[VishalNehra]

@TranceLove Anyways, I'm already working on migrating MainFragment code responsible for load list task to viewmodel, maybe incorporate these changes after they are done. It'll be easier then.

[danielzgtg]

cleanest approach

simpler steps

I can think of a much dirtier but simpler solution that can be used at first. We can replace all references to java.io.File and related things to a new class, say com.amaze.filemanager.VirtualFile. Then we emulate the API and map all the selected folders as /drive/c, /drive/d, etc. just like Wine. This would leave all the UI and file management code intact. That would only require UI code to pick the folders, this would take much less work and time (although add a lot of debt)

[EmmanuelMess]

We can replace all references to java.io.File and related things to a new class, say com.amaze.filemanager.VirtualFile. Then we emulate the API and map all the selected folders as /drive/c, /drive/d, etc. just like Wine.

I am in favor of a solution like this, if it is deemed workable, although I don't think it can't be applied.

The file needs to be on the file_operations module though.

[jbro129]

I came across these while trying to implement an Android 11 /android/data folder workaround for my own app. I couldn't find any open source code or snippets to do this so I resulted to decompiling apps that can using Jadx.

The two apps that I found that have this workaround are X-plore (as mentioned in this thread) and Solid Explorer.

X-plore use proguard for obfuscation while Solid Explorer does not. This makes decompiling and searching through the X-plore code much more difficult than doing so in Solid Explorer.

I don't know if this can be helpful, but I hope it can be! :)

Edit - removed objectionable content Edit2 - I see some of the info I provided got removed by a member of the Amaze Team. I didn't think it was objectionable as its not to copy its code but to study it, but I respect your decision and still hope it helped.

[TranceLove]

@jbro129 Did tried this approach a bit... can't talk too much about this, but did learned something how and what we can do 🙃

[AndroidDeveloperLB]

Has anyone here found how to access this folder via code? I've noticed that Mixplorer also has this ability, and Total Commander has it too (though need to be quite manual about it).

[danielzgtg]

this folder

I haven't decompiled, but I don't see why the code for accessing this folder would be special in comparison to accessing any other folder.

The official documentation (sadly without examples) about how to access a folder is available at https://developer.android.com/reference/android/content/Intent#ACTION_OPEN_DOCUMENT_TREE

[TranceLove]

Made a proof-of-concept, using code from Android's SAF examples.

https://github.com/TranceLove/ActionOpenDocumentTree

For a quick fix, may repeat #2246 with code we use for OTG. Things can get really dirty though...

EDIT: as said in their docs

You can no longer use the ACTION_OPEN_DOCUMENT_TREE or the ACTION_OPEN_DOCUMENT intent action to request that the user select individual files from the following directories:

• The Android/data/ directory and all subdirectories.
• The Android/obb/ directory and all subdirectories.

Being a bad guy (to those who still need access to these directories without root) I wonder if @danielzgtg's issue report to Android was indeed legit?

[AndroidDeveloperLB]

@TranceLove Seems this shows how to reach Android/data, but how can you reach Android entirely, like the file manager apps do?

[TranceLove]

@AndroidDeveloperLB Normal java.io.File API should do. Although, I prefer uses DocumentFile based APIs to navigate to other directories too.

My example is to show the intent to build, and the use of initial URI to directly navigate to the folder, instead of asking users to navigate to the directory by hand. It shouldn't affect the other parts at all.

[AndroidDeveloperLB]

@TranceLove I understand that you used SAF alone, but you did it only for Android/data. Also, I tried now to reach the same folder using the same code you've used, but using File class, and it failed. It gave me null as a value for when I used listFiles() on it. Plus, when trying to use Android instead of Android/data using the logic of what you did, it failed to get me all children files using SAF of Android folder. Weird thing is that in this case, I could see the "media", "obb" and "data" folders using File class, but only "media" when using SAF.

See attached. SafSample.zip

[TranceLove]

@TranceLove I understand that you used SAF alone, but you did it only for Android/data. Also, I tried now to reach the same folder using the same code you've used, but using File class, and it failed. It gave me null as a value for when I used listFiles() on it. Plus, when trying to use Android instead of Android/data using the logic of what you did, it failed to get me all children files using SAF of Android folder. Weird thing is that in this case, I could see the "media", "obb" and "data" folders using File class, but only "media" when using SAF.

See attached. SafSample.zip

Yes, exactly same as what I saw, both java.io.File and the SAF cases. So it implies we may not use only DocumentFiles but a mix of java.io.File and SAF that will switch OpenMode automatically underhood.

Change Android/data to Android/obb will give you other directory... just too lazy to do both. And the value was derived after watching how DocumentUI and Total Commander interact over Intents.

[AndroidDeveloperLB]

@TranceLove You don't understand. While your method does allow me to reach the Android path to request access to it, there is nothing that I can do to reach the content of Android/data or Android/obb after that, as opposed to what I saw on file manager apps like mixplorer .

There, once you grant access to the Android folder, you can see everything in it. Even if File class tells me there is the "data" folder inside (and it does), it doesn't help me, because trying to reach it via SAF doesn't work (using documentFile.findFile("data") will return null), and trying to get its content using File class doesn't work.

Here, made a new sample that target API 30, like mixplorer does (previous sample targeted API 29) : SafSample.zip

[AndroidDeveloperLB]

If anyone wishes, it seems to only work for Android 11: https://stackoverflow.com/questions/65967690/how-do-some-apps-reach-the-contents-of-android-sub-folders-on-android

Sadly it doesn't seem to work on Android 12, and I can't find any file manager app that can reach the folder anymore ...

[VishalNehra]

If anyone wishes, it seems to only work for Android 11: https://stackoverflow.com/questions/65967690/how-do-some-apps-reach-the-contents-of-android-sub-folders-on-android

Sadly it doesn't seem to work on Android 12, and I can't find any file manager app that can reach the folder anymore ...

How can you say it's not working on android 12? Did you accept the permission for folder at Android/data from DocumentUI screen?

[AndroidDeveloperLB]

@VishalNehra I can say it because I have Pixel 4 with Android 12 beta 5, and it failed on all file managers that used to be able to allow it: Total Commander, X-plorer, and Mixplorer. I'm the one that created the post. I thought this workaround would stay for longer. I was (sadly) also been informed it was "fixed". :(

[EmmanuelMess]

@VishalNehra I can say it because I have Pixel 4 with Android 12 beta 5, and it failed on all file managers that used to be able to allow it: Total Commander, X-plorer, and Mixplorer. I'm the one that created the post. I thought this workaround would stay for longer. I was (sadly) also been informed it was "fixed". :(

Please open a new issue for Android 12.

[AndroidDeveloperLB]

@EmmanuelMess OK here: https://issuetracker.google.com/issues/200699802

Please consider starring.

I've posted about it in the past, but sadly doesn't seem like Google will do it: https://issuetracker.google.com/issues/192637587 https://issuetracker.google.com/issues/155215180

[VishalNehra]

@VishalNehra I can say it because I have Pixel 4 with Android 12 beta 5, and it failed on all file managers that used to be able to allow it: Total Commander, X-plorer, and Mixplorer. I'm the one that created the post. I thought this workaround would stay for longer. I was (sadly) also been informed it was "fixed". :(

Ok i tried on android 12 emulator. The document ui even doesn't show android/data or Android/obb for me to select them. So this is a miss from Google's end, I'm assuming this will be fixed by the time 12 reaches production.

[AndroidDeveloperLB]

Wait, why is this re-opened? Do you guys know of a workaround for Android 12 ?

[VishalNehra]

For android 12 will create a new issue later. Reopened this as I broke file / folder create functionality for normal folders while fixing for android/data. Will fix this as a part of this issue only.

[AndroidDeveloperLB]

@VishalNehra Do you think there is a solution for Android 12?

[TranceLove]

@AndroidDeveloperLB I cannot think of one, if others cannot do so without going through the Google Play review too.😕

[VishalNehra]

@VishalNehra Do you think there is a solution for Android 12?

I'll try to look for some way later, will update here. The android 12 images are quite laggy on my Macbook.

[VishalNehra]

Note to self - Our existing logic to create file / directory was, build newPath = oldPath + newFileName, create directory on this newPath. This however, doesn't work in case of DocumentFile in Android 11, as when we try to iterate to newPath it'll throw an exception saying the newPath is not valid. So we have to modify our workflow to include oldpath while creating file / directory instead of utilizing just the newPath (getting newPath.parent doesn't work either).

content://com.android.externalstorage.documents/tree/primary%3AAndroid%2Fdata/document/primary%3AAndroid%2Fdata%2Fcom.activision.callofduty.shooter%2Ffiles

content://com.android.externalstorage.documents/tree/primary%3AAndroid%2Fdata/com.activision.callofduty.shooter/files

[HasinduLanka]

If your phone updated from Android 10 to 11 recently and amaze can't access these folders, uninstalling amaze and installing it again would fix the issue