Storage updates in Android 11

[HarelM]

Feature Request

Support the android 11 new file system based policy.

Motivation Behind Feature

Support the changes that are planned for android 11 (API level 30)

Feature Description

I'm not sure how to fully describe this, but basically support what's documented here: https://developer.android.com/about/versions/11/privacy/storage

Alternatives or Workarounds

Not targeting API level 30 until this is supported?

[HarelM]

Seems like the fix that was merge #417 in order to solve #408 will no longer work for android 11 (API level 30). I'm not entirely sure I understand what needs to be done in android in order to get access to the file system, something like opening a file picker in order to get write permissions to a folder? I'd love to help progress this, maybe send a PR if someone can guide me to the relevant place in the code and define the appropriate solution...

[breautek]

Seems like the fix that was merge #417 in order to solve #408 will no longer work for android 11 (API level 30).

Yes, you're correct. In API 29, there is a new file system policy. #417 utilised an available flag to request the old file system system, but API 30 will ignore that flag, forcing app developers to implement the new policy model.

To learn about the new android model, you should probably read up on Scoped Storage. I briefly read up on this last weekend and it appears that some apps may require to migrate their data locations depending on where they store data. Scoped storage means just that, you're access to the filesystem is limited to a certain scope. Parts of the file system that you might have had access before may be impossible to access in API 30. This means app developers should take this time to determine if they need to migrate data. Read the android notice for more information on migration. Given the usual pattern, users have probably until August 2021 before Google force API 30 for new apps, and November 2021 before users are forced to use API 30 on existing apps.

The new MANAGE_EXTERNAL_STORAGE permission I believe will essentially make the app behave pre API 29 without the legacy storage flag, but will likely be highly scrutinised by Google if the app uses it without a justified reason. This could be an opt in feature by documenting how one can add this permission using config-file or edit-config. The users must be targeting API 30 in order to use this permission, it isn't available in API 29.

Lastly, getting familiar with the MediaStore APIs sounds like is quite important, because apps can access some external filesystem storages, for accessing pictures, videos, and other media-related resources.

What needs to be changed exactly in the Apache codebase I'm not so certain myself, I'm not that familiar with the code base.

[olsbg]

Adding ' android:requestLegacyExternalStorage="true"' to AndroidManifest.xml solves the problem.

Read more at https://developer.android.com/training/data-storage/use-cases.

[HarelM]

@olsbg this won't work for android 11, it does and will work for android 10. I've migrated my entire code that saves file to the external file system to share files that are created in memory to avoid any future limitations...

[Vatsov]

@olsbg this won't work for android 11, it does and will work for android 10. I've migrated my entire code that saves file to the external file system to share files that are created in memory to avoid any future limitations...

Can you share how you achieved this?

[HarelM]

Instead of saving files to the file system I use the social sharing plugin. This way a user is prompted with options on what he wants to do with the file I just created in memory: https://github.com/IsraelHikingMap/Site/blob/4e479a5a92051ae5ec84136ca8c20547a4d11f26/IsraelHiking.Web/sources/application/services/file.service.ts#L152

[Vatsov]

Thanks, @HarelM , Unfortunately this won't work in my case because I'm trying to read a file

[HarelM]

@Vatsov Are you reading a file a user selects? If so I don't think there's an issue that is related to Android 11, or at least I hope not as you open the file picker and this should be OK even in Android 11 as far as I know. The other way I solved the file open is by registering my app to open certain file types and allowing the user to share these files with the app. https://github.com/IsraelHikingMap/Site/blob/4e479a5a92051ae5ec84136ca8c20547a4d11f26/IsraelHiking.Web/config.xml#L45 And: https://github.com/IsraelHikingMap/Site/blob/4e479a5a92051ae5ec84136ca8c20547a4d11f26/IsraelHiking.Web/sources/application/services/open-with.service.ts#L85 Hope that helps... :-)

[Vatsov]

@HarelM Yes, I’m trying to read the file which the user selects with this.fileChooser.open() The interesting thing is that this.file.checkFile returns true, but when I try with this.file.readAsText it returns null

So I decided firstly to copy the file to this.file.dataDirectory and after that to read it but the this.file.copyFile but returns FileError {code: 1, message: "NOT_FOUND_ERR"}

[smartyw]

FYI

window.resolveLocalFileSystemURL(cordova.file.externalRootDirectory + "/Download" used to work but now fails with Code 1 on Android 10 both without specifying android:requestLegacyExternalStorage="true" and with. I can't test on Android 11.

It would be really useful to know if there's a plan for the maintainers to create a new release which makes the plugin work with Android 10 and 11 without the need to manually tweak files like the manifest. Hope there is. This is a really useful plugin.

Thanks

[breautek]

What should work

window.resolveLocalFileSystemURL(cordova.file.externalRootDirectory + "/Download" used to work

• This should work as is when targeting API 28 (now forbidden by Google Play)
• This should work when targeting API 29 with the dev version of the plugin (which has the android:requestLegacyExternalStorage="true"), alternatively you can use the edit-config to add this flag.
• This won't work when targeting API 30, as API 30 ignores the android:requestLegacyExternalStorage attribute.

It's important to read the Android Notes before you target API 30. You may need to migrate your files to another folder to maintain access to them when targeting API 30 using the new APIs.

Who this will affect

This will affect any app that reads or writes to the external file system (aka the cordova.file.external*). If you don't use external file paths, then you probably don't need any changes in your app.

I'll be adding a help wanted label. We don't have a planned solution yet. If a volunteer would take the lead and finding a solution to prepare a PR, I encourage to use our Dev Mailing List so that we can discuss requirements and potential solutions.

Useful Links: https://developer.android.com/about/versions/11/privacy/storage

[smartyw]

@breautek Thanks for the quick response and useful summary. I'll have to think about this :-)

[HarelM]

I've recently implanted a small android app which had to save images to a folder/media storage so I might publish my conclusions here once I'm done...

[jfoclpf]

window.resolveLocalFileSystemURL(cordova.file.externalRootDirectory + "/Download" used to work

• This should work when targeting API 29 with the dev version of the plugin (which has the android:requestLegacyExternalStorage="true"),

Did you mean running cordova plugin add https://github.com/apache/cordova-plugin-file.git Will be this backward compatible for API <= 28 also?

[breautek]

Will be this backward compatible for API <= 28 also?

Yes, but you still need to use build tools version 29, otherwise the build will fail to recognize android:requestLegacyExternalStorage.

[jfoclpf]

Great, thanks, indeed I made

cordova plugin rm cordova-plugin-file
cordova plugin add https://github.com/apache/cordova-plugin-file.git

I tested it now and it allows me to use cordova.file.externalRootDirectory + "/Download" in Android 10 (API 29)

Yes, but you still need to use build tools version 29, otherwise the build will fail to recognize android:requestLegacyExternalStorage.

Did you mean this in config.xml?

<preference name="android-targetSdkVersion" value="29"/>

Thanks in advance

[breautek]

I tested it now and it allows me to use cordova.file.externalRootDirectory + "/Download" in Android 10 (API 29)

Yes, but you still need to use build tools version 29, otherwise the build will fail to recognize android:requestLegacyExternalStorage.

Did you mean this in config.xml?

"Build Tools" is part of the Android SDK, you can download it using the Android Studio's SDK Manager tool.

[jfoclpf]

@breautek thanks. I also used your answer, citing the source, to reply on stackoverflow the open issue.

[snr-lab]

Adding android:requestLegacyExternalStorage="true" is throwing AAPT: error: attribute android:requestLegacyExternalStorage not found. Because cordova 9.x.x is building with SDK-28 even i mention targetSdkVersion as 29. Is there any workaround for that?

[breautek]

You need to be using build tools 29.

https://cordova.apache.org/docs/en/dev/guide/platforms/android/index.html#adding-sdk-packages

[HarelM]

Seems like opening a file in android 11 doesn't work. It does work in android 10. I'm not using any code but simply input with type=file to pick a file. If anyone has a solution please let me know, I'll see if I can send a PR once I get to trying and fixing it...

[jfoclpf]

@HarelM do you have any updates on this issue? Is the social sharing plugin the only option yet to save a file in Android 11?

[HarelM]

Not yet, but we are talking about two different issues:

1. Save to file
2. Open a file

To summarize my experience:

1. I have "solved" save to file by using social sharing - both due to security and mainly due to better user experience (IMO of course)
2. I will be looking into the issue of file open on android 11 soon - see issue https://github.com/IsraelHikingMap/Site/issues/1451 in my repo.

Also I've just seen recently that starting from August/November this year Google is changing the policy to target API level 30 for all new/updated applications so this has to be solved soon from my point of view...

[jfoclpf]

Thank you @HarelM , regarding point 1. could you kindly tell me exactly which social sharing plugin do you use?

[HarelM]

cordova-plugin-x-socialsharing https://github.com/IsraelHikingMap/Site/blob/6b658ae41f5a21d582ac15ee18385af4f128c6ac/IsraelHiking.Web/package.json#L116

[jfoclpf]

received yesterday, Google is pressing on time limits

[Carlosps]

Same here @jfoclpf. I will paste the text here so that people can find it easier.

Starting May 5th, you must let us know why your app requires broad storage access

We've detected that your app contains the requestLegacyExternalStorage flag in the manifest file of 1 or more of your app bundles or APKs.

Developers with apps on devices running Android 11+ must use Scoped Storage to give users better access control over their device storage. To release your app on Android 11 or newer after May 5th, you must either:

Update your app to use more privacy friendly best practices, such as the Storage Access Framework or Media Store API Update your app to declare the All files access (MANAGE_EXTERNAL_STORAGE) permission in the manifest file, and complete the All files access permission declaration in Play Console from May 5th Remove the All files access permission from your app entirely For apps targeting Android 11, the requestLegacyExternalStorage flag will be ignored. You must use the All files access permission to retain broad access.

Apps requesting access to the All files access permission without a permitted use will be removed from Google Play, and you won't be able to publish updates.

There is a learn more

[jfoclpf]

@breautek does this plugin request MANAGE_EXTERNAL_STORAGE, that is, does it request "Use of All files access permission"? This last sentence is very alarming: "Apps requesting access to the All files access permission without a permitted use will be removed from Google Play, and you won't be able to publish updates."

Do you know how can we request use of just one single public directory, for example, the Download directory?

Is there any plugin or means to interact with the Android Storage Access Framework?

[breautek]

@breautek does this plugin request MANAGE_EXTERNAL_STORAGE, that is, does it request "Use of All files access permission"? This last sentence is very alarming: "Apps requesting access to the All files access permission without a permitted use will be removed from Google Play, and you won't be able to publish updates."

Do you know how can we request use of just one single public directory, for example, the Download directory?

Is there any plugin or means to interact with the Android Storage Access Framework?

This plugin does not use MANAGE_EXTERNAL_STORAGE. It was something I suggested people to use as a workaround in the past before Google made that announcement. Obviously unless if you're building a file manager type of app, it will be hard justifying the usage of MANAGE_EXTERNAL_STORAGE permission and therefore won't be recommended for most apps.

I think the single largest challenge with this plugin is how generic access the file plugin is suppose to be, and how do we translate file paths to a specific container type (such as Audio, Video, Pictures, Documents, etc), using the appropriate MediaStore APIs. Based on my experience, relying on mime type information inferred by file extension is not really reliable.

Someone will need to do the research, but it might not be possible to use this plugin for external storage anymore, or it may be more efficient to have a new plugin that uses the MediaStore APIs with a cleaner API. like getAudioList(), getDownloadList(), etc...

[jfoclpf]

Thank you so much @breautek but I really don't know either how come Play Store is complaining about MANAGE_EXTERNAL_STORAGE in my cordova app, because my config.xml does not make any reference to MANAGE_EXTERNAL_STORAGE. If my cordova APP has set this MANAGE_EXTERNAL_STORAGE where should it be? How can I turn it off?

[breautek]

Thank you so much @breautek but I really don't know either how come Play Store is complaining about MANAGE_EXTERNAL_STORAGE in my cordova app, because my config.xml does not make any reference to MANAGE_EXTERNAL_STORAGE. If my cordova APP has set this MANAGE_EXTERNAL_STORAGE where should it be?

Probably in your <cordova-root>/platforms/android/app/src/main/AndroidManifest.xml

If you see it there, I'd suggest try removing the platform and re-adding it. Cordova does a good job making sure things gets added to files, but has a hard time cleaning up if configurations gets removed.

[jfoclpf]

Thank you @breautek , I will check it out

If I understood correctly, it seems that a way to overcome the objections raised by new policy on Android 11+ is to use the Storage Access Framework. This Framework opens a system file picker and we can use it to open or save files with the intents ACTION_OPEN_DOCUMENT or ACTION_CREATE_DOCUMENT.

This plugin could have a method that would trigger these intents with a file picker, something like cordova.file.openFilePicker

// Request code for creating a PDF document.
private static final int CREATE_FILE = 1;

private void createFile(Uri pickerInitialUri) {
    Intent intent = new Intent(Intent.ACTION_CREATE_DOCUMENT);
    intent.addCategory(Intent.CATEGORY_OPENABLE);
    intent.setType("application/pdf");
    intent.putExtra(Intent.EXTRA_TITLE, "invoice.pdf");

    // Optionally, specify a URI for the directory that should be opened in
    // the system file picker when your app creates the document.
    intent.putExtra(DocumentsContract.EXTRA_INITIAL_URI, pickerInitialUri);

    startActivityForResult(intent, CREATE_FILE);
}

You can then access external storage

On devices that run Android 4.4 (API level 19) and higher, your app can interact with a documents provider, including external storage volumes and cloud-based storage, using the Storage Access Framework. This framework allows users to interact with a system picker to choose a documents provider and select specific documents and other files for your app to create, open, or modify.

in https://developer.android.com/training/data-storage/shared/documents-files#java

Maybe I'm missing or misunderstanding something, please correct me if I am.

[breautek]

This plugin could have a method that would trigger these intents with a file picker,

The goal of this plugin is to be able to programmatically control the file system files, so if the Storage Access Framework requires the use of UI to pick files, I don't think that will be acceptable in this plugin. And if that's the only way to access external files, then it sounds like we'll need a new plugin for accessing the external storage. Although I don't think that's the case, the MediaStore has Direct File starting in API 30, which I think could be used, but it is documented that this is an inefficient way (performance wise) of accessing MediaStore/Scoped storage files. Would be great if someone could experiment with this.

[Davparis]

Here is my case :

I have an app in production that actually :

• target API29, with android:requestLegacyExternalStorage="true"
• and use : 1) cordova.file.externalRootDirectory 2) file path to external card, if the external card exits, via cordova.plugins.diagnostic.getExternalSdCardDetails

According to my tests, with a device using Android 11 (API 30) and android:requestLegacyExternalStorage="false" in AndroidManifest.xml - I can read an existing file

• but can't create a new one

EDIT: When i change android:requestLegacyExternalStorage from "false" to "true" :

• with a simple update of the app, Read and Write are OK
• when the app is uninstalled, and re-installed, no Read, no Write

To support the changes requested by Google, I'm planning to :

• check if access to external is actually granted via cordova.plugins.diagnostic.requestExternalStorageAuthorization and then :
• copy user existing files (as far as i can read them) to cordova.file.dataDirectory
• put a variable in localstorage, so that the app use cordova.file.dataDirectory for the next starts.

If it is a fresh install, I would use only cordova.file.dataDirectory for storing the new files.

But i'm stuck for the access to external card. Is there a way to get a path to a folder in external card, even if that path is acceded only by my application ?

What is strange is that the path returned by getExternalSdCardDetails seems to use a sandboxed folder : file:///storage/0123-4567/Android/data/com.mycompany.myapp/files/

[victorvhpg]

Please update plugin to use MediaStore APIs too, and remove flag "requestLegacyExternalStorage" or make this flag optional

[jfoclpf]

Please update plugin to use MediaStore APIs too, and remove flag "requestLegacyExternalStorage" or make this flag optional

If you want to opt out that flag simply install this plugin from npm and not github

cordova plugin add cordova-plugin-file

But then it it won't work an Android 10+. If you just want to save a file, a temporary workaround is to use the cordova-plugin-x-socialsharing and then the user saves the file wherever they want. I got this tip from @HarelM and it works nicely on Android 11.

[victorvhpg]

i wanna save a file, not share a file

Please update plugin to use MediaStore APIs too, and remove flag "requestLegacyExternalStorage" or make this flag optional

If you want to opt out that flag simply install this plugin from npm and not github

cordova plugin add cordova-plugin-file

But then it it won't work an Android 10+. If you just want to save a file, a temporary workaround is to use the cordova-plugin-x-socialsharing and then the user saves the file wherever they want. I got this tip from @HarelM and it works nicely on Android 11.

[jfoclpf]

i wanna save a file, not share a file

You can save a file but you have to use the application directories, which are not accessible to the user, you can use cordova.file.applicationStorageDirectory which is read/write.

If you want to save a file for the user to later open (for example a pdf), you have to use cordova.file.externalRootDirectory. And right now, on Android 11+ you can't, unless you use the workaround I mentioned.

[jfoclpf]

@Davparis just a question, for what purpose do you need to save and open the files? For the user later to open them or just for the internal storage of the app? If it's the latter you can use applicationStorageDirectory which works on every Android version.

[victorvhpg]

Me

i wanna save a file, not share a file

You can save a file but you have to use the application directories, which are not accessible to the user, you can use cordova.file.applicationStorageDirectory which is read/write.

If you want to save a file for the user to later open (for example a pdf), you have to use cordova.file.externalRootDirectory. And right now, on Android 11+ you can't, unless you use the workaround I mentioned.

I need save to a shared folder like cordova.file.externalRootDirectory (write/read) Google will ignore requestLegacyExternalStorage

[jfoclpf]

Right now on Android 11 you can't with this plugin

[victorvhpg]

Right now on Android 11 you can't …

I know. This a big issue. This plugin need to use MediaStore API for that issue

[jfoclpf]

@victorvhpg Google will not remove apps with that flag to true, it will simply ignore that flag on Android 11+. Google will remove indeed apps with MANAGE_EXTERNAL_STORAGE permission with no valid reason therefor.

[victorvhpg]

@victorvhpg Google will not remove apps with that flag to true, it will simply ignore that flag on Android 11+. Google will remove indeed apps with MANAGE_EXTERNAL_STORAGE permission with no valid reason therefor.

Sorry, you are right, the flag will ignored. I dont wanna use MANAGE_EXTERNAL_STORAGE ( all device files) Only a public/shared folder like "download" its good for me.

[HarelM]

@breautek do you think a PR with only removing the requestLegacyExternalStorage=true here would move this issue a bit forward? Here's what I have in my inbox in play store console right now (I'm using a version from this repo and not a official version):

1. New apps must target API 30 by August, updates must target it by November
2. You are using the above flag, don't, we'll remove your app by may 5th

So just removing this flag would allow using the latest version of this plugin from this repository which is not/will not be possible in a few weeks...

As I said earlier, I have stopped using external storage almost completely after the release of Android 10 since I kinda figured this would be the direction google is going for... I still have issues opening files though which is annoying but I also solved this by associating files with my app and in general avoid using files but rather upload content to a server and store user data in a database... I'm beginning to think that using files is what we are used to from the old windows environment where folder and files were important, but as time progresses I see that more and more services are cloud based, and the usage of files is becoming annoying and inconvenient (compare the UX of word to google docs - the fact that in google docs everything is there, available, allows multiple users to edit, share, see and what not seems like the future to me... I might be completely wrong though :-))

[jfoclpf]

@HarelM check the differences between the npm latest version 6.0.2 and this github dev version.

The differences in the core code, if I could fully understand, are negligible

The npm version does not have the flag requestLegacyExternalStorage to true, so my advice is to use the npm version, that is what I plan to do

cordova plugin rm cordova-plugin-file
cordova plugin add cordova-plugin-file

That said, I also agree with you that that flag should be set to false. You can find it here.

[breautek]

API 29 enforces the scoped storage rules and that's what the legacy flag is for. To opt out of the new storage constraints. It was always meant to be a temporary flag with API 30 completely ignoring it and enforcing scoped storage constraints.

While removing the flag will have to be done eventually, it alone is not going to solve API 30 changes.

If you're app have no real reason to use external storage, then yes, it's probably best to migrate to internal storage. Internal storage is completely sandboxed and your app can read/write to this sandbox with no special permissions.

If time permits, I'd like to start experimenting with solutions next weekend.

[jfoclpf]

If you're app have no real reason to use external storage, then yes, it's probably best to migrate to internal storage. Internal storage is completely sandboxed and your app can read/write to this sandbox with no special permissions.

Does that mean we need to amend this line in config.xml ?

<preference name="AndroidExtraFilesystems" value="files,files-external,documents,sdcard,cache,cache-external,assets"/>

In the case we just use internal storage, how this line should be? Is this ok then?

<preference name="AndroidExtraFilesystems" value="files,cache,assets"/>

[alexisGandar]

But then it it won't work an Android 10+. If you just want to save a file, a temporary workaround is to use the cordova-plugin-x-socialsharing and then the user saves the file wherever they want. I got this tip from @HarelM and it works nicely on Android 11.

@jfoclpf I am not sure that I understood how the workaround works. Using the cordova-plugin-x-socialsharing you will ask the user to share the document (on drive, by mail...) so he can get the document. Or is there a way to use this plugin on the 'Files' android app so the user can save the file on his device ?

[HarelM]

Users can install xiaomi file manager for example and share a file with that app or simply share with mail, messaging, Facebook, drive etc. The idea is that they can choose what to do with the file instead of the app deciding on an arbitrary location they should "guess".