New permission required for MM: WRITE_MEDIA_STORAGE for SD card

[frispete]

after upgrading to a SGS7 with Marshmallow (Android 6.0.1), I stumbled across a problem of Syncthing not preceeding with its work (given, some sync jobs existed and were partly synced to some SD card folder already. Looking into the web interface revealed, that all missing items suffered from permission problems.

Rooting option wasn't the real McCoy either, since it behaved strangely (don't remember details, sorry). Searching the net up and down finally stumbled me across some thread, that work around that issue by adding

    <item name="android.permission.WRITE_MEDIA_STORAGE" granted="true" flags="0" />

in the perms section of the affected package in /data/system/packages.xml. Given, that this has to be repeated after any update, I kindly request this permission for the official release.

Version Information

App Version: 0.14.3
Syncthing Version: 0.8.1
Android Version: Android 6.0.1

[wweich]

Syncthing cannot write to external SD cards, but only to the internal storage, which in most cases is sd card 0. The Syncthing App should have requested the needed permission for Marshmallow on first use.

Given, that this has to be repeated after any update

What makes you think that? Did you update the App since you set this? I'm pretty sure this is the Marshmallow permission, which was (for some reason) not requested / set on first use and should stay once it is granted in the Android system settings.

[frispete]

Syncthing cannot write to external SD cards, but only to the internal storage, which in most cases is sd card 0

Hmm, I cannot speak for the general case, but using the mentioned manual modification, syncthing can write to arbitrary folders on my external sd card. I have several related jobs defined, that all write to top level folders in /storage/nnnn-mmmm (n and m being decimal digits) without using the root option.

Note: MM uses 2 different modes for external SD cards (I use this silly description, since MM on a SGS7 shows its internal memory as sdcard 0 (/storage/emulated/0). In one mode, it combines its internal memory with the memory of the external sd card (the one, you can plug in) by the price of encrypting both. In the other mode, the external SD card stays separate, thus the sd card keeps a valid filesystem of its own, that can be used outside of the device. This is the mode, I'm using. Syncthing is used here to keep about 75 GB out of 120 GB in sync with my desktop, so yes, it is doing a great job, as long as it doesn't suffer from permission problems.

What makes you think that? Did you update the App since you set this? I'm pretty sure this is the Marshmallow permission, which was (for some reason) not requested / set on first use and should stay once it is granted in the Android system settings.

After installing the latest update (the 0.14.x syncthing), my phone suffered from the very same permission problem as initially. That issue vanished, after I added the permission item in /data/system/packages.xml again. For that reason, I believe, that adding this permission in the first place would solve my use case.

BTW, if comparing src/main/AndroidManifest.xml with com.nutomic.syncthingandroid section in /data/system/packages.xml, it appears, that all permissions from the manifest get into packages.xml but the WRITE_EXTERNAL_STORAGE, probably because the mode I'm using, the sd card is declared as media storage. This might create some confusion with the WRITE_MEDIA_STORAGE permission.

BTW2, those apps, that define WRITE_EXTERNAL_STORAGE (such as syncthing) usually also define READ_EXTERNAL_STORAGE.

[Nutomic]

Please read this article regarding the WRITE_MEDIA_STORAGE permission. According to that, only system apps should be granted that permission (but it might work on some ROMs). But it also says that some vendors may support this permission for apps. Can you maybe compile Syncthing-Android with that permission, and see if it still works for you?

Regarding READ_EXTERNAL_STORAGE, that is already included in WRITE_EXTERNAL_STORAGE.

[frispete]

Of course, Google doesn't care about this issue, since their devices traditionally came without any sd card slot. I bet, they will change this fact, if they start deliver devices with a slot. According to the article (thanks for the pointer, Nutomic), Samsung added the /system/etc/permissions/platform.xml hack for some devices, but not doing so anymore since S4 at least (I tried to use syncthing on my wifes S4, but failed miserably, due to that fact, combined with refraining from rooting it...). I'm applying this hack on all my devices below MM in order to be able to use the sd card space and syncthing in the way, I want.

It's a pity, that there are such wonderful tools like syncthing, that are crippled due to some brain damaged company policy.

Will try to compile syncthing myself with this modification in the course of setting up my wifes new S7 next. Again, I will attempt to use syncthing without rooting the phone, in order to sync some folders from the external SD card. Will let you know, where I got...

[kilrah]

Just discovered Syncthing and really liking both the concept and implementation, thanks. Would love to replace my "common" cloud providers with it, but unfortunately the inability to sync on the external SD of my Android 6 phone (S7) is pretty much a deal breaker as it doesn't allow syncing the only thing I usually use Dropbox for which is my photos.

Hope a solution can be found, whether this or another. All other apps I use that initially had trouble with that could solve it pretty quickly through the "select external SD and give permission" trick.

[Nutomic]

I added the permission for the next release. Please let us know if it works ;)

[frispete]

Updated to the new version, but the new permissions seem to have adverse effects on packages.xml:

I expected to be asked for new permissions, but I was not. But now both permissions WRITE_EXTERNAL_STORAGE and WRITE_MEDIA_STORAGE are gone?!?

Needless to say, syncing fails due to permission problems.

[frispete]

I started to build some variants based on the current version, but wasn't able to put my head around this issue, yet. It's strange.

My aim is simply put: be able to sync arbitrary folders on an external sd card without using root on Lollipop and Marshmallow.

With Lollipop (non root) and SA 0.8.1, I harvested a permission denied, when attempting to sync an external DCIM folder. After creating a .stfolder file manually (with es file explorer), SA is operating fine.

Now MM seems to change this area again. I've prepared a test environment, based on a non rooted MM today (S5 flashed with MM and external SD), with a test folder. syncthing (0.8.2) wasn't able to create a folder on the external space (/storage/xxxx-xxxx), and after creating it with es datei explorer, it stumbled across creating .stfolder:

Although es is able to create these items. It just asked a strange question on the first attempt, and presented a selector with 2 items, internal space and SD card space. The path looked exactly like before (/storage/xxxx-xxxx), but now it was able to write to it. Is it using getExternalMediaDirs perhaps?

Anyway, after that, syncthing still fails with permission issues:

While it's not possible to look into /data/system/packages.xml itself, the permissions of syncthing look fine:

What's the trick, that es uses to modify the SD card space and that syncthing is missing?

[frispete]

Okay, after reinstalling es, I was able to capture its behavior.

Here, I've to select the SD card:

Now, es is able to modify it arbitrary:

[wweich]

It was already discussed at length, that Apps can write external SD when using the new File Access API (which ES File Explorer uses). But Go currently doesn't support this API.

[frispete]

@wweich: thanks for the hint, mind pointing me to that discussion? Do you mean: https://developer.android.com/reference/android/support/v4/provider/DocumentFile.html as explained here: https://code.google.com/p/android/issues/detail?id=67570#c4444 It looks like this is exactly, what is currently missing in SA, but why is go support necessary than? If I'm not mistaken, it's a matter of launching the intent once for a sync job path, and operate on the returned tree as usual. It even makes the WRITE_EXTERNAL_STORAGE permission unnecessary. The only downside I see, this will work for LOLLIPOP and greater only. @Nutomic: if I understand the topic correctly, you need to revert the change, I requested (sorry). Another quuestion is: how and when could this be implemented.

[wweich]

The Android App is just a wrapper for the syncthing binary which is written and compiled in GO. The App writes the path, which should be synced, into the config.xml file and the syncthing binary will then sync it. As the binary cannot implement the API, it cannot sync folders which can only be written to with that API.

The discussion is in #29 and on the forums: https://forum.syncthing.net/t/writing-to-external-sd-card-in-android-5-0-and-above/2492 And in many other issues and forum threads which have been closed or abandoned with reference to the two mentioned

[Nutomic]

What is that packages.xml exactly? Never seen that before. And you think that adding the permission introduced a problem, and should be removed again?

[frispete]

MM seems to persist permissions and other parameter in the file /data/system/packages.xml. I don't think, that this permission will cause any problems, but it will not solve the issue either, and why add permissions, that don't provide any value. @wweich: thanks again for the pointers.

IOW, Google created a programming language, that is positioned to replace system code written in C, and now, using exactly that language is the reason, that the usability of syncthing is greatly reduced. Wow, what an achievement.

[Nutomic]

As I wrote before, the permission is normally only granted to system apps (which is probably the case on your device). But the article I linked earlier says it might work for user apps on some devices.

So I'm not sure if we should keep it or remove it again.

[kilrah]

For me the permission unfortunately didn't change anything.

But thanks to @frispete's workaround I can actually use it now, even if not ideally.

With Lollipop (non root) and SA 0.8.1, I harvested a permission denied, when attempting to sync an external DCIM folder. After creating a .stfolder file manually (with es file explorer), SA is operating fine.

Manually creating the .stfolder file in the external SD's DCIM folder after setting it up in ST allowed sync, obviously one-way only but that's already something, not worse than what Dropbox does.

[Anton-Latukha]

Have same problem now, as frispete.

App Version: 0.14.5 Syncthing Version: 0.8.3 Android Version: Android 6.0.1 Lenovo K3 Note

--------- beginning of.txt

[licaon-kter]

Did you update to Android 6 recently? Is the mount point/folder there?

[Nutomic]

@Anton-Latukha See the FAQ.

[Catfriend1]

sed -i "/com.nutomic.syncthingandroid/{N;N;N;N;N;s/.item name=\"android.permission.RECEIVE_BOOT_COMPLETED\".*$/<item name=\"android.permission.WRITE_MEDIA_STORAGE\" granted=\"true\" flags=\"0\" \/>\n\t\t\t\0/i}" "/data/system/packages.xml"

Put this in tasker, custom recovery script or whatever has access to the file. That automates it. Use this line at your own risk. I only tested on Windows with SED port.

[andrewufrank]

i just updated to 0.9.3 on a galaxy note 10 running 5.1.1 i can sync a folder on /storage/extSdCard/Android/data/com.numotic.syncthingandroid/files/xx but this seems not much value, given that i cannot make the camera or any other app point there. could this be generalized to work for arbitrary folders?

[mherrmann3]

Despite this topic being closed, I just wanted to mention that adding the WRITE_MEDIA_STORAGE permission in /data/system/packages.xml is the only solution for me to sync folder contents on the SD card (reading and writing, top level). After every update I have to add this permission again.

@Nutomic: you mentioned that this permission is granted to system apps. Would it be an alternative for me to convert Syncthing into a system app (my phone is rooted so it would be no problem to do so)? I used the superuser option once but it breaks up the config; I refrain from using it again 😉

[Nutomic]

No idea, you will have to give it a try :)

[frispete]

@mherrmann3 Thanks for the feedback and let us know about the outcome ;)

@Nutomic I tried to run with root permissions again lately, and it resulted in exactly the same issue, @mherrmann3 described: damaged config :( (that is: activate root, close, syncthing, open syncthing, enjoy the damaged config error message). It might be worth to look into that issue, since editing huge system files on every update isn't that convenient, either... In fact, it makes running syncthing so painful, that I use it a few times per year only, and refrain from suggesting it anybody else, who's trying to manage the SD card storage liberally. That's a pity still, as it is performing great on other systems (linux desktops, notebooks, windows minus installation woes).

And I still remember the times (before marshmellow), where my son and me entered our house. A few minutes later, all new pictures/films were available on the local server without sharing them involuntary with "the big companies"... Oh well.

[frispete]

Anyway, what's the state of using the new go interface, that would allow for adding the missing pieces? Wouldn't that be a great GSoc project for some student, that could be mentored by you guys?

[mherrmann3]

@frispete: yeah, I have the impression we are the only ones that sync folder contents on the SD card 😉

I will let you guys know after the next Syncthing app update (when the permission will be removed from the packages.xml again)... then I will convert Syncthing into a system app and check how it will behave...

[Nutomic]

@frispete I haven't really looked into that, so I'm not sure what it does.

Gsoc application for projects ends in 3 days, so we'd have to apply really fast (and I don't really know how that works).

[frispete]

@Nutomic I've brought it here in the hope for wider audience..

[Jaizu]

So can I ask what's the current status? Solid Explorer has read and write permission to my SD Card and my internal Storage. However, Syncthing can only read and however not sync. I would use my SD as internal storage but MIUI doesn't allow me to, so I'm currently looking for a fix or an alternative.

[Nutomic]

See #29

[bluelightsaber]

Just so everyone here knows, after some digging around, and without needing to do any hacking of config files or rooting devices (which is not ideal); on Android (lollipop and up), syncthing actually can sync folders on an SD card, with a single caveat: It has to be a folder that's either copied or created into syncthing's designated secure sandboxed folder automatically created within android. On my device, said folder is in the path "/storage/3C8F-1310/Android/data/com.nutomic.syncthingandroid/files", where "C38F-1310" is my SD card.

So, not exactly ideal, but there's a reason for it being like it is, as well. But with an explorer shortcut, it shouldn't be that bad either.

[frispete]

• when you deinstall syncthing, all your files are gone as well, which is not always wanted...
• how do you sync photos that way, e.g. the DCIM folder on the SD, sure you can put it there, but...

[wweich]

This info is not new and has been mentioned in one of the other issues and/or forum threads ages ago. Another advantage of that folder (also on internal storage) is, that Syncthing can chmtime, so e.g. dates of file based notes aren't screwed up.

[Anton-Latukha]

• TIP: And that shortcoming can be solved with soft symlink.

Who from devs knows the best way possible in this case?

I use open terminal-type way, but it is very painful. sh, on very chopped android console environment, on touchscreen, with touch android keyboard, while paths are lond and you doesn't have autocomplition, while doing one error - aborts commands, or yo need to clean-up wrong link.

• About deleting files upon uninstall - Syncthing needs to warn bluntly several times - that you need to copy your data with file manager, if you want to have it on your phone before you uninstall application.

[jmorgannz]

For what it's worth, I'm having this issue too. Seems like a major shortcoming of Syncthing on Android to me. Most of my data is on SD card - so if anything happens to the phone the data is safe.

Rooting or mucking with permissions is not an option.

+1 vote for figuring out how to get GO updated to make this possible

[Nutomic]

This issue is not relevant any more, please refer to #29.