Restoring data seems to not work as it should

[Sopor]

Yeah, i seems to have a similar problem on my S8+ as issue #170 have. I don't know what happens to @niksabaldun but i can see everything going forward as it should but when i start the apps/games they are all as i'm running them for the first time. I have backed up these apps/games before with TB and Helium and it has always worked perfect. Maybe i'm doing something wrong but what more can i do than backup and then restore? I only backup DATA but that must be enough? I do have the backup on external SD card but you support it so it should work. When i manually copy files i have noticed that i can't overwrite existing files. Maybe it is the same problem with oandbackup? Samsung S8+ and Nougat have a lot of new features and protections.

[ildar]

You don't seem to have the same as #170. Please dump the logcat

[Sopor]

I did found an old backup from TB and that worked perfect to restore. I will try to create a logcat for you.

[Sopor]

This is what i get when I backup to my external SD card: logcat

I will try to backup to my internal storage and make a new logcat when i restore.

[ildar]

That's

java.io.FileNotFoundException: /storage/50A5-F397/oandbackups/com.slipcorp.microtrip/com.slipcorp.microtrip.zip (Permission denied)

@jensstein , the version capable to work with secondary SDCARD needs to be released.

[qwertfisch]

Since Android 6.0, oandbackup cannot backup onto external storage, nor restore correctly from there. Access permission is requested and given, but creating the zip file (containing app data) or extracting it does not work. Restoring from external storage only copies the .apk but cannot extract the .zip file, therefore not copy the app data.

The problem has been mentioned in at least these issues:

124, #125, #145, #165, and #157. And here of course.

@jensstein Do you have an Android device on Android 6.0+ and external SD card available?

[Sopor]

Then i don't need to do any logcat for restore because this was my issue from the beginning.

[jensstein]

i do have devices running android 6 and 7+ but none of them have external sdcards so it's a little hard to work on this issue. i'll try to see if one of my older devices with external sdcard is upgradable to 6+ one of the next days, but i don't think it is.

[kmicu]

Same story here on my S3 with crDroid 3.6 (LineageOS 7.1.2). @qwertfisch do you have some workaround? E.g. copy oandbackup’s backup folder to internal memory; zip each app folder manually and then try to restore?

[ildar]

Emm, @jensstein ,

1. Try in emulator or Android-x86
2. Build apk with exhaustive debug we'll get you logs.
3. Do I remember it wrong you had a test apk behaving right?

[qwertfisch]

@kmicu The only workaround current is using the internal storage (mostly located under /sdcard), which should be safe when installing a new ROM. But I do not feel safe, and the external storage has the advantage to be readable in other devices. Also the internal storage does not provide enough space on my device.

[qwertfisch]

After some research I want to summarize the results I found:

History and Current State of Storage Permissions

Write Permissions

The permission WRITE_EXTERNAL_STORAGE grants access to primary external storage. This is not stated expclicitly in the official documentation, where “external storage” is just a name that points to anything that is not internal storage (system files, apps, application data). It can be an SD card or (like all devices for a few years) part of non-removable storage on an internal eMMC card.

Write access to secondary external storages (like here with additional SD card) is not allowed since API 16, but it seems this has been enforced only since API 19, quoting from here:

For all the other External Storage, an app can only write in its package folder. e.g. if you app package is com.sample.externalstorage, It can only write in the folders <sdcard root>/Android/data/com.sample.externalstorage. There’s no way, for a normal application, to write anywhere else on the SDCard.

Write access still was possible when an application had root permissions like oandbackup, as it seems until (including) Android 5.1 (API 22).

Read Permissions

Read access to primary external storage is granted by READ_EXTERNAL_STORAGE or (starting with Android 6) requesting this permission on the fly. You can access the complete external storage. There are more restricted permissions requesting only access for pictures or other media folders.

Read access to secondary external storages was and is still possible if you know the exact path to start navigating. This is more problematic since Android 6.0, as the folder /storage does not reveal its contents anymore to non-privileged apps (see below).

Possible Solutions and Conclusion

The issue is that oandbackup needs write access to secondary external storage (removable SD card) on Android 6+. The standard permissions are not sufficient, and write access is impossible even using root permissions.

Example at Lineage OS

I have seen the same behaviour on a nightly build of Lineage OS: the camera application Snap provides an option to store pictures and videos on external storage. (Un)fortunately it did not work and there were several issue reports. The logcat reported a permission fail:

07-30 15:30:03.491 E/CAM_VideoModule( 6082): prepare failed for /storage/3B9E-1BFA/DCIM/Camera/VID_20170730_153003.mp4 07-30 15:30:03.491 E/CAM_VideoModule( 6082): java.io.FileNotFoundException: /storage/3B9E-1BFA/DCIM/Camera/VID_20170730_153003.mp4: open failed: EACCES (Permission denied)

Further research resulted in this StackOverflow issue describing that an application would need android.permission.WRITE_MEDIA_STORAGE to have full access to secondary external storage, but it would have to be a system application. This corresponds to the bugfix from Lineage where the system app privilege is readded.

Conclusion

As root access does not seem sufficient anymore, I see two options:

• install oandbackup as system application or move it using /system/app mover
• change the permission property on system level described here and here (Update 2)

I will test the first option later today.

Edit: After writing the lengthy summary I overlooked that oandbackup also needs to request the permission WRITE_MEDIA_STORAGE. @jensstein Could you add this permission and provide at least an intermediate build? Then we could test it on Android 6+ systems using on of the options from above.

[qwertfisch]

I am not sure how the file access is currently done in oandbackup since it works for me on Android 5.1 (Cyanogenmod). Maybe using Storage Access Framework can be a solution? I read something about providing full access to all storage locations. But that would imply changes and restricting oandbackup to API21+ (Android 5.0).

And of course, there are file manager apps like ES File Explorer which do not have a problem reading and writing anywhere.

[kmicu]

Thank you @qwertfisch

I have checked permissions for oandbackup and busybox, but they both have access granted to SD card. I also thought that maybe it is related to https://jira.lineageos.org/browse/BUGBASH-10

In File Manager I see that folders created by oandbackup on internal memory have rwx permissions, but on SD card they have rx, *.log files are not created and *.zip files are also absent, but somehow copying of the rest was allowed ― thanks to that I was able to restore my most important apps.

For the (un)lucky ones, like me, with only a partial backup on SD card I was able to restore my apps data by doing following (fresh system, Conversations as an example app for recovery):

1. install oandbackup (e.g. from APK in oandbackup’s backup directory)
2. in oandbackup settings set a backup location to some folder on internal storage (should be the case by default)
3. install an app; here, it is Conversations (I use APKs from my backup on SD card)
4. backup it in oandbackup (that will create a proper backup folder with a *.log file in it)
5. go to backup location on SD card, compress Conversations’s folder to zip (I use the same name for zip as its folder name i.e. eu.siac.conversations becomes eu.siac.conversations.zip)
6. go to backup on internal storage and drop zip from 5th step into Conversations’ backup folder to replace a zip file already there
7. now you can successfully restore Conversations' data in oandbackup

My backup on SD card does not have *.log files and this is probably the main problem preventing restoration.

[varac]

Same problem here with lineageos 14.1, I get set permissions error when restoring apps.

[ildar]

In https://github.com/jensstein/oandbackup/issues/125 the solution was published recently