search

NeoApplications / Neo-Backup

backup manager for android
GNU Affero General Public License v3.0
2.61k stars 127 forks source link

Invalid argument and Operation not permitted errors #45

Closed Tiefkuehlpizze closed 4 years ago

Tiefkuehlpizze commented 4 years ago

The mentioned errors occur on symlinks and special files like named pipes:

2020/06/14 - 16:26:16: cp: /storage/emulated/0/OAndBackupX/org.fdroid.fdroid/org.fdroid.fdroid/files/fdroid/index.html: Operation not permitted [F-Droid]
2020/06/14 - 16:26:16: cp: /storage/emulated/0/OAndBackupX/org.fdroid.fdroid/org.fdroid.fdroid/files/fdroid/repo/org.fdroid.fdroid_1000011.apk: Operation not permitted [F-Droid]
2020/06/14 - 16:28:10: cp: /storage/emulated/0/OAndBackupX/com.keramidas.TitaniumBackup/com.keramidas.TitaniumBackup/files/fifo0: Invalid argument [Titanium Backup]
PiQuer commented 4 years ago

I also get many errors like:

2020/06/20 - 16:20:17: cp: /storage/emulated/0/OAndBackupX/pl.solidexplorer2/pl.solidexplorer2/lib: Operation not permitted [SolidExplorer]

Is my backup complete nevertheless?

machiav3lli commented 4 years ago

@PiQuer it's mostly complete. Even if you get a notification that says otherwise. This problem has to do with apps with special folders.

Tiefkuehlpizze commented 4 years ago

I'm not sure at the moment. But it seems, that the compression function has issues. Maybe the strategy needs to be changed.

I cannot tell, if the backups are useless. It might be corrupt, but there is also a good chance, that it's a false or unproblematic error or the app can recover itself.

But it's better find a good solution. I don't like error messages on my backups.

Tiefkuehlpizze commented 4 years ago

Antother update. I've fixed issues with the F-Droid installation on my device. It's a bit weird. If I just backup F-Droid or my last backup in batch mode is F-Droid, it was throwing these errors now: 2020/06/21 - 23:45:30: cp: /data/user/0/org.fdroid.fdroid/files/fdroid/repo/org.fdroid.fdroid_1000011.apk: No such file or directory [F-Droid]

I've checked it via shell and yes: It was following dead symlinks. I guess an old version of F-Droid created these or they came because I'm carrying my Titanium Backup'ed version for ages.

Anyway, I have another issue with a similar nature: 

2020/06/21 - 16:58:03: cp: /storage/emulated/0/OAndBackupX/cz.zdenekhorak.mibandtools/cz.zdenekhorak.mibandtools/files/default.realm.management/access_control.new_commit.cv: Invalid argument [Mi Band Tools]
2020/06/21 - 16:58:03: cp: /storage/emulated/0/OAndBackupX/cz.zdenekhorak.mibandtools/cz.zdenekhorak.mibandtools/files/default.realm.note: Invalid argument [Mi Band Tools]```

These files are pipes and here we run into a big problem:

% ls -l           
total 4
-rw------- 1 pizze users   0 Jun 12 10:22 access_control.control.mx
-rw------- 1 pizze users   0 Jun 12 10:22 access_control.write.mx
prw------- 1 pizze users   0 Apr 11 21:49 access_control.new_commit.cv
prw------- 1 pizze users   0 Apr 11 21:49 access_control.pick_writer.cv

% zip test.zip *    
    zip warning: ignoring FIFO (Named Pipe) - use -FI to read: access_control.new_commit.cv
    zip warning: ignoring FIFO (Named Pipe) - use -FI to read: access_control.pick_writer.cv
updating: access_control.control.mx (stored 0%)
updating: access_control.write.mx (stored 0%)

% tar czvf test.tar.gz *
access_control.control.mx
access_control.new_commit.cv
access_control.pick_writer.cv
access_control.write.mx
test.zip

% tar -tvf test.tar.gz 
-rw------- pizze/users          0 2020-06-12 10:22 access_control.control.mx
prw------- pizze/users          0 2020-04-11 21:49 access_control.new_commit.cv
prw------- pizze/users          0 2020-04-11 21:49 access_control.pick_writer.cv
-rw------- pizze/users          0 2020-06-12 10:22 access_control.write.mx
-rw-r--r-- pizze/users        374 2020-06-22 00:02 test.zip

zip does not support pipes while tar does. My suggestion here is pretty obvious. :)

darhma commented 4 years ago

I've errors when backup some apps:

BackupFailedException: cp: /storage/emulated/0/OABX/it.wind.myWind/data/files/eime.realm.enc.note: Invalid argument

BackupFailedException: cp: /storage/emulated/0/OABX/im.vector.app/data/files/matrix-sdk-auth.realm.management/access_control.new_commit.cv: Invalid argument

BackupFailedException: cp: /storage/emulated/0/OABX/com.xda.labs/data/files/.realm.temp/5716813994014153188_realm.note: Invalid argument

machiav3lli commented 4 years ago

@darhma please check #84

darhma commented 4 years ago

@machiav3lli Thank you, I've checked #84 and I've solved by deleting their respective folders in OABX's backup folder. I've "verify apps over usb" in developer settings turned off.

Update: the problem isn't solved the backup of the apps doesn't work. When I try to restore the backup isn't there. It says that the last backup is from 1 january 1970 01:00:00. In the folder there is only an apk file, no data.

I also noticed that the app does not create the copy of the oandbackupx apk in the OABX folder

opusforlife2 commented 4 years ago

I know you guys are looking for a solution for symlinks and named pipes. In the meanwhile, is it possible to include the ability to manually exclude a file from backup as a temporary workaround?

BTW, isn't #84 a duplicate of this one?

Tiefkuehlpizze commented 4 years ago

It's not possible to detect and exclude symlinks that easy, since we only have the toybox tools running as root. This means, we would need to parse output from ls and let the logic which is currently simply a call to toybox cp -R react on symlinks. I'm building something like that, but it'll need some more time.

opusforlife2 commented 4 years ago

I did read your comments on the other issue, so I'm aware it isn't easy. But what about manual exclusion?

Tiefkuehlpizze commented 4 years ago

Can you describe it a bit more? The user is able to delete symlinks from the data directories with a root shell or file explorer. It could be done with a shell script using find. I'm not sure about the effects, but at least it would destroy the data in the same way, as it would happen when symlinks are simply ignored and then a backup is restored. :)

opusforlife2 commented 4 years ago

Ah. Well, I didn't think to try that angle...

Then I have no problem. I can use that instead. Sorry!

opusforlife2 commented 4 years ago

Hmmm. Even with root, I can't copy either symlinks or named pipes. So I can't even back them up in case something goes wrong.

Tiefkuehlpizze commented 4 years ago

The backup strategy is, to copy everything to the backup folder and compress it then. Since the storage is sdcardfs (FAT32 or so), it does not support symlinks or pipes.

opusforlife2 commented 4 years ago

I don't mean the app. I mean using a root file explorer, I still can't copy a symlink or named pipe. Or are you saying they can only be deleted using root, but not copied?

machiav3lli commented 4 years ago

It has to do with the fact that storage file system is for most devices sdcardfs which basically doesn't support following symlinks for standard operations.

opusforlife2 commented 4 years ago

I see. Thanks!

opusforlife2 commented 4 years ago

Hmmm. Even with root, I can't copy either symlinks or named pipes. So I can't even back them up in case something goes wrong.

UPDATE. Apparently, I can copy the symlink if I stay in root storage. I just can't copy it to the non-root internal storage. So I'm going to make a random folder in /data/data and put the whole Fenix profile backup there and then try to update it. Hopefully, nothing breaks, but if it does, at least I can manually paste back my profile. Wish me luck!

Edit: Well, I didn't need to restore the backup, but for now, my procedure is: cut and paste lock file outside Fenix data folder -> backup the app and data -> paste the lock file back where it was -> update Fenix.

I also tried to copy a Named Pipe, but that process is either endless or gets stuck.

Tiefkuehlpizze commented 4 years ago

So I'm going to make a random folder in /data/data and put the whole Fenix profile backup there and then try to update it. Hopefully, nothing breaks, but if it does, at least I can manually paste back my profile. Wish me luck!

As long as you don't wipe the data partition, you're going to find it there again. This can also be a backup strategy, but you have to keep in mind, that also the system data remains. Updating like this is called "dirty flash" because you're not wiping your data to have a clean system.

opusforlife2 commented 4 years ago

@Tiefkuehlpizze I don't "dirty flash". When I have to restore an app backup, I delete the app completely. Since this is Fenix, and we know backups are currently a problem, I will also check if the org.mozilla.fenix folder is gone from /data/data or not. Only then will I restore the backup. Since the lock file I'm keeping separately from the backup is the same "version" as the backup itself, it shouldn't cause any issues, in theory.