Backup of whatsapp media data stalls

[dwardor]

Neoback works great except for whatsapp backup where it systematically stalls (backup file size does not increase while neobackup says its work is ongoing) on the media data backup. I've let it "work" overnight and not progress at all.

Is there a log or something i can do to provide insight on the issue ?

Device: fajita Os: los 19 Rooted: magisk 25

[dwardor]

Phone reboot solved it ...

[dwardor]

Actually backup stalls with compression (level 5), but succeeds with no compression (level 0).

[hg42]

so what about level 1? I guess level 5 would work, too, but uses too much resources (cpu, memory) and your device starts swapping (un/loading app pages, not to disk) or something. You just didn't wait long enough. I'm not a compression expert, but I know that compressing non-redundant data like in videos and images is a tough task and also useless. Most Whatsapp data is like this. The lower the level the easier for the compressor.

[dwardor]

Here is a funny behavior using level 1 compression :

• Manual backup of whatsapp media stalls (launched in the evening and "working" overnight and still "working" in the morning)
• Programmed backup of whatsapp media (in the middle of the night) succeeded in less than 20min (so actualy happenend while the manual was still "working")

[dwardor]

Here is a funny behavior using level 1 compression :

• Manual backup of whatsapp media stalls (launched in the evening and "working" overnight and still "working" in the morning)
• Programmed backup of whatsapp media (in the middle of the night) succeeded in less than 20min (so actualy happenend while the manual was still "working")

Actualy that bizarre behavior happens also with level 5 compression so the issue lies in manual backup...

[hg42]

was the test with level 0 manual?

Which versions of NB and Android?

Batch and schedules use methods to keep the service running (foreground notification and wakelock). The manual backup doesn't do that. May be it's put to sleep? Did you put NB in background (by doing something else)?

[dwardor]

was the test with level 0 manual?

Which versions of NB and Android?

Batch and schedules use methods to keep the service running (foreground notification and wakelock). The manual backup doesn't do that. May be it's put to sleep? Did you put NB in background (by doing something else)?

LineageOS 19 NeoBackup 8.0.2 (with no battery restrictions --> no optimisations)

So here is the behavior Manual stalls just for whatsapp backup, except if compression level is 0 Schedule works fine what ever the compressoin level

[hg42]

Ok, sorry, los 19, so it's Android 12

You didn't answer

Did you put NB in background (by doing something else)?

It's a pretty interesting combination: The code doing the actual backup is identical (using the same functions). This means, compression handling is also identical.

So, it should be caused by that surrounding code, with the exception that compression=0 prevents the effect. It could be time based. Putting applications to sleep is time dependent.

How is the duration of a scheduled backup with compression=0 and with compression=1 ?

I think I have read somewhere, that Android 12 made this more restrictive. Disabling battery optimization probably doesn't disable all kinds of sleep mechanisms. I guess restricting an app that's working in background, is one of them.

[dwardor]

Ok, sorry, los 19, so it's Android 12

You didn't answer

Did you put NB in background (by doing something else)?

It's a pretty interesting combination: The code doing the actual backup is identical (using the same functions). This means, compression handling is also identical.

So, it should be caused by that surrounding code, with the exception that compression=0 prevents the effect. It could be time based. Putting applications to sleep is time dependent.

How is the duration of a scheduled backup with compression=0 and with compression=1 ?

I think I have read somewhere, that Android 12 made this more restrictive. Disabling battery optimization probably doesn't disable all kinds of sleep mechanisms. I guess restricting an app that's working in background, is one of them.

No I did not do anything to put it in background.

[hg42]

How is the duration of a scheduled backup with compression=0 and with compression=1 ?

[dwardor]

I recall all app backup takes 15min with compression 0 whereas it takes 18min with compression 5

[hg42]

usually compression=1 or may be 2 is the fastest.

If you say manual backup you mean starting it for a single app in it's app sheet, right?

Because a batch backup (which is also manual in some way) is just like scheduled, queing a job for each package in the Android Work Manager. Batch and schedule differ in the way they,are started (from button or from service).

A single backup from app sheet directly starts the action without queing. So the app executes it directly. Batch/schedule jobs are started by the system instead, and they explicitly say that long running jobs work better in the work manager.

Is Whatsapp the app with the longest running backup?

If you used the app sheet, then try batch instead.

[dwardor]

Yes Whatsapp is the biggest backup in size of all the apps.

Tried manual (starting from app sheet), batch (selecting just Whatsapp in list of apps and no other, and scheduled yesterday.

manual worked, batch stalled, scheduled worked

see screenshot of stalled batch (which I cancelled/killed after 2h)

[hg42]

Yes Whatsapp is the biggest backup in size of all the apps

I explicitly asked for the time, because size isn't the same. E.g. usually lot's of small files take longer than only a few big files. Obviously I mean the backup that completes normally.

manual worked, batch stalled, scheduled worked

wow, that's even more unexpected.

You didn't tell us, if the former "manual" tests were also from the app sheet? I'm mainly interested in this:

Manual stalls just for whatsapp backup, except if compression level is 0 Schedule works fine what ever the compressoin level

And with which compression did you use for "manual worked, batch stalled, scheduled worked"? I hope it was the same for all, right?

The screen shot says it's queued. A single job should immediately change from queued (=people standing in a row) to running (=running man) (for todays devices 6-8 jobs will run in parallel). If the job fails, it is queued again. After it failed the maximum count configured, it would be put in front of the status counters (with an angry face).

So, did it run at least once, or is it kept in queued state from the beginning and never changes to running?

[hg42]

as a summary from my POV:

F=fail O=Ok

in case "manual" meant "app sheet" before: A:

     comp:  0   1   5
manual      O   F   F
batch
schedule    O   O   O

in case "manual" meant "batch" before: B:

     comp:  0   1   5
manual      
batch       O   F   F
schedule    O   O   O

now we have X:

     comp:  ?
manual      O
batch       F
schedule    O

If comp=0 then B would conflict with X (batch) If comp>0 then A would conflict with X (manual)

[encrus]

This issue is still open. Does this mean that today, in Nov-2023 also this bug/issue is still present in the latest version of the app ?

[hg42]

I'm not sure if I would call it a bug...

I think, for most users it seems to work (though we don't get much positive feedback, but missing negative feedback should be something positive).

NB cannot do much about it, if ROMs pause jobs for arbitrary reasons. We can try our best to improve the conditions... but at the end there are always ROMs/vendors that beat the hell out of their batteries (creating good numbers for marketing purposes). I won't say that's definitely the case here, but conditions that don't sound logical are often caused by factors not yet considered.

[hg42]

as we (devs and some power users) don't experience the behavior, we depend on more investigation from users that encounter that issue.

@encrus, are you one of them?

[lefth]

I'm seeing odd behavior for Whatsapp backups as well. For me, manual backups usually don't work, but it seems okay if I don't touch the phone until it finishes. The behavior is the same for compression level 0 and 2. I can check other scenarios if that's helpful.

[hg42]

well,

you should not touch the phone while the backup runs

is a general rule anyways. It's obvious, that performing backups while simultaneously changing the data under the hood does not work. There is a t least a good chance that the backup will be inconsistent.

To say the truth: Backups in a running system (at least of active apps) are kind of impossible. I am always surprised, that it works.

The less you touch the thing, the better is your chance to get a good backup.

And there is no way to verify if the backup is (in)consistent, because the backup software does not know anything about the internal dependencies of the data of other apps.

This may work with transactional data (special technics that allow interruptions without data loss). However, the backup itself is still not transactional. And a snapshot with data from different times is at least suspicious.

I guess it still works, because apps are expecting some of the problems for other reasons (network outage, power loss, etc.). Maybe these strategies also help with inconsistent backups.