Decrypt progress

[AuYuHui]

Can you add one Decrypt progress

[jsharkey13]

Hi, I am not sure what this means! Are you asking for a progress bar? It is hard to know how long decryption for a single file will take, and when decrypting multiple files the sizes of the files will have more of an affect than how many there are, so there is no obvious metric to use for "how complete is it".

[AuYuHui]

Yes, need a progress bar。Can there be a vague progress bar

[KnugiHK]

Maybe it could show the progress of the extract_files() function when it is invoked via the CLI?

Let's say, if there are 100 rows for a specified domain or relative path, we can show how many files have been decrypted so far.

[jsharkey13]

I am trying to avoid any unnecessary dependencies for this library; I mean it to be as lightweight and simple as possible so that it can be used in other more user-facing projects (e.g. Whatsapp-Chat-Exporter) without being opinionated or having side-effects. It should still be usable by itself, just as simple as possible.

I'd be up for adding an optional callback to extract_files, though. If it was provided, it could be called with useful data for every file in the loop (like the relativePath, domain, current and total number of files). I can see that just knowing how many files there are and how many are done so far is useful.

If I added a new argument, called filter_callback, to extract_files you could use a library like tqdm to make a nice progress bar yourself using that callback:

import tqdm

class DecryptionProgressBar:

    def __init__(self):
        self.tqdm = None

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.tqdm is not None:
            self.tqdm.close()

    def update(self, *, total_files, **kwargs):
        if self.tqdm is None:
            self.tqdm = tqdm.tqdm(total=total_files, desc="Files decrypted", unit="files")
        self.tqdm.update()
        return True

and then if I make extract_files call the callback with the total and relativePath, etc, you would be able to do things like:

with DecryptionProgressBar() as p:
    backup.extract_files(..., filter_callback=p.update)

and get the nice progress bar without this library needing any new dependencies, and without stderr being polluted for people who don't want any progress bar. It's a bit back-to-front, but I think it would work. Does that sound sensible?

I could even make the "callback" not be a callback but a filtering function, where if it returns something false-y it skips decrypting that file (~then update() above would need to also return True, but otherwise be unchanged~). That would simultaneously solve another use-case (more complex filtering than relativePath/domain allows) and allow the progress bar use-case with only a handful more lines of code in this library.

EDIT: I have added the callback as I proposed and slightly modified the above, so the code example should work as-is once v0.8.0 is out.

[jsharkey13]

v0.8.0 is released, so I am going to close this since I have tested the code above and it does work. Thank you for the idea, it's quite neat now!

[KnugiHK]

I am trying to avoid any unnecessary dependencies for this library; I mean it to be as lightweight and simple as possible so that it can be used in other more user-facing projects (e.g. Whatsapp-Chat-Exporter) without being opinionated or having side-effects. It should still be usable by itself, just as simple as possible.

Yeah. I understand (because I am also trying to prevent too many dependencies for my projects too), which is why I haven't made any pull requests yet as I am not sure which of the features or modifications I have added should be included in your repo.

Anyway, the callback feature is great, and I will sync your changes to my fork when I am available. Thanks!

[jsharkey13]

I think when I last looked at your fork, quite recently, I realised I'd basically rebuilt several bits of functionality you had actually already added there! But I think you still had enough "custom" stuff (particularly threading support whilst exposing a database connection, which made me nervous, but also errors and an option to disable cleanup) that I didn't think you could replace your fork with this version from PyPI.

I tried to make it so that you could point multiple EncryptedBackups at the same on-disk files simultaneously, but SQLite threading is so tricky to get right I didn't try to make this class thread-safe. I think I still need to work on error messaging, but you might find the chunk-based file decryption useful for really large files (though I had to restructure almost the whole codebase to make it work!).