Encrypted annotations

[EverardNisse]

Encrypting annotations file requires frequent decryption (and key unlocking) despite annotations.gpg being kept open and decrypted at startup.

Expected behavior: Prefer decrypted annotations file buffer over disk when saving annotations or initializing the mode

[cage2]

Hi!

I never thought about annotate encrypted files, interesting!

For key unlocking i wonder if could be useful increase the cache of gpg like explained here: https://superuser.com/a/624488

I think even epa mode can do something similar (M-x customize-group epa) but i never tried.

If you annotate the file the buffer is marked as modified so epa will try to encrypt anyway.

Probably i could add a customizable variable to prevent buffer to be marked as modified as i proposed here:

https://github.com/bastibe/annotate.el/issues/74

What is your opinion about that proposal? Would this fix your issue?

Bye! C.

[EverardNisse]

Increasing cache time decreases security a bit, so I avoid it and have considered not caching at all. It's a sort of last resort option.

Below is the log for when an annotated 'testing.org' file is saved. My hunch is that being aware of the already open decrypted "annotations.gpg" buffer in "annotate-load-annotations" which is called by "annotate-save-annotations" will fix it. Then only encryption will be needed to save. That's the extent of my elisp ability at the moment.

Saving file /home/user/.emacs.d/testing.org...
Wrote /home/user/.emacs.d/testing.org
Decrypting /home/user/.emacs.d/annotations.gpg...done
Encrypting /home/user/.emacs.d/annotations.gpg... [2 times]
Annotations saved.
Reverting buffer ‘annotations.gpg’.
Decrypting /home/user/.emacs.d/annotations.gpg...done

[cage2]

On Thu, Jan 14, 2021 at 08:12:47AM -0800, EverardNisse wrote:

Hi!

Increasing cache time decreases security a bit, so I avoid it and have considered not caching at all. It's a sort of last resort option. Below is the log for when an annotated 'testing.org' file is saved. My hunch is that being aware of the already open decrypted "annotations.gpg" buffer in "annotate-load-annotations" which is called by "annotate-save-annotations" will fix it. Then only encryption will be needed to save. That's the extent of my elisp ability at the moment.

[...]

I think i am starting to understand. Probably the bottleneck is in the procedure that checks for file formats (one of those reads the entire file triggering the decryption functions of epa). I can skip the slowest routines if i recognize that the file is in "gpg" format (for example, checking if the file have extension ".gpg").

This is just an idea but let's see if this actually speed up the annotations saving! :)

Bye! C.

[cage2]

Hi @EverardNisse !

I pushed some code that, using caching, could help with the issue you raised.

Anyway there are many problems with this code, but the first is: i do not know if i addressed your problem at all! ;) may i ask you to make some tests?

The (crude) code is in my fork of this project:

https://github.com/cage2/annotate.el

branch caching

Anyway even if this patch works there are important caveats that i need to discuss with the maintainer and other users, but i need to know if this code is useful at loading encrypted annotation faster or not , first.

Bye and thanks in advance. C.

[EverardNisse]

It doesn't seem to me that much has changed, the logs remain the same and the functionality as well. My solution would be, check if a buffer exists with the same name as the annotations file variable. If it does, use that, save that. This behavior might be tested by logging file opens. The file should open only once on mode initialization / saved desktop load.

[cage2]

On Fri, Jan 22, 2021 at 12:23:43PM -0800, EverardNisse wrote:

Hi!

Thank you for donating some of your time checking this code!

It doesn't seem to me that much has changed, the logs remain the same and the functionality as well. My solution would be, check if a buffer exists with the same name as the annotations file variable. If it does, use that, save that.

This is really surprising because what i tried to do is more or less the suggestion you just given.

I tried to do some profiling and i can notice a general improvement, I am reporting the results below:

the use case is:

• non empty annotation database in a symmetric encrypted file using gpg: foo.gpg

visiting a buffer without the minor mode annotation active

bench-marking activating annotate mode (this command will trigger the loading of annotations)

[ M-x benchmark (annotate-mode) ]

results:

without caching (branch master): 1.6s with caching (branch caching):

first time: 0.9s

the first call will fill the so we can expect that subsequent call to the same command will use the cache instead of loading (and decrypts) the database from disk.

second time: 0.04s <- using cache

(annotate-save-annotation)

Here is difficult to benchmark because i have to type the password each time i save the buffer so measures are not reproducible anyway we can analyze the messages:

without caching:

---

Decrypting /home/cage/elisp/annotate.el/pluto.gpg...done Encrypting /home/cage/elisp/annotate.el/pluto.gpg... [2 times] Annotations saved.

with caching:

---

Encrypting /home/cage/elisp/annotate.el/pluto.gpg... [2 times] Annotations saved.

The later is skipping a decrypting operation, because of caching.

I wonder if you are able to reproduce this behaviour. Otherwise i am missing something here!

Bye! C.

[EverardNisse]

The variable 'epa-file-cache-passphrase-for-symmetric-encryption' may help you with testing. My encryption is purely asymmetric.

Been having no luck reproducing the desired behavior. If the key is cached, there is only one decryption and then encryption on saves. A buffer of 'annotations.gpg' does seem to trigger decryption when it's modified on disk, so reckon a separate cache is better than just looking for the buffer, but it's not consistent enough for me to say.

When the key isn't cached any more though it reverts to old behavior.

[cage2]

On Sat, Jan 23, 2021 at 11:55:34AM -0800, EverardNisse wrote:

Hi!

Things are becoming complicated but i am confident that we will get a solution eventually! :)

The variable 'epa-file-cache-passphrase-for-symmetric-encryption' may help you with testing. My encryption is purely asymmetric.

Been having no luck reproducing the desired behavior. If the key is cached, there is only one decryption and then encryption on saves. A buffer of 'annotations.gpg' does seem to trigger decryption when it's modified on disk, so reckon a separate cache is better than just looking for the buffer, but it's not consistent enough for me to say.

When the key isn't cached any more though it reverts to old behavior.

I think we have to go back to a much simpler text

• again we have a non empty (with empty database the caching does not work, this is a bug that i am going to fix later) database of annotations in an symmetric encrypted format like before (maybe i am wrong but, i do not think symmetric or not does matter in this case)

• then i registered, in my Emacs configuration file, annotate as minor mode for for all lisp-mode file (of course you can change with any minor mode you are most used with).

(add-hook 'lisp-mode-hook 'annotate-mode)

So far, so good.

• Now, from a terminal emulator, i kill my gpg-agent to ensure no key is cached (of course if you are not caching keys this step is not needed)

• then i open Emacs and visit a lisp file: a password should be asked to decrypt the database

• again kill my gpg-agent (again, if needed), no keys are cached anymore

• now i switch to Emacs frame and visit another lisp file, the buffer is filled with file's contents without asking for password, despite we instructed gpg-agent to not forget about keys before, as we are using the cached database.

Are you able to reproduce this use case? I hope yes! :)) Bye and thank you again! C.

[EverardNisse]

I've been testing intermittently and cannot replicate your / the desired behavior. Emacs still tries to decrypt the annotations file at least once on running 'annotate-mode' on a new file or when saving any file when the key isn't cached. The logs appear as usual.

[cage2]

On Thu, Jan 28, 2021 at 12:59:38AM -0800, EverardNisse wrote:

Hi!

I've been testing intermittently and cannot replicate your / the desired behavior. Emacs still tries to decrypt the annotations file at least once on running 'annotate-mode' on a new file or when saving any file when the key isn't cached.

So you are getting to insert the passphrase each time you visit a new file? Not just the first time?

If so, this is weird, just to be sure i am using an asymmetric scheme for encrypting the database and still works as expected (caching the database contents).

Even if unlikely may i ask if are you sure using the the branch "caching"?

Bye! C.

[EverardNisse]

I had a moment of epiphany. I had assumed emacs would recompile the patched file when changed / on start. Once I recompiled annotate.el the desired behavior was there. Can confirm that a cached database is now used. Thank you for your patience.

[cage2]

On Sat, Jan 30, 2021 at 01:07:37PM -0800, EverardNisse wrote:

Hi!

I had a moment of epiphany. I had assumed emacs would recompile the patched file when changed / on start.

To be honest i would assumed the same. So I guess i just do not compile to bytecode this package, good to know! :)

Once I recompiled annotate.el the desired behavior was there. Can confirm that a cached database is now used.

Great! 🎉

Thank you for your patience.

No problem! :)

I hope this patch solves (only partially, maybe?) the issue you reported.

Bye! C.