Annoyances with plstore

[tollerm]

Hi, After updating to use the OAuth2 authentication with Google calendar I need to type 5 times (!) password for plstore. It seems that symmetric encryption is used by default and I need to authenticate 2 Google calendars. Is it possible to do something with it? For me it would be much better not to use encryption at all for tokens.

[dengste]

Does setting plstore-cache-passphrase-for-symmetric-encryption to 't' help?

[tollerm]

No, It did not help. It seems for me that the problem is because of my using two calendars. During the initial authentication I am asked to sign in only for the first calendar, and then caldav tries to sync the second calendar with the token of the first calendar. Is the caldav limited to use only with one Google calendar now?

[dengste]

Yep, currently you can only do one Google calendar. I'm always amazed how fast users will do stuff which I didn't think of. ;-) I'll see to it that I fix that.

[tollerm]

Ah, ok. It would be nice to implement multiple Google calendars in the org-caldav.

[dengste]

Please try again with latest master.

[tollerm]

Unfortunately, I still have the same problem: I am prompted 5 times to enter my password. Something is wrong in communication org-caldav with plstore: clearly, the store list is recreated every time when the pinentry cache is empty. When tokens are cached I am not asked for pinentry passwords during lifetime of cache. BTW, I do not use pinentry for enetering passwords for gpg encryption/decryption in Emacs. I have (setq epa-pinentry-mode 'loopback)) in my init.el file to enter a password from the mini-buffer prompt. Plstore clearly does not recognise this setting.

[dengste]

Could you please post your *org-caldav-debug* buffer so that I can see where exactly it is asking 5 times? An alternative would be to do 'M-x toggle-debug-on-quit', hit 'C-g' when the question appears for the fifth time and post the resulting backtrace.

[tollerm]

This is begin of my "org-caldav-debug" buffer (Real gmail address is replaced by 'user@gmail.com').

========== Started sync. Syncing first calendar entry: (:calendar-id "user@gmail.com" :files ("~/Documents/org_agenda/appt_private.org") :inbox "~/Documents/org_agenda/inbox_private.org") Check connection for https://apidata.googleusercontent.com/caldav/v2/user@gmail.com/events/. Got error while checking connection (will try again): (error "Error while checking for OPTIONS at URL https://apidata.googleusercontent.com/caldav/v2/user@gmail.com/events/: 401 Unauthorized") Generating ICS file /tmp/org-caldav-16351KR. === Updating EventDB from Org Org UID e43fbea8-e3ca-4015-8dfb-3cea6f660caf: Synced Org UID 77196149-34f6-4071-994b-a9674efbfdb4: Synced and many similar lines

[dengste]

I'm afraid I'm still confused where it is asking you five times. Does it happen right at the beginning or during the sync? In the above I can see that the first authorization fails, which is normal because of a bug in the Emacs 'url' package. But even there it shouldn't ask you again.

[tollerm]

Weird thing is that when I use edebug with org-caldav I am prompted for password only one time. When I tried to instrument org-caldav-sync-calendar function, I got immediately prompt for a password from the pinentry window without stepping into the org-caldav-sync-calendar function.

[rgemulla]

I am using multiple calenders with org-caldav and it works flawlessly. Your issue might be related to #122.

If you run Emacs on Windows, there is a bug in Emacs' plstore implementation. This is fixed in the latest master. Alternatively, use after (require 'plstore):

;; For some reason, plstore adds spurious carriage return characters on
;; windows, which messes up the decoding. This changes the open function to
;; use insert-file-contents and coding system raw-text instead of
;; insert-file-contents-literally. Required to make org-caldav store
;; authetication tokens correcty.
;;
;; Reported as Emacs bug #28114
(defun plstore-open (file)
  "Create a plstore instance associated with FILE."
  (let* ((filename (file-truename file))
         (buffer (or (find-buffer-visiting filename)
                     (generate-new-buffer (format " plstore %s" filename))))
         (store (plstore--make buffer)))
    (with-current-buffer buffer
      (erase-buffer)
      (condition-case nil
          (let ((coding-system-for-read 'raw-text))
            (insert-file-contents file))
        (error))
      (setq buffer-file-name (file-truename file))
      (set-buffer-modified-p nil)
      (plstore--init-from-buffer store)
      store)))

[niclasborlin]

@tollerm, I had the same issue. edebug-ing revealed a magit/vc-related error (see https://github.com/magit/magit/issues/1274). Setting (setq vc-handled-backends nil) in my .emacs resolved the issue. It is my understanding that the reason you are asked for the passphrase multiple times (when it should be cached, right?) is that the magit/vc stuff comes after the first request and before it is stored in the cache.

[rgemulla]

Well, the reason was that plstore.el had a bug that added these carriage returns when reading the file from disk. This then led to problems in decrypting. The above fix is pushed into Emacs master, but probably won't make it into the Emacs 25 branch. Here is the link to the bug report: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28114

[niclasborlin]

Thanks @rgemulla, but are you sure that bug #28114 causes this problem? I'm on linux and was able to fix the problem with the above mod to .emacs.

[rgemulla]

Right, this bug affected Windows only (since Linux does not add carriage returns in the first place).

Strangely enough, I don't see this issue on Linux either. I have vc-handled-backends set to '(RCS CVS SVN SCCS SRC Bzr Git Hg Mtn). According to the link you pasted, the issue seems to be that this error occurs if one puts data into the .git directory. If so, a better fix would be to not do that ;)

[niclasborlin]

I don't have any data in .git either, mind you :p. I think the "error" message that was supposed to be printed was something "...changed file was added to git...", which would be perfectly normal since the file I was sync'ing had been changed. But since the message triggered a bug in vc, the caller (here: plstore/org-caldav) was collaterally affected. Anyway, the fix above sorted out my issues.

[niclasborlin]

I may have found a clue for what is going on here... I have the same symptoms as @rgemulla reported, i.e I have to input the passphrase MANY times in a popup window. My edebugging took me to epg-start-decrypt where the environment variable GPG_AGENT_INFO is queried. On my system (ubuntu 14.04 w. emacs 25.1.1) the variable is set. Near the end of the function epg--start, a process is created (code below) that calls /usr/bin/gpg, etc., and the passphrase dialog appears.

       (setq process (make-process :name "epg"
                  :buffer buffer
                  :command (cons (epg-context-program context)
                         args)
                  :connection-type 'pipe
                  :coding '(binary . binary)
                  :filter #'epg--process-filter
                  :stderr error-process
                  :noquery t)))

SO, it appears that the reason why plstore-cache-passphrase-for-symmetric-encryption does not help is that it is the GPG agent that is asking the questions!

If I set the variable epg-debug, I get the following debug output in a buffer named *epg-debug* (note the leading space in the buffer name):

GPG_AGENT_INFO=/run/user/1106/keyring-qLsmxd/gpg:0:1
/usr/bin/gpg --no-tty --status-fd 1 --yes --use-agent --enable-progress-filter --command-fd 0 --output /tmp/epg-output17099mfq --decrypt -- /tmp/epg-input17099ZVk
[GNUPG:] PROGRESS /tmp/epg-input17099Z ? 0 742
[GNUPG:] NEED_PASSPHRASE_SYM 3 3 2
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 0 3
[GNUPG:] PROGRESS /tmp/epg-input17099Z ? 742 742
[GNUPG:] PLAINTEXT 62 1535124527 
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] END_DECRYPTION
GPG_AGENT_INFO=/run/user/1106/keyring-qLsmxd/gpg:0:1
/usr/bin/gpg --no-tty --status-fd 1 --yes --use-agent --enable-progress-filter --command-fd 0 --output /tmp/epg-output17099A02 --decrypt -- /tmp/epg-input17099zpw
[GNUPG:] PROGRESS /tmp/epg-input17099z ? 0 742
[GNUPG:] NEED_PASSPHRASE_SYM 3 3 2
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 0 3
[GNUPG:] PROGRESS /tmp/epg-input17099z ? 742 742
[GNUPG:] PLAINTEXT 62 1535124527 
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] END_DECRYPTION
GPG_AGENT_INFO=/run/user/1106/keyring-qLsmxd/gpg:0:1
/usr/bin/gpg --no-tty --status-fd 1 --yes --use-agent --command-fd 0 --armor --output /tmp/epg-output17099y9F --symmetric
[GNUPG:] NEED_PASSPHRASE_SYM 3 3 2
[GNUPG:] BEGIN_ENCRYPTION 0 3
[GNUPG:] END_ENCRYPTION
GPG_AGENT_INFO=/run/user/1106/keyring-qLsmxd/gpg:0:1
/usr/bin/gpg --no-tty --status-fd 1 --yes --use-agent --command-fd 0 --armor --output /tmp/epg-output17099_HM --symmetric
[GNUPG:] NEED_PASSPHRASE_SYM 3 3 2
[GNUPG:] BEGIN_ENCRYPTION 0 3
[GNUPG:] END_ENCRYPTION
GPG_AGENT_INFO=/run/user/1106/keyring-qLsmxd/gpg:0:1
/usr/bin/gpg --no-tty --status-fd 1 --yes --use-agent --enable-progress-filter --command-fd 0 --output /tmp/epg-output17099mme --decrypt -- /tmp/epg-input17099ZcY
[GNUPG:] PROGRESS /tmp/epg-input17099Z ? 0 750
[GNUPG:] NEED_PASSPHRASE_SYM 3 3 2
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 0 3
[GNUPG:] PROGRESS /tmp/epg-input17099Z ? 750 750
[GNUPG:] PLAINTEXT 62 1535140245 
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] END_DECRYPTION
GPG_AGENT_INFO=/run/user/1106/keyring-qLsmxd/gpg:0:1
/usr/bin/gpg --no-tty --status-fd 1 --yes --use-agent --enable-progress-filter --command-fd 0 --output /tmp/epg-output17099NFx --decrypt -- /tmp/epg-input17099A7q
[GNUPG:] PROGRESS /tmp/epg-input17099A ? 0 750
[GNUPG:] NEED_PASSPHRASE_SYM 3 3 2
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_INFO 0 3
[GNUPG:] PROGRESS /tmp/epg-input17099A ? 750 750
[GNUPG:] PLAINTEXT 62 1535140245 
[GNUPG:] DECRYPTION_OKAY
[GNUPG:] END_DECRYPTION

[niclasborlin]

@dengste @tollerm @rgemulla : When you were asked multiple times for the passphrase, was the question asked in a POPUP WINDOW or in the emacs message bar? If the former, then it might be your gpg-agent that is asking the questions! And the agent obviously does not know about plstore-cache-passphrase-for-symmetric-encryption...

[niclasborlin]

POSSIBLE FIX: If I unset the GPG_AGENT_INFO variable in my xterm environment, start emacs and run org-caldav-sync, I am asked for the passphrase ONCE (in the emacs message bar). :)

POSSIBLE FIX 2: Add (setenv "GPG_AGENT_INFO") to my .emacs file also worked.

[niclasborlin]

Added the text above to README in pull request #159.

[MarinosK]

Is this considered fixed? I'm having the same issue (along with the probably related #122 which also happens here): I'm asked several times for my passprhase (4 or 5), it then fails with a "401 Unauthorized" error, I then try again, it asks if I want to resume and then it works. I think the problem with the multiple passphrases is related to what @niclasborlin says - it's probably the GPG agents asking them since I'm not asked in the emacs bar but in a pinentry pop-up window. @niclasborlin solutions don't work here, however, as I'm not on ubuntu, I'm on Mac Os 10.14.6. I'm using org-caldav from MELPA, on emacs 26.3.

[garrett-hopper]

I was having the same issue, though GPG_AGENT_INFO wasn't defined at all. Killing gpg-agent (gpgconf --kill gpg-agent) did seem to fix it though. Is there another way to keep gpg-agent running while still hiding it from Emacs (oauth2, plstore, epg)? I don't suppose there's a way to store the tokens outside of plstore.el and bypass all this GPG password stuff?

[MarinosK]

@Jumblemuddle gpgconf --kill gpg-agent doesn't do the trick here, still the same behaviour.

[fanpeng-kong]

I am experiencing the same issue with emacs 26.3 on Ubuntu 18.04. It is so annoying that I have to stop using org-caldav for now. I really hope we can find a solution for people using google calendars.

[dengste]

Since org-caldav is using the oauth2 package, and this in turn is using the Emacs built-in plstore, there's not much I can do from the org-caldav side. This may be an Emacs bug, but first we need to find a simple test case.

Evaluate the following line by line (hit C-x C-e behind each one):

 (setq store (plstore-open (expand-file-name "~/.emacs.d/test_plstore.plist")))
 (plstore-put store "baz" '(:host "baz.example.org") '(:user "the_user" :password "very_secret"))
 (plstore-save store)
 (plstore-close store)

Evaluating the plstore-save should ask you for a password. This should usually be done by gpg-agent. Look in the title of the password window, on my system there is a PID like [16501], which means the process with PID 16501 is using the pinentry, which is /usr/bin/gpg2.

After you have evaluated the last line and closed the plstore, navigate to ~/.emacs.d and open test_plstore.plist. It should NOT ask you for a password because gpg-agent should still have it in cache. This should also work in a completely new Emacs session. Close the file again, say

 gpgconf --reload gpg-agent

and load the file test_plstore.plist in Emacs again. This time it should ask you again for the password. It should do so once, and then it should be in the agent's cache again for some time, so you should be able to close and open it again many times until the cache expires.

[fanpeng-kong]

@dengste Thanks for the instructions. I have followed the procedures and achieved the outputs as you described with two exceptions:

1. After evaluating the (plstore-save store) I got he pop-up window for entering the passphrase but did not notice the PID in the window title (I am on Pop OS 18.04 have gpg 2.2.4).
2. After closing the emacs session and executing gpgconf --reload gpg-agent and load the file test_plstore.plist in emacs again, it didn't ask me to enter the password.

I then re-enabled my org-caldav configurations for the Google calendars. And this time after calling org-caldav-sync for the first time, it asked me to enter the password twice (I should mention that I have 2 calendars) and start to sync the events.

So it is working well so far and I will update if the same issue of multiple passphrase and following error happen again.

[MarinosK]

@dengste In my case it's exactly as you say is should be (after executing gpgconf --reload gpg-agent it DID ask me to enter the password). Then I tried org-caldav-sync and it asked me 5 times for my password, then failed, then I run org-caldav-sync opting for y when asked if I want to resume and it worked as it normally does without asking the password. Then I run gpgconf --reload gpg-agent and tried again immediately - this time it did not asked for any password at all and it synced without problems. Just to add that one of these days I run org-caldav-sync and accidentally pressed cancel on the popup window - then I run org-caldav-sync again and it only asked me my password once before it synced without any problems.

[fanpeng-kong]

@MarinosK I got the exactly same behaviour as you now. Like @dengste said, this might not be a problem from org-caldav, I guess we can bear with the problem for now as long as we press the cancel button on the popup window?

[dengste]

I'm afraid I'm more confused than before. I have no idea what's happening. It looks to me like gpg-agent isn't correctly set up on your systems. First test on the command line by doing:

echo "foo" > bar.txt
gpg2 --use-agent --symmetric bar.txt

You should be asked for an encryption password twice.

Doing

gpg2 --decrypt --use-agent bar.txt.gpg

should give you "foo". You should be able to do this repeatedly without the agent asking for the password until you either wait until the cache expires or you do "gpgconf --reload gpg-agent". Does that work for you?

If yes, please make sure you don't have any epa- or epg- variables set in your init file. Make sure that GPG_AGENT_INFO is set properly inside Emacs by doing

M-x getenv RET GPG_AGENT_INFO RET

[MarinosK]

@dengste yes it all works the way you describe. There are no epa- or epg- variables in my .emacs (can't be sure if there are any, however, in some other init file loaded by .emacs - although I don't believe so). BUT, there is no GPG_AGENT_INFO variable - after M-x getenv RET GPG_AGENT_INFO RET I get a no match. Could that be related to the problem? What should I set this variable to?

[dengste]

@MarinosK Actually, since GnuPG 2.1 this variable is not needed anymore. So if you have v2.1 or newer this is fine, the variable is ignored anyway. You could fire up a shell in Emacs by doing

M-x eshell

and see if everything works within Emacs as well, using the same recipe as above. If it works, then I'm at a loss why plstore shouldn't work as well, since it is pretty much doing the same thing. You can set

(setq epg-debug t)

and check the " *epg-debug*" buffer after running org-caldav (note the space at the beginning). Maybe there's some hint what is happening.

[fanpeng-kong]

I'm afraid I'm more confused than before. I have no idea what's happening. It looks to me like gpg-agent isn't correctly set up on your systems. First test on the command line by doing:

echo "foo" > bar.txt
gpg2 --use-agent --symmetric bar.txt

You should be asked for an encryption password twice.

Doing

gpg2 --decrypt --use-agent bar.txt.gpg

should give you "foo". You should be able to do this repeatedly without the agent asking for the password until you either wait until the cache expires or you do "gpgconf --reload gpg-agent". Does that work for you?

If yes, please make sure you don't have any epa- or epg- variables set in your init file. Make sure that GPG_AGENT_INFO is set properly inside Emacs by doing

M-x getenv RET GPG_AGENT_INFO RET

It works as you described on my system too. To be honest, I don't feel it is a gpg or gpg-agent problem as I am already using a .authinfo.gpg to store various emails (for mbsync/mu4e/msmtp), spotify (in emacs of course) and even the org-caldav-oauth2-client-id and org-caldav-oauth2-client-secret without having any problem.

Further, we followed your instructions on the test_plstore.plist and bar.txt.gpg and had the desired behaviour. This makes me wonder that if this problem is particular to the Google calendars. Could the Google calendar side be the source of these errors?

[MarinosK]

@dengste first thing to say is that there is no epg-debug after I run the command, just a Compile-log (which should be the same thing) with the following, not very relevant output anyways:

~/.emacs.d/elpa/oauth2-0.11/oauth2.elc:Warning: reference to free variable
    ‘url-http-method’
~/.emacs.d/elpa/oauth2-0.11/oauth2.elc:Warning: reference to free variable
    ‘url-http-data’
~/.emacs.d/elpa/oauth2-0.11/oauth2.elc:Warning: reference to free variable
    ‘url-http-extra-headers’
~/.emacs.d/elpa/oauth2-0.11/oauth2.elc:Warning: reference to free variable
    ‘oauth--token-data’
~/.emacs.d/elpa/oauth2-0.11/oauth2.elc:Warning: reference to free variable
    ‘url-callback-function’
~/.emacs.d/elpa/oauth2-0.11/oauth2.elc:Warning: reference to free variable
    ‘url-callback-arguments’

What is probably more helpful is the output from Messages, on the first try (the one typically asking 4-5 times for my password), I get:

Decrypting /Users/marinoskoutsomichalis/.emacs.d/oauth2.plstore...done
Contacting host: apidata.googleusercontent.com:443
Continue connecting? (always, session only, no, details, ?): 
Accepting certificate for apidata.googleusercontent.com:443 this session only
Contacting host: apidata.googleusercontent.com:443 [3 times]
Continue connecting? (always, session only, no, details, ?): 
Accepting certificate for www.googleapis.com:443 this session only
Saving file /Users/marinoskoutsomichalis/.emacs.d/oauth2.plstore...
Error: (void-function force-backup-of-buffer)
Wrote /Users/marinoskoutsomichalis/.emacs.d/oauth2.plstore
helm-M-x: Wrong type argument: plistp, ""

(note that the certificate thing is not always there, it depends on the network I'm connecting from, I don't think this is relevant in any way)

The are two errors tho Error: (void-function force-backup-of-buffer) and helm-M-x: Wrong type argument: plistp, "" (note that I got the same error with void-function force-backup-of-buffer with one of your test examples, too - so I just googled for a definition of that function and temporary used that in this case, but normally it's missing)

after it fails, when I run again org-caldav-sync (successfully this time) Messages show:

Last sync seems to have been aborted. Should I try to resume? (y or n) y
Contacting host: apidata.googleusercontent.com:443 [3 times]
Added 1 event for today
Deleting event 1 from 2
Deleting event 2 from 2
Getting event 1 of 3
Saving file /Users/marinoskoutsomichalis/.emacs.d/org-caldav-backup.org...
Error: (void-function force-backup-of-buffer)
Wrote /Users/marinoskoutsomichalis/.emacs.d/org-caldav-backup.org
Property "LOCATION" deleted
Getting event 2 of 3
Saving file /Users/marinoskoutsomichalis/.emacs.d/org-caldav-backup.org...
Error: (void-function force-backup-of-buffer)
Wrote /Users/marinoskoutsomichalis/.emacs.d/org-caldav-backup.org
Property "LOCATION" deleted
Getting event 3 of 3
Saving file /Users/marinoskoutsomichalis/.emacs.d/org-caldav-backup.org...
Error: (void-function force-backup-of-buffer)
Wrote /Users/marinoskoutsomichalis/.emacs.d/org-caldav-backup.org
Property "LOCATION" deleted
Wrote /Users/marinoskoutsomichalis/.emacs.d/org-caldav-0904f74.el
View mode: type C-h for help, h for commands, q to quit.
Finished sync.

Note that error with void-function force-backup-of-buffer is still there, so I don't think this is relevant at all with the sync. The other error tho is missing.

--EDIT-- It should be noted that after this successful sync and for quite some time, all subsequent attempts would properly sync without even asking for password. If, then, restarting emacs and running gpgconf --reload gpg-agent it would only ask for a password once and successfully sync immediatley (and would keeps syncing every time I invoked org-caldav-syn the command again without asking for my password at all). Initially I though that this behaviour is because of the gpgconf --reload gpg-agent command, but I can now confirm it is not because of that - I tried again after an hour or so and it was once again asking me for my password 5 times. As I understand it, it is only after a successful sync that the system starts behaving normally for some time interval and the problem persists after that.

[hron]

I had the same issue. I've just tried to remove ~/.gnupg and it seems it fixes the issue. @MarinosK give it a try. I mean you can rebuild the GnuPG database and reimport the keys. I didn't have any private keys, so it was easy for me.

[hron]

It seems resetting the GnuPG database doesn't help... Today I see the same problem again. org-caldav asks for password multiple times.