Suggestion: Integrate BibDesk with orb-note-actions-interface

[rhstanton]

I use BibDesk to manage my Bib files and keep track of associated PDF files. Following the org-ref discussion at https://github.com/jkitchin/org-ref/issues/273, I can open the PDF file pointed to from my Bib file from the org-ref citation selection window, which is great. Please could I request that the same BibDesk functionality be integrated into the PDF finding/scraping options that pop up when you use the hydra option for orb-note-actions-interface? Thanks!

[myshevchuk]

Hi @rhstanton , thank you for this request! Please check the implementation of BibDesk support in PR #229. I'll need to update the manual and then will merge the PR into the master branch. BibDesk support is switched off by default, set orb-use-bibdesk-attachements to symbols t or only to enable it. You may also want to check the docstrings to orb-get-bibdesk-filenames, orb-get-attached-file and orb-open-attached-file for more information.

It works for me for note creation (template expansion) as well as for opening attachments and running the Scrapper through ORB Note Actions. Please let me know if it works for you too.

[rhstanton]

Thanks for the update. I just have one question - I've put the repository + pull request on my disk, but I'm not sure how to tell straight.el to load the locally modified version rather than the default. I've tried using :local-repo and :load-path in init.el, but it seems to keep loading the default version of org-roam-bibtex. Thanks!

[rhstanton]

I got the new version of the package to load by just copying the new version of the files over the version in the straight.el build directory. Seems a bit clumsy, but at least the right files are getting loaded.

However, when I type C-c n a to open the hydra, then a to open the pdf, nothing happens. In particular, there's no error message - about a missing pdf file or anything else - so this is at least different from what used to happen!

[myshevchuk]

@rhstanton Have you also set (setq orb-use-bibdesk-attachments t) or (setq orb-use-bibdesk-attachments 'only)?

I'm not particularly familiar with straight.el, although I'm using it under the hood as a part of Doom Emacs. From my experience, removing a package both from the build and the cloned repos directories is necessary when switching from an upstream source to a local repo source.

If this doesn't work, you can try to trace what's happening with Edebug. The functions orb-open-attached-file and orb-get-attached-file are the first candidates for it.

In any case, I'll merge the pull request soon, so this will at least eliminate the problems with installation.

[rhstanton]

I've tried both values for that variable. I also tried edebug-on-entry with both of the function names listed above, and nothing happened. In case this is just the package not being loaded properly, I'll wait until you merge the pull request then try and track down what's going on.

[myshevchuk]

I also tried edebug-on-entry with both of the function names listed above, and nothing happened.

You should use edebug-defun:

1. load the library
2. C-h f orb-get-attached-file
3. go to the function definition (follow the link to orb-core.el)
4. M-x edebug-defun

I'll wait until you merge the pull request then try and track down what's going on. This may take a couple of days or more.

You can also post your Org-roam and ORB config here. Maybe there are some interfering things there.

[myshevchuk]

An example of a BibTeX entry would also be helpful!

[myshevchuk]

I got the new version of the package to load by just copying the new version of the files over the version in the straight.el build directory.

Also make sure you've removed all old *.elc files - when they exist, they'll be loaded instead of source files.

[rhstanton]

Also make sure you've removed all old *.elc files - when they exist, they'll be loaded instead of source files.

The new functions are visible, so it's at least partially loading the new code.

[rhstanton]

Debugging the code, it correctly returns the pdf file name, then runs the command

(funcall bibtex-completion-pdf-open-function attachment)

where attachment contains the correct file name, but nothing displays on the screen.

Investigating further, I found that Emacs does not have permission to read the relevant file (I don't know why). I edited this permission, verified that I can open the file in an Emacs window, then tried C-c n a a again. Still nothing!

The variable bibtex-completion-pdf-open-function is set to:

bibtex-completion-pdf-open-function is a variable defined in ‘bibtex-completion.el’.
Its value is (lambda (fpath) (call-process "open" nil 0 nil fpath))
Original value was ‘find-file’

I tried resetting bibtex-completion-pdf-open-function to find-file, and now the paper opens in an Emacs window when I type C-c n a a, so we're making some progress.

I also tried running the command "open filename.pdf" from the command line, and Adobe Acrobat then opens and displays the paper just fine, so I suspect it may be a MacOS permission problem.

[myshevchuk]

OK, that's something. Check MacOS System Preferences -> Security & Privacy -> Privacy tab -> Files and Folders. There should be Emacs on the list to the right and the Documents, Desktop, Downloads folders, anywhere else you keep your files you access with Emacs, should be checked to allow Emacs access them. It seems it's not possible to add a new application to the list, so if Emacs is not on the list, try accessing the parent folder you store your documents in from within Emacs. The folder, not the file.

P.S. I just tried to reproduce it and in my case although the Downloads folder was checked in the list, I still couldn't open a PDF that was there. I'm using company completion, which can autocomplete file names. So when I started typing the path into the buffer, a MacOS popup window appeared asking me to grant Emacs access to the Downloads folder. It worked fine after that. This should be done only once, at least until your next Emacs update. I believe anything that reads a folder should do, you can try find-file or dired.

P.S.S. I deleted you comment with the BibTeX entry because it's not needed here anyway and that BibDesk filename could be deciphered to give away a potentially sensitive file path.

[rhstanton]

Thanks for the reply. Yes, I was able to change the file permissions for Emacs so that it can read files in the relevant directory. And when I use "find-file" as the relevant file opening-function, it opens the file fine in an Emacs window. But when I leave it set to

(lambda (fpath) (call-process "open" nil 0 nil fpath))

nothing happens, even though I can type "open filename.pdf" at the command line and see the file fine.

MacOS application permissions are probably my least-favorite thing about the OS, as it's so hard (and in some cases impossible) to modify them when things go wrong.

[rhstanton]

Apart from not being able to run the default system PDF viewer, it's getting very close! I can view the PDF file in an Emacs window and (after telling Emacs where to find the anystyle executable) can create a new BibTeX bibliography list using Orb PDF Scrapper. However, I don't seem to be able to customize the way this creates the BibTeX keys. I've tried editing orb-autokey-format but this doesn't seem to have any effect. Is there a different setting I should be using?

[myshevchuk]

Yes, I was able to change the file permissions for Emacs so that it can read files in the relevant directory. And when I use "find-file" as the relevant file opening-function, it opens the file fine in an Emacs window.

I meant M-x find-file -> navigate to the directory where you keep your files. A confirmation dialog like the one below should appear. Press OK and this should be it, no need to change the bibtex-completion opening-function var or file permissions.

If you experience any issues, e.g. the dialog does not appear, go to MacOS Settings -> Security & Privacy -> Privacy tab -> Files and Folders, find Emacs in the list and remove it from the list pressing the – button. Then repeat the M-x find-file step.

I hope it works!

[rhstanton]

I think we’re talking about two different things! Emacs now has the permissions it needs to read files in the relevant directory. It didn’t, but that’s no longer the problem. But I suspect it still lacks some permission that allows it to execute the “open filename” command.

[myshevchuk]

However, I don't seem to be able to customize the way this creates the BibTeX keys.

ORB does not create BibTeX keys in the first step of the scraping process. The keys are created by Anystyle and are left untouched by design. However, in the BibTeX buffer you can press C-c C-u to generate new keys. This secondary generation is done by ORB and is controlled by orb-autokey-format variable.

If you still experience any problems with PDF Scrapper, please open a new issue.

[myshevchuk]

I think we’re talking about two different things!

Ok, I now see what you mean and I actually can reproduce your issue. Strangely enough, if I evaluate the following form, the PDF is opened externally:

(let ((fpath "/path/to/pdf"))
  (funcall bibtex-completion-pdf-open-function fpath))

It didn't work until I removed and added Emacs application permissions as per above.

However, basically the same does not work from within the code when I try to open a file with ORB Note Actions. I'll investigate this. It doesn't strictly belong to your original feature request, but we can keep it here since it looks to be MacOS specific.

[rhstanton]

ORB does not create BibTeX keys in the first step of the scraping process. The keys are created by Anystyle and are left untouched by design. However, in the BibTeX buffer you can press C-c C-u to generate new keys. This secondary generation is done by ORB and is controlled by orb-autokey-format variable.

That solves it. Thanks!

[myshevchuk]

It looks like the problem was that (lambda (fpath) (call-process "open" nil 0 nil fpath)) doesn't work with abbreviated file names, e.g. ~/Documents/..., but rather requires full file names. I've pushed a new commit to the branch that fixes that.

[rhstanton]

Glad you solved it! And glad it wasn't a permission problem - those are really hard to solve, not least because two different people don't necessarily experience the same behavior.

[myshevchuk]

I've merged that branch into master. It should become available later today on MELPA.