Enable org-capture template expansions for file notes

[zaeph]

Hello,

I wanted to include timestamps in the header of my file-notes, but I realised that bibtex-completion-notes-template-multiple-files didn't handle org-capture template expansions. Since I don't think there is any drawback to it, I figured we could add it in master.

Thanks for your work on helm-bibtex. I've been running it for a few years, and it's been stellar.

HTH.

[zaeph]

I've just noticed that doing this removes all but one newlines at the end of the template, which might not be ideal. I'm looking for a solution now.

[zaeph]

I've found a solution, but it is a little hackish. Hopefully one of you will be able to find something better.

[tmalsburg]

Neat. But I think there is a proposal to switch to the org-capture mechanism entirely. I think there is even a PR for that, but I was unsure about the implementation. Have to revisit this whole issue because I agree that we can easily improve in this area.

[tmalsburg]

See here #263 for the PR I mentioned.

[zaeph]

Whoops, wasn't aware that there'd been previous attempts, sorry.

I'll have a look at the related issue (#260), and I'll audit the PR when I get a chance.

[zaeph]

I've had a look at #263, and the scope is quite a bit larger than the feature I'm proposing here.

Moving the note-creation to org-capture is a good milestone to have, but one that will take a fair amount of refactoring (as exemplified by #263). In the meantime, my code provides the bulk of what we'd gain by moving to org-capture—namely its template expansion for timestamps (%T, %U, etc.) and sexp-evaluation—with a much smaller footprint.

I've made a handful of commits to propagate the feature to one-file notes. For those, I've also made it possible to have a %? string to indicate where point should be after the expansion.

The code should not change the behaviour of existing configurations besides squashing newlines at the end of one-file notes (standard in org-capture templates, but which can be controlled with plist options).

[tmalsburg]

I like that this code doesn't change existing behaviour.

Is there a way to force new lines, for example, via escaping \n?

[zaeph]

Is there a way to force new lines, for example, via escaping \n?

Thanks for your remak, it's put me on the right track. :)

With the current implementation, every newline in the template will appear in the final output. This is done with a helper function which protects the newline during org-capture-fill-template, and only replaces them after.

I think doing it this way removes any hint of hackyness from the original implementation.

[zaeph]

Also, on the topic of fully migrating to org-capture, I won't be able help much more than what I've done here.

I've been working on a connector package between helm-bibtex and org-roam over at org-roam-bibtex, and the project is bearing fruit. We've incorporated an org-capture setup that expands both the BibTeX fields with bibtex-completion-apa-get-value and the %-expressions of the org-capture-templates. Since I'm shadowing bibtex-completion-edit-notes with it, I'm not going to continue development on your function, but you might be able to get some inspiration for your work on #263.

If you want notes on the implementation, the efforts can be browsed in jethrokuan/org-roam#437 and jethrokuan/org-roam#415.

Thanks for your work! 🎉

[tmalsburg]

Also, on the topic of fully migrating to org-capture, I won't be able help much more than what I've done here.

No worries, I didn't expect anything. Thanks for your contributions so far.

Regarding org-roam-bitex, this looks like a really good idea and I might use it myself. One question: is this an add-on for org-ref or does it also work with plain helm/ivy-bibtex as well? I read through some of the relevant threads and some of the source code and the information on that appears conflicting. I'm not using org-ref myself and would favour if i could use org-roam-bibtex without installing org-ref as a dependency.

[tmalsburg]

Regarding the templating mechanism, do you think it would make sense to generalize the mechanism in bibtex-completion such that it works for org-roam-bibtex? Or does org-roam-bibtex have specific requirements that make this code unsuitable for bibtex-completion? Two reasons for my question: First, I guess we should try to avoid developing two highly similar mechanisms in parallel. Second, if there were two, potentially incompatible mechanisms that could be confusing for users.

[zaeph]

Regarding org-roam-bitex, this looks like a really good idea and I might use it myself. One question: is this an add-on for org-ref or does it also work with plain helm/ivy-bibtex as well? I read through some of the relevant threads and some of the source code and the information on that appears conflicting. I'm not using org-ref myself and would favour if i could use org-roam-bibtex without installing org-ref as a dependency.

org-roam-bibtex only requires org-roam and helm-bibtex. The only interaction with org-ref is that it shadows org-ref-notes-function (here's the relevant part). However, org-roam-bibtex is strongly coupled with org-roam: you could use it just to manage your bibliographical notes, but you'd be missing half the fun.

Regarding the templating mechanism, do you think it would make sense to generalize the mechanism in bibtex-completion such that it works for org-roam-bibtex? Or does org-roam-bibtex have specific requirements that make this code unsuitable for bibtex-completion?

We've actually reused a lot of helm-bibtex's functions to expand BibTeX elements in our templates. We then feed the output to org-roam's wrapper around org-capture, but the default org-capture could probably be used as well for other applications.

[tmalsburg]

However, org-roam-bibtex is strongly coupled with org-roam:

I am a happy org-roam user, so that's fine :)

We've actually reused a lot of helm-bibtex's functions to expand BibTeX elements in our templates. We then feed the output to org-roam's wrapper around org-capture, but the default org-capture could probably be used as well for other applications.

So if bibtex-completion had templating machinery along the lines of what you implemented, the only missing piece would be the ability to sneak in the org-capture wrapper? That doesn't sound too bad. What do you think?

I think there might another area where org-roam-bibtex could benefit from changes in bibtex-completion: helm/ivy-bibtex show symbols next to entries that do have notes. As far as I can see, with current org-roam-bibtex you don't get these symbols because the parser in bibtex-completion is not aware of the org-roam backend. I wanted to implement a plug-in infrastructure for finding PDFs and notes for a long time because users have different preferences and needs here. Perhaps now is the time to attack this?

[zaeph]

So if bibtex-completion had a templating machinery along the lines of what you implemented, the only missing piece would be the ability to sneak in the org-capture wrapper? That doesn't sound too bad. What do you think?

Pretty much, yes. But depending on how much energy you want to sink into it, this PR really covers 90% of what we'd achieve by fully moving to org-capture templates.

I think there might another area where you could benefit from changes in bibtex-completion: helm/ivy-bibtex show symbols next to entries that do have notes. As far as I can see, with current org-roam-bibtex you don't get these symbols because the parser in bibtex-completion is not aware of the org-roam backend. I wanted to implement a plug-in infrastructure for finding PDFs and notes for a long time because users have different preferences and needs here. Perhaps now is the time to attack this?

I've already played around with it, and I've got a working implementation on a helm-bibtex branch. There's a demo over at jethrokuan/org-roam#423 which shows it in action, reproduced here for convenience (pay attention to the 'PNS' in helm-bibtex):

It's based off deprecated code, but it'd be trivial to adapt for org-roam-bibtex. Since we'd be modifying bibtex-completion-prepare-entry, we'd have to create a bibtex-completion-find-entry-fn which, by default, would contain the lambda we use in bibtex-completion-prepare-entry to find the file, but which we could modify.

[zaeph]

It's based off deprecated code, but it'd be trivial to adapt for org-roam-bibtex. Since we'd be modifying bibtex-completion-prepare-entry, we'd have to create a bibtex-completion-find-entry-fn which, by default, would contain the lambda we use in bibtex-completion-prepare-entry to find the file, but which we could modify.

I went ahead and did it. #311 addresses the helm-bibtex side of things, and the org-roam-bibtex side of things is addressed in this branch. I'll hold off on the merge until you've reviewed my PR.

[tmalsburg]

Pretty much, yes. But depending on how much energy you want to sink into it, this PR really covers 90% of what we'd achieve by fully moving to org-capture templates.

Yes, but my understanding was org-roam-bibtex is not building on this PR's mechanism but replacing it. Did I misunderstand? Ideally I'd want the mechanism in bibtex-completion to be flexible enough for org-roam-bibtex purposes. Plus I/we intended to move to org-capture anyway.

[zaeph]

Yes, but my understanding was org-roam-bibtex is not building on this PR's mechanism but replacing it. Did I misunderstand? Ideally I'd want the mechanism in bibtex-completion to be flexible enough for org-roam-bibtex purposes. Plus I/we intended to move to org-capture anyway.

Ah, I see what you mean now, sorry for being long on the uptake. Yes, org-roam-bibtex is shadowing bibtex-completion-edit-notes, so you understood correctly.

We could turn bibtex-completion-edit-notes into a funcall to a customisable bibtex-completion-edit-notes-fn which org-roam-bibtex could modify, and that’d already remove the need for an advice.

Did you have something more advanced in mind?

[zaeph]

We could turn bibtex-completion-edit-notes into a funcall to a customisable bibtex-completion-edit-notes-fn which org-roam-bibtex could modify, and that’d already remove the need for an advice.

Proposed implementation is in 11cfa03.

[tmalsburg]

Merged with minor modifications. Thank you. Anything else we can or should do to accommodate org-roam-bibtex?

[zaeph]

Merged with minor modifications. Thank you. Anything else we can or should do to accommodate org-roam-bibtex?

Nope, I think we're all set. Thank you for your work! 🎉

[PavelNovichkov]

Hello, I'm getting an error Symbol's function definition is void: org-capture-fill-template from the bibtex-completion-fill-template function if I don't load org-capture in advance. I'm new to elisp packages, so I'm wondering whether this is a problem with my setup, or one should add (require 'org-capture) somewhere in the code, e.g. at the beginning of bibtex-completion-fill-template.

[zaeph]

Hello, I'm getting an error Symbol's function definition is void: org-capture-fill-template from the bibtex-completion-fill-template function if I don't load org-capture in advance. I'm new to elisp packages, so I'm wondering whether this is a problem with my setup, or one should add (require 'org-capture) somewhere in the code, e.g. at the beginning of bibtex-completion-fill-template.

I'm looking into it, thanks for bringing this to our attention.

[tmalsburg]

Thanks both! Fix is merged.