Make style selector annotation smart

[bdarcus]

Even if you don't want to do this yourself, @jkitchin, I thought I'd pull this into a separate issue, in part because export processors are another area I have not yet worked with.

If this can work, the advantage is the UI can adapt to the user setup, and so it can be a general solution, without need for user config.

WDYT?

Also, little thing: how do I parse a string into a citation object?

I feel like this is not possible. What would you do in an org-file that is going to be used for multiple exports, e.g. There will be a pdf version (via latex), an html version, an ODT version, and a plain text version. What does an accurate preview even mean? It will be different depending on the output.

The idea was just to run it to produce the plain text output, since it's not intended to be WYWIWG; more just help the user know what style to choose.

So for the TeX processors, would produce the same output as we have in our current UIs, but CSL would produce the kind of output you see in the csl-activate processor (which is also not a precise representation, and uses a default style).

The user would likely be able to choose which to display, and/or it would be auto-configured based on the export processors the user has set up.

Might even involve curated input example, to be sure to show what some of the variants (notable "full" and "caps) will produce.

Originally posted by @bdarcus in https://github.com/jkitchin/org-ref-cite/issues/6#issuecomment-888388491

[jkitchin]

Even if you don't want to do this yourself, @jkitchin, I thought I'd pull this into a separate issue, in part because export processors are another area I have not yet worked with.

If this can work, the advantage is the UI can adapt to the user setup, and so it can be a general solution, without need for user config.

WDYT?

Also, little thing: how do I parse a string into a citation object?

Here is one way to what I think you asking for (it is literally what you asking for, but I am not sure why you would want to do this, what do you need it for?)

#+BEGIN_SRC emacs-lisp :results code
(let ((c "[cite/t:global prefix; @lin-2021-exper-theor suffix]"))
  (with-temp-buffer (insert c) (goto-char 1) (org-element-citation-parser)))
#+END_SRC

#+RESULTS:
#+begin_src emacs-lisp
(citation
 (:style "t" :begin 1 :post-blank 0 :end 53 :prefix
     (#("global prefix" 0 13
        (:parent #0)))
     :contents-begin 23 :contents-end 52))
#+end_src

I feel like this is not possible. What would you do in an org-file that is going to be used for multiple exports, e.g. There will be a pdf version (via latex), an html version, an ODT version, and a plain text version. What does an accurate preview even mean? It will be different depending on the output.

The idea was just to run it to produce the plain text output, since it's not intended to be WYWIWG; more just help the user know what style to choose.

For my export to latex, it is possible for it to map to literally how it will export (at least on the style part). On the keys I am less concerned about it being just right, there it is sufficient for me to know what the key points too.

So for the TeX processors, would produce the same output as we have in our current UIs, but CSL would produce the kind of output you see in the csl-activate processor (which is also not a precise representation, and uses a default style).

The user would likely be able to choose which to display, and/or it would be auto-configured based on the export processors the user has set up.

The only way I can see this working in practice is that the tooltip uses a default exporter (defined by the user), or the cite_export keyword if it is defined. Otherwise, you don't know what the export is until it is done. Sometimes, a single document is even exported to more than one format, so there isn't a single format that is even right.

I think it would be easy to implement this, but I am having trouble getting a csl string from a citation at the moment. I feel like it might be an error in my local setup, but can't figure it out right now.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'latex t)
#+END_SRC

#+RESULTS:
: \cite{lin-2021-exper-theor}

This results in an error.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'html t)
#+END_SRC

Might even involve curated input example, to be sure to show what some of the variants (notable "full" and "caps) will produce.

Originally posted by @bdarcus in #6 (comment)

[bdarcus]

Here is one way to what I think you asking for (it is literally what you asking for, but I am not sure why you would want to do this, what do you need it for?)

Thank you.

To experiment with a preview using pre-cooked data. See below.

The idea was just to run it to produce the plain text output, since it's not intended to be WYWIWG; more just help the user know what style to choose.

For my export to latex, it is possible for it to map to literally how it will export (at least on the style part). On the keys I am less concerned about it being just right, there it is sufficient for me to know what the key points too.

Yes, I agree the keys, or even the actual citation item in CSL, is not important for this. Users just need to see what to expect.

Why I asked the question above.

So for the TeX processors, would produce the same output as we have in our current UIs, but CSL would produce the kind of output you see in the csl-activate processor (which is also not a precise representation, and uses a default style). The user would likely be able to choose which to display, and/or it would be auto-configured based on the export processors the user has set up.

The only way I can see this working in practice is that the tooltip uses a default exporter (defined by the user), or the cite_export keyword if it is defined. Otherwise, you don't know what the export is until it is done. Sometimes, a single document is even exported to more than one format, so there isn't a single format that is even right.

Does that matter?

Just to make sure I'm being clear, what I have in mind is specific to the annotation function for the style UI, and which only needs ascii

1. iterate through org-cite-supported-styles
2. for each oc export processor (how this is determined or configured TBD), run the ascii export process, and insert results in the annotation EDIT: see https://github.com/jkitchin/org-ref-cite/issues/9#issuecomment-889218922 for a simpler first step based on your test example

So if you have org-ref-cite and csl configured for 2, the annotation might look like:

\citep{doa2019}           (Doa, 2019)

... but the user wouldn't have to configure any of that, and it would work across export processors.

Also, fallback behavior for unsupported style and/or variants would automatically be accounted for.

I think it would be easy to implement this, but I am having trouble getting a csl string from a citation at the moment. I feel like it might be an error in my local setup, but can't figure it out right now.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'latex t)
#+END_SRC

#+RESULTS:
: \cite{lin-2021-exper-theor}

This results in an error.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'html t)
#+END_SRC

I don't get that error, FWIW.

 (org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'html t)
"<p>
(NO_ITEM_DATA:lin-2021-exper-theor)</p>
"

But per above, all we care about for the annotation is the ascii; right?

On the "pre-cooked" data issue from above, I was thinking of this for two reasons:

1. because I use org-cite-make-insert processor, I can't get access to the citation, and I don't want to write all that.
2. pre-cooked data can better and more consistently show some of the variants; most notably "full" and "caps"

[bdarcus]

PS - I guess I get this because CSL is my default export processor?

 (org-export-string-as "[cite/t:@low2001]" 'ascii t)
"(Low, 2001)
"

If yes, then maybe the simplest and most general approach is to only render the default processor in the annotation, using org-export-string-as?

Aside: note how it renders with the fallback behavior here. Because the first iteration of oc-csl doesn't have support for citet citations, it falls back to the detault.

Once Andras merges his updates, that will correctly render.

Hmm .. seems slow; previews might need to be cached.

Also, it doesn't only generate output for the default backend.

(foo/bar--make-previews)
("[citeauthor:@low2001]
" "[citeauthor/f:@low2001]
" "[citeauthor/a:@low2001]
" "[citenoauthor:@low2001]
" "[citenoauthor/c:@low2001]
" "[citenoauthor/bc:@low2001]
" "[citenoauthor/b:@low2001]
" "[citetext:@low2001]
" "[citetext/bcf:@low2001]
" "[citetext/cf:@low2001]
" "[citetext/bf:@low2001]
" "[citetext/bc:@low2001]
" "[citetext/f:@low2001]
" "[citetext/c:@low2001]
" "[citetext/b:@low2001]
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
")

[jkitchin]

I had something weird going on, the commits above seem to do what I want. at least I get a latex export by default for the tooltip, and I get a csl preview with a cite_export keyword set to csl.

[bdarcus]

So this would be trivial to extend for this?

[jkitchin]

I guess we are doing different things. I was interested in better previews on tooltips that better reflect the best guess of the intended export.

I did get that to work. And, I was able to adapt it to make the style annotation a little smarter. Now if I try to change the style on

with #+cite_export: csl

I get this annotation (with an author/date csl style):

In contrast, with my default exporter (latex) I see

It seems fast enough to me.

[bdarcus]

WDYT?

I like it; I'd just make the CSL one ascii of course ;-)

So is this pretty straightforward then?

I guess I'm wondering if I should try to adapt your code to my group-based UI, or if you want such a group-function here? If the latter, I can send you a PR that is at least the start.

[jkitchin]

It could be interesting to see what it would look like. I don't know what I would do with a group function here at the moment.

[bdarcus]

Hook it up?

Like annotation, It's ignored if not supported, but if running Emacs 28, you get the groups.

[bdarcus]

BTW, in case it's helpful to see where it's going, this the patch for oc-csl that Andras is working on.

https://gist.github.com/andras-simonyi/c79865da0b3252199b55d093f5a4ad64

[bdarcus]

I tried to test org-ref-cite-select-style from IELM, and also org-ref-cite-annotate-style.

The latter gave me:

(org-ref-cite-annotate-style "text")
*** Eval error ***  Wrong type argument: integer-or-marker-p, nil

... and the former a stringp type error.

[bdarcus]

I also just tried outside IELM using org-cite-insert, as I would put in a README, and of course got a similar error, this time stringp.

(org-cite-register-processor 'bibtex-actions-alt-insert
  :insert (org-cite-make-insert-processor 
                #'oc-bibtex-actions-insert
                #'org-ref-cite-select-style))

(setq org-cite-insert-processor 'bibtex-actions-alt-insert)

[jkitchin]

(org-ref-cite-annotate-style "text")

I would not expect that to work. My annotator requires the point to be on a citation, so it can get the references, and make an appropriate preview for it.

This is what I referred to in the mail list, it requires the key selector to insert the citation first, and then to update the style on it.

[bdarcus]

Right, but per subsequent comment, I also get the errors when in org, at-point.

[jkitchin]

I don't have "text" as a style. I pushed a few more probable fixes for this to make it work when there is not a citation at point, and to be more robust when a style is not defined.

[bdarcus]

That seems to work, but I only get the natbib previews. Is that intended?

I thought, from previous discussion, and looking at your code, it should use csl if either that is default, or it's defined in the document?

I don't have "text" as a style. I pushed a few more probable fixes for this to make it work when there is not a citation at point, and to be more robust when a style is not defined.

If you just use org-cite-support-styles does that not (at least partially) avoid you having to worry about this?

[jkitchin]

That seems to work, but I only get the natbib previews. Is that intended?

I thought, from previous discussion, and looking at your code, it should use csl if either that is default, or it's defined in the document?

It looks for a cite_export keyword and uses that, or defaults to latex. I have not figured out how to find what the default exporter is otherwise, because it depends on what you choose to export too, which is done "at run time". I could add a defcustom where you choose the backend you use most often, and fall through to that.

I don't have "text" as a style. I pushed a few more probable fixes for this to make it work when there is not a citation at point, and to be more robust when a style is not defined.

If you just use org-cite-support-styles does that not (at least partially) avoid you having to worry about this?

Not really. That format is very inconvenient to work with (and overly complicated imo), and I guess I have not defined the styles correctly (I use a simple, flat list of strings, with no aliases) so it returns nil for org-ref-cite. It doesn't make sense to me to have all the aliases, and to have to generate the combinations (e.g. when you asked how to get a flat list of styles before, it is not obvious).

#+BEGIN_SRC emacs-lisp :results code
(org-cite-processor-cite-styles (org-cite--get-processor 'org-ref-cite))
#+END_SRC

#+RESULTS:
#+begin_src emacs-lisp
("t" "p" "num" "a" "a/f" "a/c" "a/cf" "na/b" "na" "nocite" "t/b" "t/f" "t/bf" "t/c" "t/cf" "t/bc" "t/bcf" "/b" "/bf" "/bc" "/bcf" "/f" "/c" "/cf" nil)
#+end_src

[bdarcus]

BTW, this is another, related, topic, but the tooltip previews do work well, except since I use biblatex for the bib files, they omit the crucial date field.

How do you actually generate the previews there? With oc-csl? So then the formatting is done in citeproc-el?

If yes, that will be getting full biblatex support "any day", so that issue will be solved upstream.

If you are formatting it here, be sure to consider missing data, and maybe support "date" (though I know it's a slippery slope).

That format is very inconvenient to work with (and overly complicated imo), and I guess I have not defined the styles correctly (I use a simple, flat list of strings, with no aliases) so it returns nil for org-ref-cite.

I wonder if we should suggest changing this?

Andras had the same problem figuring out how to define it properly, and it's definitely a PITA to work with on our end.

To be clear, the style+variant distinction does matter to ensure proper fallback behavior, but that's an internal thing, and maybe simple strings at this level will work fine, and be much easier?

Because the idea of an export processor exposing the styles it supports is valuable.

WDYT?

If you support the idea, I can post it to the org list.

[bdarcus]

I have not figured out how to find what the default exporter is ...

What about this?

(cadr (assoc t org-cite-export-processors))

[jkitchin]

I have not figured out how to find what the default exporter is ...

What about this?

(cadr (assoc t org-cite-export-processors))

That is the fallthrough case, not the default. I think the best way is to look for cite_export, which is explicit when available, or default to a user chosen value.

[bdarcus]

That is the fallthrough case, not the default.

Forgive the ignorance: what's the difference?

[bdarcus]

I wonder if we should suggest changing this?

Hmm ... I guess he did it that way to support full names and short-cuts.

That might get too verbose as just strings.

But there must be some way to simplify.

[jkitchin]

BTW, this is another, related, topic, but the tooltip previews do work well, except since I use biblatex for the bib files, they omit the crucial date field.

How do you actually generate the previews there? With oc-csl? So then the formatting is done in citeproc-el?

If yes, that will be getting full biblatex support "any day", so that issue will be solved upstream.

If you are formatting it here, be sure to consider missing data, and maybe support "date" (though I know it's a slippery slope).

I don't do any formatting. It is generated from `org-export-string-as' with the export backend.

That format is very inconvenient to work with (and overly complicated imo), and I guess I have not defined the styles correctly (I use a simple, flat list of strings, with no aliases) so it returns nil for org-ref-cite.

I wonder if we should suggest changing this?

Andras had the same problem figuring out how to define it properly, and it's definitely a PITA to work with on our end.

I should think about it. The current data structure is maximally flexible in matching full and abbreviated styles. But, doing that requires what I consider an upper echelon of elisp coding via pcase and pattern matching. It also breaks the DRY idea, as the styles are defined in multiple places, e.g. in org-cite-natbib--style-to-command and where the processor is defined, which I avoided, at the expense that I didn't (but could, and it would be long as you mention above) support the full names.

To be clear, the style+variant distinction does matter to ensure proper fallback behavior, but that's an internal thing, and maybe simple strings at this level will work fine, and be much easier?

Because the idea of an export processor exposing the styles it supports is valuable.

WDYT?

If you support the idea, I can post it to the org list.

[bdarcus]

Here's an idea to consider, that would solve the DRY issue AND make this easier;

Nicolas should add a function that does the conversion to string, expose it, and use it in the included tex export processors.

Might that be the best balance?

[jkitchin]

That is the fallthrough case, not the default.

Forgive the ignorance: what's the difference?

My default for export will 99% of the time be to latex for a publication. The fall-through case is for an export backend you haven't defined in org-cite-export-processors, e.g. say I need a word doc, and export to odt. That isn't defined in there, so the the fall-through processor will be used. The recommended processor in the documetation is basic. You want this fall through to be maximally compatible with an arbitrary exporter, e.g. ascii, md, odt, jupyter notebook, calender, ...

So, in general there is not a way to say what the desired default is without a defcustom specifying it (which I added), or by using cite_export I think.

[jkitchin]

Here's an idea to consider, that would solve the DRY issue AND make this easier;

Nicolas should add a function that does the conversion to string, expose it, and use it in the included tex export processors.

Might that be the best balance?

It might be better to see if we can adapt the approach I took, which is to combine the styles in Nicolas fashion with the commands they map on to. Then, you have a way to lookup the output command, and you can generate the styles in the processor definition.

I will see if I can do that this morning. I feel like it is all that is left in org-ref-cite. I am on vacation starting tomorrow, and it would be nice to wrap it up!

[bdarcus]

So, in general there is not a way to say what the desired default is without a defcustom specifying it (which I added), or by using cite_export I think.

So maybe another suggestion to include in org-cite?

[jkitchin]

So, in general there is not a way to say what the desired default is without a defcustom specifying it (which I added), or by using cite_export I think.

So maybe another suggestion to include in org-cite?

I feel like I got the cite_export idea from org-cite already.

The citation triplet can also be set with the CITE_EXPORT keyword. E.g.,

+CITE_EXPORT: basic note numeric

or

+CITE_EXPORT: basic

In that case, `basic' processor is used on every export, independently on the back-end."

[bdarcus]

I will see if I can do that this morning. I feel like it is all that is left in org-ref-cite. I am on vacation starting tomorrow, and it would be nice to wrap it up!

Yes, I remember.

Maybe if you come up with a good general solution, you can submit as a patch to org when you get back?

[jkitchin]

I think that push fully supports org-cite like styles. There are a lot of style options, since it supports every combination of full and abbreviated style. Everything I tried worked for me, but I won't be surprised if I missed some things that rely on the old style code.

I am going to take a break for a few hours probably. if you spot an issue, I can probably fix it later this afternoon, or when I get back in a week.

[bdarcus]

I'm starting to lose track of the different pieces of code I'm tracking, so possible this is my fault. But I don't think so.

I'm getting a stringp type error when using the update, so am thinking there's a little bug somewhere.

[jkitchin]

I'm starting to lose track of the different pieces of code I'm tracking, so possible this is my fault. But I don't think so.

I'm getting a stringp type error when using the update, so am thinking there's a little bug somewhere.

what are you doing that causes that?

[bdarcus]

I'm starting to lose track of the different pieces of code I'm tracking, so possible this is my fault. But I don't think so. I'm getting a stringp type error when using the update, so am thinking there's a little bug somewhere.

what are you doing that causes that?

Just M-x org-cite-insert on the "cite". Same on a key correctly opens bibtex-actions-read.

Using this:

(org-cite-register-processor 'bibtex-actions-alt
  :insert (org-cite-make-insert-processor #'oc-bibtex-actions-insert
                      #'org-ref-cite-select-style))

(setq org-cite-insert-processor 'bibtex-actions-alt)

[jkitchin]

Does this work:

(org-cite-register-processor 'my-inserter
 :insert (org-cite-make-insert-processor #'org-ref-cite--complete-key-28
                     #'org-ref-cite-select-style))

(setq org-cite-insert-processor 'my-inserter)

in [cite:@rose-2019-pybliom]

if you are on "cite" it should change the style. Does (org-ref-cite-select-style) work?

this is the combinatorial explosion of trying to support modularity...

[bdarcus]

Does this work ... in [cite:@rose-2019-pybliom]

if you are on "cite" it should change the style. Does (org-ref-cite-select-style) work?

No. I get a "wrong type" error on eval, similar to what I saw in my own code when I load order issues.

this is the combinatorial explosion of trying to support modularity...

I still don't understand why this is an issue here, but admittedly haven't looked at this code closely.

Maybe just remove that "28" function, and see if we can get the basics working?

Earlier today, what you had was working correctly, so I suspect some subtle thing changed since.

[bdarcus]

I did just try to first load only the org-cite-make-insert-processor bit, and then the setq, but that also didn't work; same error.

But it's a stringp error, which from my experience is not the error that arises from org-cite-make-insert-processor (when it complains currently, it gives a cryptic functionp error).

[jkitchin]

We also getting bitten by the inability to debug what comes out of org-cite-make-insert-processor. For me, the following things work.

(org-ref-cite--complete-key-28 nil) -> get one key as a string

(org-ref-cite--complete-key-28 t) -> get multiple keys as list of strings

(org-ref-cite-select-style) -> gets a style string

Are you saying that (org-ref-cite-select-style) is the problem? What commit is your org-ref-cite on?

[bdarcus]

Hmm ... outdated. e014ad0

Let me try again.

[jkitchin]

no worries. the style code is too clever IMO. It is very hard to debug.

[bdarcus]

Commit: 1092aa6

Same stringp error, even from IELM using org-ref-cite-select-style. So the issue is not related to org-cite-make-insert-processor.

[jkitchin]

I don't know what to tell you. that works for me. You will probably have to edebug it to see where it is failing. You can do that by going to the definition, and typing C-u C-xC-e

Then when you run the command it will let you step through the function by pressing n, and you can see the output and exactly where it is failing. it works for me with vanilla, vertico, ivy, and helm-mode.

I assume you have (require 'org-ref-cite-core) somewhere right?

[jkitchin]

Does this code work?

(defun test-select-style ()
  "Select a style with completion."
  (interactive)
  (let ((candidates (cl-loop for (style . command) in (org-ref-cite-get-combinatorial-style-commands)
                 collect
                 (concat (car style)
                     (when (cdr style) (format "/%s" (cdr style)))))))
    (completing-read "Style: " candidates)))

[bdarcus]

Does this code work?

(defun test-select-style ()
  "Select a style with completion."
  (interactive)
  (let ((candidates (cl-loop for (style . command) in (org-ref-cite-get-combinatorial-style-commands)
               collect
               (concat (car style)
                   (when (cdr style) (format "/%s" (cdr style)))))))
    (completing-read "Style: " candidates)))

ELISP> (test-select-style)
*** Eval error ***  Wrong type argument: listp, "t"
ELISP> (org-ref-cite-get-combinatorial-style-commands)
*** Eval error ***  Wrong type argument: listp, "t"

[jkitchin]

I don't understand why it works for me. What is org-ref-cite-styles set to?

[bdarcus]

ELISP> org-ref-cite-styles
(("t" . "\\cite")
 ("p" . "\\citep")
 ("num" . "\\citenum")
 ("a" . "\\citeauthor")
 ("a/f" . "\\citeauthor*")

... etc.

I'll see if I can figure it out later, when you're on vacation ;-)

I am running a recent version of Emacs 28, though I seldom run into bugs related to that. But it's possible.

[jkitchin]

oh, that is wrong. something is not updating right. see https://github.com/jkitchin/org-ref-cite/blob/main/org-ref-cite-core.el#L32.

or, something is getting loaded from somewhere else.

[bdarcus]

Ok, that would explain it. Let me see if I can figure that out.

[jkitchin]

If you have been running one session all this time, you probably have an old version in memory. You might be able to fix it by going to that file and typing C-u C-xC-e on the defcustom. it doesn't update automatically when you just require it. Maybe M-x eval-buffer would do it too.

as annoying as it is, I usually close and re-open emacs to make sure I don't have old definitions hanging around.

[bdarcus]

It's not that; seems something is wrong with straight and the git repo.

I'll just wipe that directory and use the "local" repo.

[bdarcus]

OK, progress! Is this basically what I should see?