Make style selector annotation smart

[bdarcus]

Even if you don't want to do this yourself, @jkitchin, I thought I'd pull this into a separate issue, in part because export processors are another area I have not yet worked with.

If this can work, the advantage is the UI can adapt to the user setup, and so it can be a general solution, without need for user config.

WDYT?

Also, little thing: how do I parse a string into a citation object?

I feel like this is not possible. What would you do in an org-file that is going to be used for multiple exports, e.g. There will be a pdf version (via latex), an html version, an ODT version, and a plain text version. What does an accurate preview even mean? It will be different depending on the output.

The idea was just to run it to produce the plain text output, since it's not intended to be WYWIWG; more just help the user know what style to choose.

So for the TeX processors, would produce the same output as we have in our current UIs, but CSL would produce the kind of output you see in the csl-activate processor (which is also not a precise representation, and uses a default style).

The user would likely be able to choose which to display, and/or it would be auto-configured based on the export processors the user has set up.

Might even involve curated input example, to be sure to show what some of the variants (notable "full" and "caps) will produce.

Originally posted by @bdarcus in https://github.com/jkitchin/org-ref-cite/issues/6#issuecomment-888388491

[jkitchin]

yes, that is it! I don't love the grouping, but that is a bike shed to build another day!

[jkitchin]

Do the inserter's work now?

[bdarcus]

yes, that is it! I don't love the grouping, but that is a bike shed to build another day!

Key thing is basic approach is working, which is great!

So something easy enough to iterate.

I'll play with it while you're gone. If I come up with some good ideas, I may send a PR.

I definitely prefer long descriptions for the group titles. Do you think that needs to be configurable?

Do the inserter's work now?

Yes; with bibtex-actions-read! I haven't looked at what you added to your README, but I'm guessing code examples should work, which is always nice.

What doesn't work is the preview mode; I still just get natbib. But should be easy enough to fix later I'd guess.

Thanks for your attention to details; enjoy the break!

[jkitchin]

yes, that is it! I don't love the grouping, but that is a bike shed to build another day!

Key thing is basic approach is working, which is great!

So something easy enough to iterate.

I'll play with it while you're gone. If I come up with some good ideas, I may send a PR.

I definitely prefer long descriptions for the group titles. Do you think that needs to be configurable?

Anything reasonable works for me. I pushed some code that does a better job grouping. It might be fragile to changes and I predict doesn't do the right thing if the styles are changed much. The order in the cond matters. A solution here might be to move this out similar to the org-ref-cite-annotate-style function. Same with the sort function I guess.

Do the inserter's work now?

Yes; with bibtex-actions-read! I haven't looked at what you added to your README, but I'm guessing code examples should work, which is always nice.

What doesn't work is the preview mode; I still just get natbib. But should be easy enough to fix later I'd guess.

Thanks for your attention to details; enjoy the break!

I think to get another preview you can try:

1. (setq org-ref-cite-default-preview-backend 'html) which will use the export processor you have defined for the 'html backend. Presumably this would be set to 'csl for you?
2. Use #+cite_export: csl as a keyword in the document.

Hopefully it is all working now, and there are just few refinements left!

[bdarcus]

1. which will use the export processor you have defined for the 'html backend. Presumably this would be set to 'csl for you?

I was actually debating with someone else today how best to configure these. Right now I just have definitions for latex and t (as you say the fallback, which is the CSL one). Do you think I should explicitly define CSL for html and odt too?

[jkitchin]

There are two things happening here I think. One is when you export a document, you need to pick an org-cite-processor for the backend to use. org-cite-export-processors is used to get it. In your case, for latex you would get a specific processor, say org-ref-cite, or natbib or biblatex (or whatever you set it to be), but for all others you would get csl. It is unambiguous to me what should happen here because you have chosen a backend to export to. If latex is the only exception to csl, what you have done will do what you need.

The other thing I am using org-cite-export-processors for is previewing. Here it is ambiguous what backend to use for making the preview, because you have not chosen a backend by exporting yet, and we have to infer what the most reasonable choice is. For me, the fallback case is not what I ever want to see, because most of the time I will export to latex, so that should be the default for preview for me (and it is set in org-ref-cite-default-preview-backend which is a defcustom). In the event I have a document that is going to be exported to html though, the latex preview is not helpful, and then I would use #+cite_export: csl for example in the document, and then the previews would use the local definition and use csl to generate the previews.

I also don't want to make the fallback case to something 'org-ref-cite, because when I export to ascii, e.g. in emails, or to odt, etc, I don't want to see latex exports there. I want either basic or csl.

I simpler solution to this I just thought of would be to add a (preview processor) line to org-cite-export-processors. Assuming that doesn't cause an issue, then you would be able to declare what processor to use for previews by default. It might be a little more awkward to change via #+cite_export: csl, and isn't a proper backend for export though.

[bdarcus]

I'm asking, BTW, not only for me and for this code, but because I set this in the Doom PR. E.g. what I do will be what a lot of other people end up doing, so I figure should do it right.

If latex is the only exception to csl, what you have done will do what you need.

So far, the only export processors are tex-based, and CSL, and the former can't do HTML or ODT, so that was my thinking.

[bdarcus]

I'm going to keep this as a running list of comments on the latest UI, that I'll update over the next week, as I think about things:

1. Given I don't use natbib at all, or have it configured anywhere for oc, I should not see these styles; but I do. Beyond tweaking the logic that determines what gets displayed, maybe we need a defcustom for the default if it can't be 100% reliable?
2. should avoid duplicate full/shortcut variants, as it dramatically expands the number of style candidates, for no obvious benefit. If necessary, maybe this could also be configurable, but maybe not initially?
3. Group titles should be as informative and output-agnostic as possible; why I replaced "text" with "Textual/Narrative". I'll think more about specifics on this.
4. Not liking the "misc" group; I had this "default"? Another thing I'll think about.
5. Probably should sort the groups based on how common somehow? LIke, "default" should certainly be first, and maybe "text" second? This is less an issue if 2 is addressed though, and I do realize if construct the UI from org-cite-supported-styles; but "default" applies to all export processors.

[bdarcus]

On 2 above, here's what I currently see; that first group should only have four candidates.

Maybe could have a defcustom that toggled short vs long for style and variant, with possible values of:

• long-long
• long-short (probably default)
• short-short

Am I right this should be straightforward to handle?

I guess the only issue is existing documents that don't use the same pattern; but that seems a minor issue on balance best to ignore for now?

cc @andras-simonyi

[jkitchin]

I'm going to keep this as a running list of comments on the latest UI, that I'll update over the next week, as I think about things:

1. Given I don't use natbib at all, or have it configured anywhere for oc, I should not see these styles; but I do. Beyond tweaking the logic that determines what gets displayed, maybe we need a defcustom for the default if it can't be 100% reliable?

I see the issue, but not a solution at the moment. The issue is that my code uses the variable org-ref-cite-styles to populate the selection, and not (org-cite-supported-styles). These two things are not currently compatible because in org-ref-cite I combine the style and export command to avoid duplication elsewhere, and (org-cite-supported-styles) does not provide an export command. In the annotate function I have tried setting it up to use export to generate these, which might help a little on annotation, but it is very slow for csl/html for some reason at the moment.

This way, in principle the solution might be to set org-ref-cite-styles to the styles you use.

It would be a lot of redirection to try to get the styles from (org-cite-supported-styles) (again having to guess which export processor to grab them from), and then construct the desired list (combinations of long/short), and then export each candidate accordingly.

2. should avoid duplicate full/shortcut variants, as it dramatically expands the number of style candidates, for no obvious benefit. If necessary, maybe this could also be configurable, but maybe not initially?

The simplest way to avoid this is to setq org-ref-cite-styles to have what you want in it, e.g. don't list the ones you don't want to see. So, if you want long-short, it would look like: ((("text") ("b")) . "\\citealt") for example.

The problem I see is there is only a convention that you put ("long" "short") in that order, it is not required. So, there is no way to be sure a setting like long-short will actually be that. Something like first-second would be more consistent, but less obvious too.

I set this up with a default to short-short (my preference) to try it out anyway.

3. Group titles should be as informative and output-agnostic as possible; why I replaced "text" with "Textual/Narrative". I'll think more about specifics on this.

I don't have strong feelings here, at least until ivy supports groups.

4. Not liking the "misc" group; I had this "default"? Another thing I'll think about.

There is only one that ends up there at the moment (the one that maps to \cite). I don't feel real strongly about it, but the grouping function is pretty fragile imo, and relies on heuristics defined in org-ref-cite. It won't work for arbitrary styles defined by others. For that, we would have to redefine the style data structure to include group info. in any case there needs to be a fall-through group to catch everything that doesn't end up in a group, and that group needs a name.

The really bare \cite could be default, it is actually what we use the most because of a latex package called natmove which only works on \cite commands I think (and not on \citet). I haven't gotten to use this to write an actual paper yet though, so I haven't had a chance to see how the current choices actually work in practice.

5. Probably should sort the groups based on how common somehow? LIke, "default" should certainly be first, and maybe "text" second? This is less an issue if 2 is addressed though, and I do realize if construct the UI from org-cite-supported-styles; but "default" applies to all export processors.

I guess this could also be a defcustom. text is first for me always. Miscellaneous is actually second, because we use citenum somewhat frequently, e.g. see Ref. \citenum{key} so the citation number is regular text, and not stylized. Clearly this is user dependent.

I don't see how one sorts groups in completing read though. Is there a way to do that?

[bdarcus]

It would be a lot of redirection to try to get the styles from (org-cite-supported-styles) (again having to guess which export processor to grab them from), and then construct the desired list (combinations of long/short), and then export each candidate accordingly.

So there's two questions I see here, assuming there's value in a general style selector untied to any particular export-processor:

1. a basic UX one: what styles and previews should the style selector present to the user?
2. if it's at least optionally the styles supported by the active export processor, should we propose org-cite-supported-styles add an optional parameter to limit the returned result to some export processor?

Because the design of org-cite and its included processors assumes consistent style and variant names across export processors, and has a fallback logic for unsupported styles and/or variants, my assumption on 1 is just present all of what org-cite-supported-styles returns, complete with preview, which will also indicate whether fallback is active.

Like, I have oc-biblatex and oc-csl configured. The former has a wider range of style/variants, but I have no problem seeing and using those, even if my primary export processor is oc-csl.

Does that make sense?

If there's also value in restricting the displayed style options only to whatever export processor is active, should we propose 2 (or if you have FSF copyright assignment, just submit a patch)?

cc @andras-simonyi

[bdarcus]

I don't see how one sorts groups in completing read though. Is there a way to do that?

I'm actually a little unclear on some details here, but this is what I have:

                 `(metadata
                   (annotation-function . oc-bibtex-actions--style-preview-annote)
                   (cycle-sort-function . identity)
                   (display-sort-function . identity)
                   (group-function . oc-bibtex-actions--styles-group-fn))

It puts "default" and then "textual" first, but I'm not clear why now that I look.

[jkitchin]

It would be a lot of redirection to try to get the styles from (org-cite-supported-styles) (again having to guess which export processor to grab them from), and then construct the desired list (combinations of long/short), and then export each candidate accordingly.

So there's two questions I see here, assuming there's value in a general style selector untied to any particular export-processor:

1. a basic UX one: what styles and previews should the style selector present to the user?
2. if it's at least optionally the styles supported by the active export processor, should we propose org-cite-supported-styles add an optional parameter to limit the returned result to some export processor?

org-cite-supported-styles already supports this, you just pass a list of export processors that you want styles for.

We are still stuck guessing what the export backend/processor is here though because it isn't actually defined until you run an export. I added some code to try doing this. It sort of works, but feels slow at the moment for annotation.

Because the design of org-cite and its included processors assumes consistent style and variant names across export processors, and has a fallback logic for unsupported styles and/or variants, my assumption on 1 is just present all of what org-cite-supported-styles returns, complete with preview, which will also indicate whether fallback is active.

Like, I have oc-biblatex and oc-csl configured. The former has a wider range of style/variants, but I have no problem seeing and using those, even if my primary export processor is oc-csl.

Does that make sense?

Why do you want to have long-long, long-short, etc... filtering of the styles then?

If there's also value in restricting the displayed style options only to whatever export processor is active, should we propose 2 (or if you have FSF copyright assignment, just submit a patch)?

I don't think I did the right thing in the commit a4809a2. I need to make it only filter what is shown, not remove them. Otherwise, two users with different preferences won't be able to use the same document because the styles won't match. I don't like the complexity this adds.

cc @andras-simonyi

[bdarcus]

org-cite-supported-styles already supports this, you just pass a list of export processors that you want styles for.

Ah, right.

We are still stuck guessing what the export backend/processor is here though because it isn't actually defined until you run an export. I added some code to try doing this. It sort of works, but feels slow at the moment for annotation.

So then isn't the easy solution just have it configurable, and let the user decide what they want to see?

(setq org-ref-cite-styles-include '(org-ref-cite csl biblatex))

Why do you want to have long-long, long-short, etc... filtering of the styles then?

Just that I think it's confusing, particularly for new users that won't understand.

And for me personally, it's also kind of overwhelming.

It doesn't bother you?

Edit: with forthcoming enhancements to oc-csl, the list of csl and biblatex styles is awfully similar:

ELISP> (org-cite-supported-styles '(biblatex))
((("author" "a")
  ("caps-full" "cf")
  ("full" "f")
  ("caps" "c"))
 (("locators" "l")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b"))
 (("noauthor" "na"))
 (("text" "t")
  ("caps" "c"))
 (("nil")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b")))

ELISP> (org-cite-supported-styles '(csl))
((("author" "a")
  ("caps-full" "cf")
  ("caps" "c")
  ("full" "f"))
 (("noauthor" "na")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b"))
 (("year" "y")
  ("bare" "b"))
 (("text" "t")
  ("caps-full" "cf")
  ("full" "f")
  ("caps" "c"))
 (("nil")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b")))

[jkitchin]

I don't see how one sorts groups in completing read though. Is there a way to do that?

I'm actually a little unclear on some details here, but this is what I have:

                 `(metadata
                   (annotation-function . oc-bibtex-actions--style-preview-annote)
                   (cycle-sort-function . identity)
                   (display-sort-function . identity)
                   (group-function . oc-bibtex-actions--styles-group-fn))

It puts "default" and then "textual" first, but I'm not clear why now that I look.

I can't tell either, I think that the group order comes from the lexical ordering of the candidate strings, in other words that the candidates are sorted, then grouped, and the groups show in the order they appear in the candidates.

If you change display-sort-function you can change the order of the groups, e.g. by reverse alphabetically sorting them. To prescribe a group order, you have to sort the candidates by group, say by assigning an integer to each group in the order you want them in. Here is an example of getting the group order to be medium, low, high. I probably wouldn't support doing this though.

(completing-read "choose: "
         (lambda (str pred action)
           (if (eq action 'metadata)
               `(metadata
             (display-sort-function . (lambda (candidates)
                            (sort candidates (lambda (a b)
                                       (let ((g1 (cond
                                          ((string< a "3")
                                           2)
                                          ((string< a "6")
                                           1)
                                          (t
                                           3)))
                                         (g2 (cond
                                          ((string< b "3")
                                           2)
                                          ((string< b "6")
                                           1)
                                          (t
                                           3))))
                                     (< g1 g2))))))

             (group-function . (lambda (candidate transform)
                         (if transform
                         candidate
                           (cond
                        ((string< candidate "3")
                         "low")
                        ((string< candidate "6")
                         "medium")
                        (t
                         "high"))))))
             (complete-with-action action '("5" "9" "1" "2" "4" "08") str pred))))

[bdarcus]

I think that the group order comes from the lexical ordering of the candidate strings, in other words that the candidates are sorted, then grouped, and the groups show in the order they appear in the candidates.

That makes sense, and explains why the "/" ones go first (though why does "text" go second for me?).

I wonder if it's possible to sort groups by recency of member selection (which is default behavior in a lot of completion UIs without the grouping)?

In any case, group order is much less of an issue if there are only 10 or 15 candidates rather than 40 or 50. This is what I currently have, which is based on static previews, showing the long-short options only.

[bdarcus]

This should work for the configurable candidates list; just need error handling when a processor isn't defined but specified for PROC ...

(defcustom org-ref-cite-styles-format 'long
  "Style format; whether to use full style names or shortcuts.
Variants will always use shortcuts."
  :type '(choice
          (const long)
          (const short)))

(defcustom org-ref-cite-style-targets nil
  "Export processor targets to include in styles list.

If nil, use 'org-cite-supported-styles' without arg."
  :group 'org-ref-cite
  :type '(repeat :tag "org-cite export processor" symbol))

(defun org-ref-cite--flat-supported-styles (&optional proc)
  "Return a flat list of supported styles.

Converts 'org-cite-supported-styles' to a flat list for use
as completion candidates.

With PROC list, limits to specific processors."
  (let ((styles (list)))
    (cl-loop for s in
             (org-cite-supported-styles
              (or proc org-ref-cite-style-targets)) do
             (let* ((style-name
                     (if (eq 'long org-ref-cite-styles-format)
                         (caar s)(cadar s)))
                    (style
                     (if (string= "nil" style-name) "/" style-name)))
               (push (if (string= "" style) "/" style) styles)
               (cl-loop for v in (cdr s) do
                        (push (concat style "/" (cadr v)) styles))))
    styles))

Result:

("year/b" "year" "/b" "/c" "/bc" "/" "text/c" "text/f" "text/cf" "text" "noauthor/b" "noauthor/c" "noauthor/bc" "noauthor" "locators/b" "locators/c" "locators/bc" "locators" "author/f" "author/c" "author/cf" "author")

[jkitchin]

I think the function in https://github.com/jkitchin/org-ref-cite/blob/main/org-ref-cite-core.el#L304 should cover this.

[bdarcus]

I got some input from the author of the group function (and of vertico), on how group-sorting works.

The group order follows the candidate order. The group of the first candidate comes first. If you want the groups to be sorted, you should sort the candidates first and specifiy display-sort-function=identity. The reason for this is, that completion styles like flex actively reorder candidates in order to get the best match first. Furthermore the completion UI may sort by history position. Grouping should not interfere with these abilities. If you set display/cycle-sort-function to identity you opt out of this, which may be a good idea in some special cases, e.g. consult-line. But I doubt you have such a case here and I generally discourage interfering with sorting.

Basically, your hunch was right.

But if we don't specify a display-sort function at all, the completion UI will usually order the groups based on the history.

E.g. if you use the "text" styles most frequently, that group will come first.

I just tested this with vertico, and it works well.

Upshot: remove the display-sort/cycle-sort lines and we don't have to worry about it.

Can also do this, for the initial display:

          ;; Sort the list upfront, but let completion UI handle beyond that.
         (let ((styles (sort (foo--flat-supported-styles) 'string-lessp))))

[bdarcus]

BTW, this is an experiment, but here's using faces to distinguish styles sans variants from styles with variants; the same faces I use to distinguish main bib entry metadata from secondary.

Just to give a hint to users. I'm still not sure though.