consult-xref not exporting to a xref major-mode buffer on embark-export

[slayer152]

Hi Daniel,

I'm not sure if this is a consult issue or an embark issue or something else; please let me know if it should be moved or I need to customize something.

Repro with emacs -Q:

(progn
  (package-initialize)
  (require 'consult)
  (require 'consult-xref)
  (require 'embark)
  (require 'vertico)
  (require 'embark-consult) ; not needed here?
  (global-set-key (kbd "C-c a") 'embark-act)
  (setq xref-show-xrefs-function          #'consult-xref
           xref-show-definitions-function #'consult-xref)
  (vertico-mode))

With the above config, if I:

1. M-x dired and select a directory
2. mark some files in the dired buffer
3. M-x dired-do-find-regexp and provide a regexp, I get a minibuffer prompt Go to xref: with a list of the results.
4. However, if I run embark-export on the list of results, I get a normal Embark Collect buffer with major-mode as embark-collect-mode (and not xref--xref-buffer-mode which would allow me to use M-g M-n and M-g M-p compilation-mode-like bindings to navigate the results).

However, If I don't use the above 2 setq xref-show... bindings and I run M-x dired-do-find-regexp and provide a regexp in step-3, I directly get a buffer with major-mode as xref--xref-buffer-mode which allows me to use M-g M-n and M-g M-p bindings to navigate the results. Would it be possible to use consult-xref and also embark-export to a xref--xref-buffer-mode buffer if needed.

Thanks.

[minad]

This is an xref/embark issue. As far as I recall, export is not possible since the candidates don't carry sufficient information. This has been discussed before on the embark issue tracker. You can use the normal xref buffer mode instead of the completing-read UI.

[minad]

I remember now - the issue was that the builtin xref completion function doesn't provide sufficient information but in principle we could make embark-export work for consult-xref. See the discussion from https://github.com/oantolin/embark/issues/162#issuecomment-784037664. Due to the impossibility to support the builtin we didn't add such an export functionality to Embark or embark-consult. But you can create an issue on the Embark issue tracker. Maybe @oantolin is willing to reconsider, adding a special exporter for consult-xref to embark-consult. An embark export would give additional justification for the existence of consult-xref besides Consult preview (See #446) ;)

[oantolin]

I'm game! If consult-xref adds the necessary information (and switches to using its own completion category), I'd be quite happy to add an exporter to embark-consult.

[minad]

@oantolin See https://github.com/minad/consult/commit/a96ec24784291153cff731deb4024dd3d3bd9967 :)

[oantolin]

Great! I'll get to work on the exporter.

[oantolin]

I added the exporter @minad, but it currently doesn't have revert functionality. There's actually a problem with that. Typically you obtain such an export buffer as follows:

• Run a command like project-find-regexp or dired-do-find-regexp.
• Input a regexp.
• At the consult-xref prompt type some filter.
• Now embark-export.

The revert=rerun functionality here does the following: it correctly reruns the command from the first step, but instead of injecting the regex from the second step, it will inject the filter typed at the consult-xref prompt. The regexp typed in the second step is never captured anywhere, so we cannot easily reenter it.

In the xref buffer g is bound to xref-revert-buffer which currently does nothing. You can see Embark's attempted revert=rerun by running revert-buffer.

[minad]

Okay, but having the export is already a big improvement even if we don't have revert :)

[oantolin]

Yes! Definitely! Plus it kind of works. It reruns the correct command, but doesn't know what to insert at its prompt.

[minad]

We could use the last value from the consult-xref--history if it would be an input history. (EDIT: But then, it wouldn't work since embark-export and embark-collect do not update the history...)

[minad]

Regarding the history we had the discussion in https://github.com/oantolin/embark/issues/452. It is truly bad that Emacs does not come with the concept of an input history. This is really missing. I often find the input more useful than the actually selected candidate, in particular for search-like commands. On the other hand, the candidate history lets us sort the candidates by history.

[oantolin]

You know, with access to the fetcher function (for example if you stuck in a another text property), we could simulate revert=rerun.

The xref--show-xref-buffer function recieves the fetcher, but can also take an alist entry with pre-fetched candidates. I can pass the exported items as pre-fetched so the correct xref buffer is bulit at first; and I could also pass a fetcher function that calls the original fetcher and lets you filter it's results.

Even just passing the original fetcher makes revert actually redo some search.

[minad]

Hmm, I am hesitant to give you access to the fetcher :-P

1. I am worried of memory leaks and serializability. Is the fetcher a function or a closure?
2. Maybe the fetcher is one shot and shouldn't be called again?

(EDIT: But probably the whole idea of the fetcher is that it can be called repeatedly. Why would there be a fetcher in the first place?)

[oantolin]

It is meant to be called again for the revert functionality.

[oantolin]

xref-revert-buffer calls (funcall xref--fetcher).

[minad]

I exposed the fetcher in https://github.com/minad/consult/commit/8fab8a0061e030bf1d0d583f5b4da687df564d14. I could also put it into a local minibuffer variable, but let binding it should work too for embark-export?

[minad]

But if you just use the fetcher for export, we won't get the minibuffer candidate filtering? Or do you want to rerun consult-xref with the fetcher? You could capture the minibuffer-contents during export and also inject that to consult-xref. I should probably expose an initial input argument then.

[oantolin]

I did mean to rerun consult-xref on the result of the fetcher.

[oantolin]

Embark already captures the minibuffer contents, but doesn't expose them nicely, they're just in a local variable of the revert function.

[minad]

I did mean to rerun consult-xref on the result of the fetcher.

Yes, or just pass the stored fetcher to consult-xref? Do you also need the initial input then to easily inject the minibuffer contents? I mean you could also use the normal Embark injection mechanism but this seems like more effort.

[oantolin]

I've got to go now, but with the local variable you exposed I should be able to figure this out tomorrow.

[oantolin]

Mmh. There's a minor decision to be made here. Embark export currently runs the exporter after quitting the minibuffer, this is to simplify window management: just let the exporter decide what to do. In particular this means that by the time the exporter runs the local variable you helpfully exposed @minad is no longer there (nor are the minibuffer contents).

I see basically two strategies to deal with this:

• Add a pre-export hook that runs in the buffer you called export from, usually a minibuffer —note that we already have a post-export hook that runs in the exported buffer.
• Run the exporter before quitting the minibuffer, afterwards use pop-to-buffer to redisplay it.

Do you have any thoughts on which approach is better @minad?

[oantolin]

Oh, upon further thought, maybe running the exporter earlier is actually simplest: a pre-export hook for this xref exporter would need to stash the fetcher somewhere it can get at later. It just feels a little dirty to display the buffer, have minibuffer quitting restore the window configuration, and then display it again... Maybe there's a third way.

[minad]

I am confused. How do you obtain the candidates when the minibuffer is already closed? This also applies to Embark collect where you obtain the annotations for example and as far as I know Marginalia expects the annotation functions to be run in the active minibuffer.

[minad]

I would collect the candidates, store them somewhere in a closure (run-at-time?), quit the minibuffer, create and display the export buffer. This is the approach I had assumed is already there.

[oantolin]

I would collect the candidates, store them somewhere in a closure (run-at-time?), quit the minibuffer, create and display the export buffer. This is the approach I had assumed is already there.

That is exactly what embark-export does!

[minad]

Ah okay, I got the problem now. The ~exporter~ collector and the buffer creation are two different things. Hmm.

EDIT: I meant that the candidate collection process and buffer creation (performed by the exporter) happen at different points.

[oantolin]

My point is that in that approach the exporter, i.e., the function that creates the export buffer runs after quitting. It has access to the candidates because embark-export passes them in as an argument.

[oantolin]

Ah okay, I got the problem now. The exporter and the buffer creation are two different things. Hmm.

No, in the current implementation the export function is in charge of creating and displaying the buffer. I did it that way to reuse existing Emacs commands as much as possible.

[minad]

I think one should implement it like this:

1. Collect the candidates
2. Create the buffer but do not display it. Is this possible? The exporters would be changed to return the created buffer.
3. Quit the minibuffer
4. Display the buffer (pop-to-buffer?)

[minad]

Another idea: Change the candidate collector such that they return additional metadata which is only available during the collection process. This seems like the easiest approach since you don't have to mess with the buffer creation and you can reuse more of the existing Emacs facilities.

(:type file :candidates ("~/..." ...) :directory ...)
(:type consult-xref :candidates ("xref string" ...) :xref-fetcher ...)

[oantolin]

2. Create the buffer but do not display it. Is this possible?

The problem is step 2. There aren't many commands in Emacs to create one of these buffers but not display it. Maybe none, in fact? There is a dired-noselect but I think (?) it does display the buffer, just doesn't select that window.

I once tried to implement step 2 by let-binding display-buffer-alist to not display windows. All the ones in the default configuration worked except ibuffer which produced some odd errror I don't recall now.

[minad]

The problem is step 2. There aren't many commands in Emacs to create one of these buffers but not display it. Maybe none, in fact? There is a dired-noselect but I think (?) it does display the buffer, just doesn't select that window.

I agree that the approach of not displaying the buffer is not feasible. I would try to extend the collector (https://github.com/minad/consult/issues/557#issuecomment-1107477609).

[oantolin]

I don't think extending the collector is feasible, it puts the responsibility in the wrong place!

Only the exporter for xref knows it needs to store the fetcher. Why should every candidate collector for every completion UI start checking if there is a fetcher to store or not? I'd have to change these functions to store the fetcher:

• embark-sorted-minibuffer-candidates
• embark-minibuffer-candidates
• embark--selectrum-candidates
• embark--vertico-candidates
• embark--ivy-candidates

[oantolin]

EDIT: I meant that the candidate collection process and buffer creation (performed by the exporter) happen at different points

Oh, yes, the collector -> exporter typo really confused me.

[minad]

I don't think extending the collector is feasible, it puts the responsibility in the wrong place!

Hmm. For target finders we have target type specific finders. For collectors the situation is a bit different as you noted. There we have these collectors which are UI-specific and in some sense too general and too lossy. I'd argue that the collectors are in principle the correct place to obtain all the needed information. But updating all the collectors doesn't seem like a good idea, in particular since they would mix in something Consult specific for consult-xref. What about changing the exporters such that they operate as a two step process?

(defun example-exporter (candidates)
   ;; do some preprocessing here, we are still running inside the minibuffer
   (lambda ()
     ;; This happens after quitting
     (create-buffer...)
     (pop-to-buffer ...)))

[oantolin]

I like that last suggestion, it would definitely work, but does represent a breaking change with the current contract for exporter: that they receive a list of candidates and they "return" a value by making some buffer current.

But an alternate design can even add the possibility of metadata collection in a backwards compatible way. I could add a variable embark-export-context-function-alist keyed by type. Then embark-export could collect the candidates, run the context function for the given type, and later once it has quit the minibuffer and is running the exporter it can let-bind a variable to the return value of the context function.

Mmh. What I don't like about that is that the context function is really part of the exporter and that design would force you to split it up. I wonder how many people have written their own exporters? Maybe a breaking change is not so terrible.

Also, maybe redisplaying the buffer is not so bad either. I should test that. Maybe just running the exporter before quitting and afterwards redisplaying is good enough.

[minad]

I would definitely go with the breaking change for the exporter. Introducing another hackish context collector variable does not seem like a good solution. As you noticed I had multiple breaking changes in Consult recently and my experiences have been good. You can keep some backward compatibility code in place for a short time. In Consult I moved rather quickly there since the packages which depend on Consult got updated within a few days. Furthermore in Consult I also took also a radical approach for the :state function breaking change. I considered implementing this in a backward compatible way but this would have meant a lot of additional work and code duplication, so I even went with a variant which only guarded the function calls, printing an error message for the incompatible calling convention and disabled preview.

[minad]

Also, maybe redisplaying the buffer is not so bad either. I should test that. Maybe just running the exporter before quitting and afterwards redisplaying is good enough.

The redisplay idea doesn't look nice to me. It may even introduce flicker if somewhere in between if a redisplay is forced. I prefer the two step process - it seems like a logical approach to me. Collecting, preprocessing and capturing additional context, quitting, creating and displaying the buffer. I would say that such processes are things which are good to clean up before going with a 1.0 version.

[oantolin]

I agree that the context function variable is not a good solution. But maybe we don't need a breaking change either. I'm trying simply calling the exporter earlier and popping to the buffer after quitting. It works great! I was worried you'd see the window appear, disappear and reappear, but no: you only see it appear onve.

[minad]

I agree that the context function variable is not a good solution. But maybe we don't need a breaking change either. I'm trying simply calling the exporter earlier and popping to the buffer after quitting. It works great! I was worried you'd see the window appear, disappear and reappear, but no: you only see it appear onve.

But what about exporters which display the buffer directly? What if there is a redisplay call? (My opinion is generally to get it right and go with a more flexible protocol, similar to what I did with the Consult :state function overhaul.)

[oantolin]

All current exporters display the buffer directly! :)

[minad]

Hmm, if it works well, no need to introduce a complication. We could still go with the more complicated solution if it turns out to be necessary for some hypothetical future exporter :shrug:

[oantolin]

Ah, you beat me to it: I was writing a comment that said, "How about this: let's just go with the redisplay solution and if we notice flicker or other problems, then we make the breaking change?"

[oantolin]

Does Emacs only redraw between commands (part of its single-threadedness)? I think that may be the case and that's why you see no flicker with this solution.

[minad]

Does Emacs only redraw between commands (part of its single-threadedness)? I think that may be the case and that's why you see no flicker with this solution.

Yes. It only redisplays when actively asked for it or in the event loop when waiting for input. (EDIT: But I think even in the event loop Emacs has some logic in place to check if a redisplay can be avoided or if only a partial redisplay is necessary.)

[oantolin]

So unless people specifically call redisplay in the exporter we should be OK.

[minad]

So unless people specifically call redisplay in the exporter we should be OK.

Yes. I image redisplay to be useful for slow exporters, for example the variable exporter. It could populate the buffer partially, redisplay and then show some progress in the echo area. But we don't have that currently so we are good.

[oantolin]

OK, I think this works reasonably well: oantolin/embark@573f2bd12e339e5a50ef9e464e90436d9ab9ce6e

[slayer152]

Awesome, works well for my use case. Thanks, Omar and Daniel.

RE: xref-revert-buffer after embark-export I'm seeing something unexpected with emacs -Q and the setup in my first comment (I haven't ever used xref-revert-buffer in my workflow, so doesn't affect me directly but tried it out of curiosity and want to mention the behavior I'm seeing).

• M-x dired-do-find-regexp
• Input a regex
• Call embark-export without using a second filter at consult-xref prompt.
• An xref buffer shows up with the candidates and I'm able to use next-error and previous-error as expected.
• If I run xref-revert-buffer, the first file with matching regexp text is loaded into the xref window but the buffer is erased (fortunately, the erase is not persisted and I can revert the buffer and get back the contents)

[oantolin]

• If I run xref-revert-buffer, the first file with matching regexp text is loaded into the xref window but the buffer is erased (fortunately, the erase is not persisted and I can revert the buffer and get back the contents)

I'm a little confused by this. What is supposed to happen is that the xref buffer is killed, and you are plopped into consult-xref again with the new search results (from all files, not just the first). You can modify the filter consult-xref applies on top of them and when you press RET, a new xref buffer is created.

I thought I had tested that, let me look into it.