Interruption?

[jdtsmith]

This is such an interesting approach, similar to but also quite different than what I took. I wonder if you've had any thoughts on handling interrupts? One of the key design goals of iPy is to (try to) never get in the way, so if auto-completion or eldoc or something has a command going in the background, and something else urgent comes up (like a new completion prefix), that pending command is interrupted and cancelled. This works... mostly, but I've found on remote systems with higher latency it works less well, and my command queue fills. Would be happy to chat about this some time.

[astoff]

Right, this is a rather general issue. Currently in my code, completion will only start if the subprocess is not doing other work, such as running eval or fetching eldoc. And the completion request gets interrupted on the Emacs side if there's new input but the inferior process is not interrupted; IOW, it remains busy until it returns that completion request (whose result is immediately thrown away).

I suspect this might be problematic when using auto-complete as opposed to hitting that TAB when you really want completions.

Also, I think there's no way around in this "single channel" communication model with the subprocess?

[jdtsmith]

Yes, I run iPy with auto-completion (from corfu), and of course eldoc is running on a timer too. I have an ipy-debug command that lets you spy on all the background traffic, and it's pretty fun to watch. My issue is, if you do not actively interrupt, you'll end up stacking up plenty of these outdated completion/syntax/doc requests. I've learned the hard way that Jedi-based completion in iPython occasionally goes searching for minutes.

So "just wait until it's ready again" is sometimes not workable. In normal iPython interactive use, the user would long ago have hit C-c, so that's the approach I take. I.e. the way around this issue is judicious use of interrupt-process. But you can't go too crazy with that or you'll end with an unhappy input loop.

I do support what I call protected commands, that demand that they run to completion, but I try to use them rarely. And I allow preempt commands to "jump the queue".

there's no way around in this "single channel" communication model

There is another comm channel: stderr. But not for remote shells (Tramp fakes it by putting both over stdin), so kind of useless. I can see why people want to use zeroMQ and such. But the burden of that approach is too high IMO, and it rules out simple "over ssh" single channel connection to remote shells.

[astoff]

After sleeping on it (to put it mildly), I concluded don't want to add any SIGINT chicanery into dREPL.

The situation here is pretty much like LSP, which works fine without it. LSP has a "cancel" message but this can only be implemented by servers with some kind of multithreading, which I'm pretty sure many of the popular and usable ones are not. If at some point I add a REPL that could make use of "cancel", then of course it can be done (well, actually the Node one probably could, but I don't care enough about that one :-)).

The preemt idea is quite interesting, I kinda thought about something similar but didn't see an immediate need yet.

Thanks for the interesting discussion! I'll close the ticket but I'm always happy to continue.

[dancol-openai]

Not supporting interruption makes drepl less useful for my use-case. Maybe that's okay --- I just have a different interaction model in mind.

I think you're thinking of drepl as "LSP, but for REPLs". I think that's great. The world needs that. For years, even before drepl came along, I'd been wanting to build the closely-related "comint, but with inferior-driven completion". I want to be able to do everything I can do in a terminal emulator, but with native Emacs facilities for completion, line editing, history, and so on. My dreams Emacs front end (which could very well be based on drepl) needs to support shell niceties like interruption, on-demand variable expansion, and job control. With what I have in mind, you get the same "experience" you might get running commands in a shell via a terminal emulator, but integrated into the broader Emacs environment.

(I think comint was originally meant as something like this, but the world has passed it by: comint's simplistic filename-based completion and regexp-based directory tracking can't capture the subtleties of today's command line interfaces.)

What I have in mind is less drepl-ipython or drepl-lua (which are fine facilities; I use the former daily now) than a drepl-comint that would drop you in $SHELL and let interact with any process that supported the protocol using a uniform interface, even if drepl didn't have any special knowledge of that process. For example, if I were to drepl-enable libreadline, then I'd be able to start bash (based on libreadline of course), get Emacs-UI completions and such, then in that bash, start gdb (which also uses libreadline) and get the same nice UI in gdb despite drepl not knowing anything in particular about gdb, bash, or readline.

You could even imagine making the protocol a multi-vendor standard so fancy terminal emulators like kitty, alacritty, and iTerm2 could provide similarly "nice" completion and editing UIs on top of standard command line tools.

It sounds like you want to take drepl development in a different direction. That's fine! It's still cool work, and I think the protocol can be used for both visions.

[jdtsmith]

I've rewritten most of iPy to provide more robust interruption via USR1 signals which are caught directly in the input loop, and it works very well, much faster and more robust than my former approach (so far). I also implemented your thinking to some degree as well: send the interrupt, but never wait on it, unless you have to. But it is not easy and likely still has some corner cases. For example, GUI toolkits hook into the internal Python event loop in unusual ways, which can prevent signal propagation (there are workarounds).

Given my experience with this in iPython, with accommodations made on both sides of the process flow, I don't think a "generic" interrupt facility feeding multiple languages is too tractable. It's just too specific to the process on the other end. So I think you are making the right call.

[dancol-openai]

Sending SIGINT is a time tested and practical approach to interruption. If it's not working, the right approach is to fix the bugs, not invent a different interruption mechanism or give up on the concept.

[jdtsmith]

Yes, and C-c C-c sends SIGINT in all comint shells, which does indeed work well enough in most cases to interrupt the command loop or current command. User-driven interruption is not the problem. Doing it robustly, in the background, on the same channel, when data (completion, eldoc, etc.) is automatically flowing between Emacs and the process, and the user is using the same channel for their own purposes... that is the challenge. And it would remain a challenge even if Python fixed >10yo bugs in their event hook handling.