Returning to the source buffer from inferior doesn't work

[vspinu]

The current implementation of python-shell-switch-to-buffer is not very practical. It stores the buffer from where the process was started. After some time that buffer need not even exists.

I think the most meaningful would be to return to either the most recent python buffer or the last buffer from where the evaluation or switch to inferiror happened. This is how ESS and Elpy, for what matter, does.

Shall I propose a pull?

[wavexx]

On Fri, Feb 19 2016, Vitalie Spinu notifications@github.com wrote:

The current implementation of python-shell-switch-to-buffer is not very practical. It stores the buffer from where the process was started. After some time that buffer need not even exists.

I guess this happens because you have a shared python process?

I think the most meaningful would be to return to either the most recent python buffer or the last buffer from where the evaluation or switch to inferiror happened. This is how ESS and Elpy, for what matter, does.

Shall I propose a pull?

I recently added the ability to track the evaluation status in the buffer, and I'm reworking a bit the evaluation logic so that -and-step functions become asynchronous, properly reacting to evaluation errors.

But this of course relies on the parent-buffer being correct at all times. I admit I never use a shared process.

Can you elaborate what you had in mind?

I guess that evaluation should always update the parent buffer if the inferior process is shared (I assumed python.el already did that!?). Likewise in switch-to-buffer, if we started a shared inferior process and the parent process is gone, we can pop to the most recent python buffer.

However I need to think how to store/update the state properly when an inferior process is shared. This is something I missed.

[vspinu]

I guess this happens because you have a shared python process?

Not really. What is shared process? There is no reference of that in python.el or python-x.el.

Can you elaborate what you had in mind?

Well, I just want to be able to return quickly from inferior to working buffer with C-c C-z. In ESS I can even switch to inferior and back with C-c C-z C-z, because C-z is cycling. That's a handy way to make the process buffer visible without having a dedicated key for showing the inferior buffer.

This functionality need not have to do anything with evaluation. Switching back to most recent python buffer is good enough. A better version would be to select most recent python buffer associated with current process (like ESS does) but given that python.el has only rudimentary provision for multi-process workflows this could wait.

[wavexx]

On Sat, Feb 20 2016, Vitalie Spinu notifications@github.com wrote:

Not really. What is shared process? There is no reference of that in python.el or python-x.el.

When you evaluate something and a new python is started, do you make a "dedicated process" or not?

Well, I just want to be able to return quickly from inferior to working buffer with C-c C-z. In ESS I can even switch to inferior and back with C-c C-z C-z, because C-z is cycling. That's a handy way to make the process buffer visible without having a dedicated key for showing the inferior buffer.

Yes, I added it for the same reason. I cycle back&forth from and to to the inferior buffer often.

However it's working nicely for me.

This functionality need not have to do anything with evaluation. Switching back to most recent python buffer is good enough. A better version would be to select most recent python buffer associated with current process (like ESS does) but given that python.el has only rudimentary provision for multi-process workflows this could wait.

Since I tie each buffer to it's own process and I have several running, I definitely don't want to pop back to an unrelated buffer. This is what python.el describes as "dedicated process". I just named "shared" the opposite.

Can you describe how do you use the process in general?

You open emacs, edit a file, start a new process, but then edit a new python source (same project probably?) and use the python instance?

[vspinu]

Since I tie each buffer to it's own process and I have several running, I definitely don't want to pop back to an unrelated buffer. This is what python.el describes as "dedicated process". I just named "shared" the opposite.

I see.

Can you describe how do you use the process in general?

I start a process with run-python (shared in your terminology). I work with it from multiple files (same project). The problem is that python-shell-switch-to-buffer returns to the buffer that started the process. If that buffer has been killed it throws the error Invalid buffer.

Sometimes I create a dedicated process. While it works for one buffer I find the workflow very limited because I cannot associated multiple files with the new process nor I can switch associations. In ESS I can any time switch the process associated with the buffer (C-c C-s) or create a new process for current buffer with the same command.

I definitely don't want to pop back to an unrelated buffer.

Yes. This is why I was proposing to switch to the most recent buffer associated with the current process.

To push it even further, one can have C-z to go through the stack of associated buffers. Thus, C-c C-z goes to most recent buffer, C-c C-z C-z switches to the second most recent buffer and so on in a loop.

[wavexx]

On Sun, Feb 21 2016, Vitalie Spinu notifications@github.com wrote:

Sometimes I created a dedicated process. While it works for one buffer I find the workflow very limited because I cannot associated multiple files with the new process nor I can switch associations. In ESS I can any time switch the process associated with the buffer (C-c C-s) or create a new process for current buffer with the same command.

I think we need some tweaks to python.el's process logic from the get-go. So imagine I had two buffers with two inferiors (two "dedicated" python processes).

I work a bit, and now I want to edit some file and associate that to inferior #1. I never tried, but if I answer "no" to a dedicated process, does python.el ask me to which inferior to attach to?

Or there's only one "shared" process? That would be incredibly limiting for those that use an emacs daemon.

Yes. This is why I was proposing to switch to the most recent buffer associated with the current process.

We can definitely have a list of associated buffers from the inferior by looking at buffers pointing back to it. Not pretty, but it would work.

To push it even further, one can have C-z to go through the stack of associated buffers. Thus, C-c C-z goes to most recent buffer, C-c C-z C-z switches to the second most recent buffer and so on in a loop.

Would you have time to propose something in that regard?

If you could split the work in a function that gets all the buffers for the current inferior, I'd use that in a few other places to update the process status correctly.

[vspinu]

I think we need some tweaks to python.el's process logic from the get-go.

I think there are multiple issues with python.el that could be better. What is the contributing to python.el looks like these days? Who is the defacto maintainer of that code?

does python.el ask me to which inferior to attach to?

No. It does literally nothing.

We can definitely have a list of associated buffers from the inferior by looking at buffers pointing back to it. Not pretty, but it would work.

An explicit list is not needed. You just need a buffer local variable pointing to the current process. Each time you execute a code or perform some other action that requires an inferior and there is no association you either automatically associate to the inferior when it's the only running process or ask the user to which inferior to associate when there are multiple processes. This is how ESS works, simple and efficient. This logic can be extended to take into account projects. The you need not be prompted each time you open a file within the same project.

I don't see the point of the distinction between shared and dedicated processes. I think all processes must have equal standing and should be available for associations from any buffer or project.

Would you have time to propose something in that regard?

Sure. But the python.el/python-x.el division will probably be a handicap here.

[wavexx]

On Sun, Feb 21 2016, Vitalie Spinu notifications@github.com wrote:

I think we need some tweaks to python.el's process logic from the get-go.

I think there are multiple issues with python.el that could be better. What is the contributing to python.el looks like these days? Who is the defacto maintainer of that code?

Fabián Gallina is the author and current maintainer. It is actively maintained, but you'd need to clone repository emacs itself (at least with a subtree) to get it.

Would you have time to propose something in that regard?

Sure. But the python.el/python-x.el division will probably be a handicap here.

I know what you mean.

But there's some stuff I'd really don't want to touch, such as basic syntax highlighting, indentation, etc. I think python.el does already an excellent job here, and even keeping a copy would be a burden to maintain.

There is also python-mode by the way, for which I was able to convince the author to make a package for melpa. python-mode has already some of the stuff I had to reimplement in python-x (such as exception handling).

In fact, I used python.el as a base because it was smaller and I knew some parts of it better than python-mode, but maybe it's time to revisit.

Do you have some recent experience with it? My last time I used python-mode was a couple of years ago with emacs 23.

[vspinu]

Do you have some recent experience with it?

Actually not. I learnt python about 3 weeks ago and I played with elpy and anaconda so far but didn't have time to try python-mode as yet. The code base is huge but it seems very clean and contains a comprehensive evaluation engine.

but maybe it's time to revisit.

Does that mean that you would consider moving python-x on top of python-mode?

Nobody wants to do double work. If python-mode is the "way to go" then I would be more than happy to help porting parts of python-x that are missing in python-mode.

[wavexx]

On Sun, Feb 21 2016, Vitalie Spinu notifications@github.com wrote:

but maybe it's time to revisit.

Does that mean that you would consider moving python-x on top of python-mode?

Nobody wants to do double work. If python-mode is the "way to go" then I would be more than happy to help porting parts of python-x that are missing in python-mode.

I'm all for a better evaluation/interaction mode for python. This is why I wrote python-x. It's not because I wanted to randomly write elisp ;)

I don't care which mode is it based on.

There's also the other option I talked about in the beginning, which would be to write a python backend for ESS itself. It would mean to rip the syntax/indentation engine from python.el (or python-mode) and use ESS for the rest. It makes a lot of sense to me, and might actually be easier from a technical perspective.

I honestly don't know which is better. I generally don't have that much time to dedicate to experimenting with a different mode and see the advantages/disadvantages. I have a lot of customizations I made to python.el so I'm a bit too biased right now as I just need the thing to work.

[vspinu]

There's also the other option I talked about in the beginning, which would be to write a python backend for ESS itself. It would mean to rip the syntax/indentation engine from python.el (or python-mode) and use ESS for the rest. It makes a lot of sense to me, and might actually be easier from a technical perspective.

Hm. I completely missed this part. That actually makes sense indeed. ESS already has a solid multi-process framework and we are moving it into dissagregation anyways (R-mode, julia-mode etc instead of one monolitc ess-mode). For historical reasons ESSs codebase is quite a mess and it's rather fragile. You cannot change something without running a risk to break some old setup for some dude with 10 years old S+ versions.

I will try python-mode and if it's not friendly enough and suffers from same issues as python.el does then I will make an attempt to built an ESS submode for that.

[wavexx]

On Sun, Feb 21 2016, Vitalie Spinu notifications@github.com wrote:

Hm. I completely missed this part. That actually makes sense indeed. ESS already has a solid multi-process framework and we are moving it into dissagregation anyways (R-mode, julia-mode etc instead of one monolitc

I was personally impressed how fast julia-mode came to be. It's a bit disheartening that python doesn't have something equivalent despite being much more mature.

ess-mode). For historical reasons ESSs codebase is quite a mess and it's rather fragile. You cannot change something without running a risk to break some old setup for some dude with 10 years old S+ versions.

That's the only risk, basically. It's of course much bigger and cruftier to modify.

Meanwhile I'll have a look at making buffer cycling working for shared processes.

[wavexx]

I did a couple of checks, and It should work correctly now for both multiple-dedicated and shared processes. In case the buffer is gone, it will pop to the most recent associated buffer.

I also added python-shell-switch-to-shell-or-buffer (now bound to C-z by default), which is cycling.

[wavexx]

Still, I'm a bit disappointed how new processes are spawn. Indeed, the moment you create a non-dedicated process, python.el never prompts again.

[fgregg]

There's also the other option I talked about in the beginning, which would be to write a python backend for ESS itself.

This sounds fantastic! Have there been any developments in this direction?

[wavexx]

On Fri, Jul 07 2017, Forest Gregg wrote:

This sounds fantastic! Have there been any developments in this direction?

Not on my part. I had a few attempts over the course of some weekends, but went nowhere. There's quite a bit of rework to make it a reality.

Also, I've been using the ess/julia mode more often than python-x lately ;)

[vspinu]

If it where after me, ESS should be rewriten as well, so no, I python backend for ESS will probably never happen.

[wavexx]

Closing this for cleanup, since C-z actually works. There's already #13 for further refinement.