Maintainership of python.el

[memeplex]

@galaunay @jorgenschaefer do you know who is currently maintaining python.el? Can you commit changes there?

I'm asking here because I've submitted some patches to the emacs tracker months ago that fix problems with inferior ipython and received little to no answer. Besides, there is no commit since about a year, except for a copyright update. Development seems pretty stalled and it's not like there are no bugs, that's starting to worry me. I even offered maintaining it but nobody answered.

I understand it's not elpy fault but since python.el is the foundation of elpy I'm asking you for some kind of advice about how to proceed. Maybe you can raise the issue in emacs-devel, your concerns will probably be taken into account as maintainers and developers of an important part of emacs support for python.

[galaunay]

I think you were looking at the emacs 26 branch. The master branch still show some activity on python.el (not much, I agree, in the past year).

The current MAINTAINERS file doesn't mention anybody being responsible for python.el. I would say the good thing to do is still to send bug reports, but being months without news is a bit odd....

[memeplex]

Ah ok, you're right, at least is not completely dead...

[memeplex]

I've been thinking about this situation. I believe the main problem is that python.el was subsumed by emacs. It's understandable for emacs to ship basic major modes for popular languages, but python.el is far from basic. Now, emacs is a large piece of software and contributing to it is not as easy as if it were hosted here: legal issues, a peculiar bug tracker, etc. (don't take me wrong, I contribute to emacs and it's not rocket science, it's just a harder proposition -vs github/gitlab- for the average potential contributor these days). More importantly, maintainers have their priorities and it's understandable that python is not always at the top of their list.

Nowadays ipython (+prompt_toolkit, +jedi) is offering a better cross-platform REPL experience than emacs. It's not difficult to improve python inferior mode to provide competitive features, I've even suggested simple advices to keep history colorized. But more basic things are broken, in particular ipython integration has been always difficult to keep working straight because of the faster release cycle and dynamism of ipython vs python.el. Some of these features are important to elpy as a python IDE and the decision to rely on python.el as included in the latest stable emacs release has its costs.

Here are some options I've thought of:

1. More aggressively contribute upstream. But for this to work someone with python as a priority has to gain access to python.el. Simply contributing patches is not an option: patches wait there for months and, even if action were immediately taken, you have to wait for the next emacs minor release or clone the emacs-2x or master branches and build it yourself. Now, I don't know what it takes to get access to python.el. A time ago, I offered to maintain it but got no answer.

2. Start a new project here or in gitlab that provides regular updates to python.el latest revision in emacs/master. This isn't intended to be bleeding edge development, I don't want it to become a kind of neopython.el, I would like improvements and fixes to be merged back to python.el eventually. This project could be distributed through melpa and could be an optional dependence of elpy. Other projects that are distributed with emacs follow this approach (for example, org mode).

3. As in 2 but include python.el as part of elpy.

4. Include some advices and overrides in elpy instead of properly patching python.el. This is a far more limited and hackish approach but, given the modular nature of many emacs APIs and frameworks (xref, imenu, eldoc, etc.) there is still some room to provide clean replacements in a piecemeal fashion.

[galaunay]

I do agree that contributing to python.el have been made more difficult and slow by its assimilation in emacs. But to be honest, I think it works quite well with bare Python. From what you are saying, maybe the problem is more the integration of IPython (genuine question, I never really used IPython in Emacs).

Have you considered a kind of ipython.el project, that would add specific support for IPython features ? This would allow to match the release cycle of IPython with the possibility to push the fixes and improvements to python.el later on. (I realize this is mainly a description of your option 2, but I think a project focused on "IPython support" would be better accepted by the community than a project focused on "fixing python.el").

[memeplex]

Have you considered a kind of ipython.el project

I any case I would consider something more like "python REPL" project.

But there are other things besides the REPL. Say, for example, proper f-string colorizing. I don't see a feature request like that gaining traction nowadays, we need a new Fabian Gallina.

Also, related to the lsp initiative, maybe in a not-so-distant future elpy would better focus on deficiencies in core python support than on ide-like features.

[memeplex]

I see a request for f-string font locking exists since almost one year (http://lists.gnu.org/archive/html/bug-gnu-emacs/2018-03/msg00236.html). Not even answered. Something will have to be done in order not to stay behind the times, python.el stagnates, do you agree with that? ipython mode is too narrow a scope for the task at hand.

[memeplex]

than a project focused on "fixing python.el"

I see your point, but we don't have to sell it this way. Given that there is no clear maintainer at this moment, it could be seen as the "developing version" of python.el (instead of the "patched version") with shorter release cycles, simpler contributing process and some cool new features. This is the model for developing org-mode, as I mentioned before.

[galaunay]

I any case I would consider something more like "python REPL" project. ipython mode is too narrow a scope for the task at hand.

I see your point.

Maybe a list of problems to address and functionalities to add would be helpful at this point to have a better idea of the quantity of work to be done. I guess some ideas can already be scrapped from Emacs bug tracker.

I still think we should keep it separated from Elpy, as non-elpy users may be interested as well. But I would be happy to help and to integrate this in Elpy.

[memeplex]

I've been thinking about this.

One requirement I keep in high regard is that any change (fix or feature) should be easily ported to emacs core. The idea is to keep an ongoing development, with already tested changes (probably indirectly through elpy), as an additional argument to convince maintainers to accept patches that would otherwise stagnate. The reasons for their reluctance may be good ones, I don't think any of the currently active maintainers is a python expert so conservatism may well be a reasonable posture for them. Besides having written some patches and minor stuff, I don't really have any credential as an emacs developer that justifies blindingly trusting in my patches. But verifiably working code is another matter.

Now, how to reconcile the objectives of currently working and easily mergeable. On the one hand, easily mergeable more or less implies keeping an entire emacs fork (I could just copy python.el and do a lot of rebasing in order to keep an up-to-date patch queue, but that seems like a lot of work). On the other hand, you can't expect people to reinstall a patched version of emacs only for this.

Here is what I propose. I already keep an emacs fork, I could simply add a github remote to it. Now, in a python or whatever branch I could keep all python.el related changes. My issue tracker could be used or I could check elpy issue tracker for stuff tagged python or whatever. Every once in a while you can check changes in my python branch and, in case you see fit, copy my patched python.el as part of elpy. This dependency could be made optional with a customization option in elpy (say, toggle python.el experimental features). With enough feedback I could then open issues in emacs tracker linking to the patches in my repo. If things go well I may eventually gain the trust of the maintainers and have direct access to python.el or, at least, more readily accepted patches.

What do you think?

[memeplex]

I changed my mind regarding the implementation. I think you will agree that what follows is a better idea. I'm going to keep a python exclusive project (not an emacs patch) and publish it to melpa. This project will be (re)based on a continuously updated copy of core python.el, so as to simplify backporting to emacs, and will strive for 100% compatibility with elpy as an optional dependency. So you can mostly forget this option exists except that you want to integrate elpy with some extended functionality not available in core python.el.

Here is a tentative list of features/fixes:

1. Thread-based completion using a client/server model to avoid prompt number madness (jumping from 123 to 567) and dangerous blocking. This will improve the completion fallback to shell in elpy.

2. Better integration with ipython jedi completion by passing more complex expressions as prefixes.

3. Fix integration with ipython (my current patches)

4. Colorized command history (not only the current prompt, requires small comint advice).

5. Support f-string syntax and other unsupported modern syntax (?)

6. Maybe support cython syntax

I'm currently facing a lot of changes in my life so I don't think I will be working on this until a couple of months, minimum. But I'm determined to do it. Still have to think a name, I don't want to get too crazy with that because I want to emphasize continuity, not disruption, but python.el is already taken ;)

[memeplex]

What do you think of pyel? I think it signalls both continuity with python.el and alignment with elpy. Besides I like short names.

[galaunay]

I'm going to keep a python exclusive project (not an emacs patch) and publish it to melpa. This project will be (re)based on a continuously updated copy of core python.el, so as to simplify backporting to emacs,

It sounds like a very good solution. It would allow to provide a more dynamic alternative for the community and still enhance python.el on the long term. Having a user base would also help to show the code maturity to the emacs developers.

What do you think of pyel?

I was thinking about python-x.el (python extra ? to mimic dired-x ?). But I am afraid it will label it as a python.el extension more than as a replacement... I find pyel very nice.

I'm currently facing a lot of changes in my life so I don't think I will be working on this until a couple of months, minimum.

Please keep me updated. I suspect there is some pieces of code in Elpy that would rather belongs in python.el. Maybe your pyel would be a good place for them ?

[memeplex]

Thank for you input @galaunay. I also thought about something like python-x.el. And came up with pyx.el which somehow combines py.el with the -x idea. It's true that it suggests "extended" but it is catchier.

[npostavs]

you have to wait for the next emacs minor release or clone the emacs-2x or master branches and build it yourself

Actually, Emacs' master branch version of python.el is also released via GNU ELPA, so whenever the Version: header is updated there is a new release available there. I don't know if this affects any of your plans about this; if you think testing out modifications/extensions to python.el in a separate repo will make things easier for you, then go for it.

1. Now, I don't know what it takes to get access to python.el. A time ago, I offered to maintain it but got no answer.

Part of the problem there is that you buried that in a bug report, it's better to send that kind of thing to emacs-devel (though I would have responded to it earlier had I not been dealing with my own Real Life Stuff; I was pretty much offline November through to March). I can tell you that step 0 for getting write access to emacs.git is signing up for an account on https://savannah.gnu.org/.

[memeplex]

Thanks for the tips, @npostavs. I've done the copyright assignment paperwork in the past, I'm not sure whether anything has to be renewed or not after a few years have passed. I will create the savannah account, didn't know about that. Despite the practical advice about contributing to emacs, do you think it is a good idea to keep a smaller project outside emacs itself and periodically update core python.el? My idea is to keep everything backwards compatible. But I would be more confident about changes contributed and, specially, tested by users of elpy (so optional indirect users of this outside project) and I believe this is easily done via github/lab (I personally have no problem with debbugs but I can't speak for the occasional contributor). That said, if contributions are substantial there still is the copyright issue.

[npostavs]

I've done the copyright assignment paperwork in the past, I'm not sure whether anything has to be renewed or not after a few years have passed.

No, as far as I know, the only time that might need to be updated is when you change employers (if your new employer might have some claim on code you write).

do you think it is a good idea to keep a smaller project outside emacs itself and periodically update core python.el?

Not sure, to me it sounds like it would add more hassle keeping things synced. On the other hand, if you really can get more testers that is a pretty significant advantage.

That said, if contributions are substantial there still is the copyright issue.

Yes, that's true.

[sten0]

@npostavs thanks for bringing up GNU ELPA :-) @memeplex and @galaunay would it be reasonable for Elpy to depend on a version greater than some value for some particular feature and to expect users to also enable GNU ELPA, or would we need to ask MELPA to mirror ELPA's python.el package? As far as I can tell, this is specifically the kind of case package.el is intended to solve!