How to connect buffer with python code to an running jupyter notebook

[vale46]

Hello - in new versions of ein, I'm unable to connect python buffer to ein jupyter notebook buffer. In previous versions, this was implemented in ein-connect.el through functions like ein:connect-to-notebook-buffer. But I noticed that ein-connect.el functionality is no longer included in new versions of ein. I found this feature to be incredibly helpful in development/debugging during my projects.

Can I please understand if this functionality has been dropped, since the ein package has been updated frequently over the past few weeks but I've been unable to find helpful docs/overview about these updates.

While really appreciating the effort put in by the ein development team, any insight on the future of this project would be great!

Thank You

[dickmao]

incredibly helpful in development/debugging

Jupyter already has enough moving parts that admitting and cleaning up after a "sidecar" buffer I felt was untidy, fragile, and clumsy. Standalone, non-notebook work is best done without EIN, that is, the various send-region functions in elpy https://elpy.readthedocs.io/en/latest/ide.html#evaluating-code-fragments more squarely hit the mark.

I would need convincing via example to bring back this functionality.

[vale46]

The biggest one would be the ability to display graphs/charts inside Emacs. For my use case, this is important and was my primary motivation to use Jupyter notebook/EIN when coding in Python, to stay close to Emacs/ESS framework which I used heavily programming in R previously.

[dickmao]

The biggest one would be the ability to display graphs/charts inside Emacs.

That's an argument for the interleaved notebook format, not ein:connect-to-notebook-buffer.

Emacs/ESS framework which I used heavily programming in R previously.

Incidentally, ESS and Rmarkdown, which I feel form a generally superior workflow to jupyter, displayed graphs outside emacs, which EIN now does by default. Elpy+Ipython is closer to the ESS experience than EIN.

[millejoh]

@dickmao, I disagree. Strongly. It is not your job to dictate a user's workflow, and removing and then refusing to reintroduce a requested feature because you don't like to code is not compelling in the least.

@vale46 - The version of ein on melpa stable still has the notebook connect functionality. You can also go to the releases and download version 0.16.2.3 from there. I have essentially negative time to work on this project at the moment, but if I don't see any changes in the next couple days I will try to reintroduce this functionality myself.

[dickmao]

My job? None of this is my job.

[vale46]

@millejoh thanks for your thoughts and looking into this despite your workload.

Despite these misunderstandings that come up in collaborative projects such as these, greatly appreciate the work each contributor has put in.

[millejoh]

@vale46 - I just merged in some commits which brings back some (but not all) of the buffer-to-notebook connect functionality. Just be sure to put (require 'ein-connect) after wherever you put your code for loading ein. Tooltips and kernel completion are no longer supported as those features were deprecated in favor of native support via polymode.

I did like having completion candidates come from the kernel so that will probably come back some day too, though most likely as a separate project/package.

Feel free to post any issue you run into along the way.

[dickmao]

@vale46 As it stands the ein-connect behavior is effectively an easter egg known only to the witnesses of this thread.

In the interest of tidiness, can you please disabuse the package owner that this functionality is preferred to a simple elpy setup? Alternatively, please furnish a compelling example how "connect" is useful. Thanks.

[vale46]

@millejoh Many thanks for taking valuable time (over weekends!) to restore "connect" functionality!

As a quick heads up, upon installing latest available EIN package from melpa (20200301.802), the initial connection to python code buffer to a Jupyter notebook session works as before. But when trying to execute code in that buffer (using C-c C-r ) returns the error: Wrong type argument: plistp, (nil)....will try and post debug/error log as soon as I find time.

[dickmao]

Lest someone can show d29716b isn't just dead code, I plan on removing it. As near as I can tell, it doesn't work for vale46 (or presumably anyone else).

[vale981]

I usually write some utility functions outside ipython and import them into the notebook. It is very nice to be able to update that file and load it into the notebook on the fly...

[dickmao]

Hmmm... unlike most of this thread, your response hits the mark. Thanks.

[vale981]

Unfortunately the functionality is broken right now. I am short on time, but may be able to try to hunt the bug down...

[dickmao]

Looking at ein-connect.el again, I continue to find the 8-year-old logic there distasteful. By invoking the ein:connect-to-notebook command, it saddles a normal python buffer with an out-of-place buffer-local global and an intrusive minor mode and keymap. I would really like to keep normal python buffers away from EIN's predations. ein-connect.el also defines a bunch of undocumented and untested functions that I wouldn't want users to stumble upon as they'd silently fail at best, and confuse EIN at worst.

For your case, isn't the straightforward solution to premise the notebook with a %run myutils.py call? If literate programming is jupyter's goal, surreptitiously importing a bunch of code via the ein:connect mechanism would seem to defeat that.

[vale981]

This is true, and I've transitioned to org babel, so this is not as relevant for me anymore. Although it is a pain to go back to the code block and re-execute it all the time.

[millejoh]

@vale981 - not sure if you are still interested, but if you are in the mood you can take a look at https://github.com/millejoh/ein-kernel-utils.

Among a couple other things, it brings back support for connect code buffers to a notebook. I've done some testing and it seems to work and I think it fixes the error you were seeing earlier. Work progresses glacially, but I hope to implement the capability to do code completion and execution in org edit buffers that use ob-ein.

Being experimental, it is unfortunately not on melpa so you will have to clone the repository yourself. Check the README for details.

[vale981]

Thank you very much. I will try it out.

[dickmao]

Not for nothing, but it would help to know whether vale46 and vale981 are the same person. I suspect two "vale"'s interested in the same obscure functionality in a relatively obscure emacs package is too much of a coincidence for them to be separate people.

I am sympathetic to someone wanting to quickly apply a change for a myutils.py type of file. In emacs, I am constantly pressing C-x C-e to do something similar. Such "sidecar" evaluation however merely reinforces the growing awareness that jupyter notebooks encourage poor software practices, so your move to org-babel is the correct one. EIN will continue to entertain the college kids.

In commit 6fc75f2 I implemented C-c C-/ e and C-c C-/ r to evaluate a statement or region respectively, and updated the README.

[vale981]

I can assure you that I am a separate person from vale49.

I usually go by hiro98. vale981 used to be my main Nick because my name is Valentin and I was born in 1998. It is rather impossible to change my Nick on GitHub. If I were to create a new account, it would be under the name hiro98. Check the website linked under my account ;)

[dickmao]

Thanks for your valuable feedback.