General feedback about improvements to Interactive Window Experience and Workflow

[taoluo]

There is a web-based ipython console in the jupyterlab project, basically a QTConsole counterpart in browser. It supports inline display of matplotlib plot and good pandas DataFrame rendering which is extremely important for interactive data processing related tasks. hope to see that console embeded in VS Code. a related issue may be (https://github.com/DonJayamanne/vscodeJupyter/issues/19)

[jrwrigh]

I've already said my support in the referenced feature request, but I figured I'd throw my hat here as well. The IPython console offers a really great interface for interacting with scripts and data while also working really well with a REPL based workflows.

[KaiLicht]

A must have for data science work with VS Code! That would be a great feature.

[ftpronk]

It would also be very nice if that IPython console could be coupled to the debugger. It's a feature that makes for very nice debugging in pudb and in spyder-ide.

[rathsandeep]

Holding on to spyder for only this. If this is implemented, I will switch to vscode. Spyder lacks some of other features a good code editor should have. I end up switching between vscode and spyder when I am doing python+something else

[mcdevitts]

100% agree on this. I would also be able to switch over if ipython was fully supported. It's absolutely required for a data science workflow.

[chenwang]

Hydrogen on Atom offers a really good experience for data science and ipython. I do hope that something similar could come to VScode.

[smortaz]

don's existing jupyter plug-in for vscode kinda sorta does the same, but it's not complete. we'll get there.

On Tue, Feb 13, 2018 at 11:49 AM, Chen Wang notifications@github.com wrote:

Hydrogen https://nteract.io/atom on Atom offers a really good experience for data science and ipython. I do hope that something similar could come to VScode.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/vscode-python/issues/727#issuecomment-365382824, or mute the thread https://github.com/notifications/unsubscribe-auth/AAtUVAdxlv4zVz0Ltk9DvxmgTPEM2pn_ks5tUec4gaJpZM4R9ydW .

[jrwrigh]

@chenwang I didn't know that existed. Having Jupyter outputs appear inline rather than have it appear in a seperate window is fantastic. If we could have something like that on VSCode it'd be great.

[GPhilo]

The original request came on Nov 3, 2016. I really hope it will be implemented soon, VS Code is fantastic under many aspects, but not having an iPython console is really a turn down for me. I've been trying out many IDEs, but none so far can replace Spyder because of how addicted I got to "run piece of code, check in the console a few values, run the next line, check some other values, repeat". Yes, technically that is already doable, but the ease of having a console with history, completion, magic functions, etc. just doesn't compare to what is now the Jupyter extension. Fingers crossed!

[mcdevitts]

Completely agree with @GPhilo. Often people mistake wanting ipython with wanting jupyter.

[jrwrigh]

@mcdevitts Are you sure that we don't mean IPython? We're referring to the interactive console that is still (to my knowledge) called IPython. We're not referring to the Jupyter QtConsole which is a fancier GUI frontend over the IPython console.

[ftpronk]

@smortaz Important priorities for sure. Maybe it would make sense, while working on improving debugging in VS Code, to already take on board the idea of (optionally) using IPython as an integral part of the debugging process, as opposed to it just being an idea which is strapped-on later in the development process? (Just stating the obvious in case it might have been overlooked. I'm not questioning the VS Code planning in any way..)

@u2berggeist I guess they are referring to the ipython functionality common to the interactive console and the QtConsole, as opposed to the jupyter cell evaluation which is currently available in VSCode.

[smortaz]

@ftpronk - ironically jupyter/ipython does not have any debugging support. one of our devs is working w their team to add debugging to jupyter - stay tuned. hopefully that'll land and enable better vsc/jupyter integration. if interested in following the progress, see: https://github.com/jupyter/debugger/issues/9

[ftpronk]

@smortaz - Very interesting! I was always a bit surprised that it wasn't used more as debugging tool, but that explains it I guess. So such a development will also end up making jupyter/ipython better tools too. Exciting stuff! Thank you for the link, I'll definitely stay tuned. Keep up the great work!

[mcdevitts]

@u2berggeist I do mean IPython... as in it would be great to have an IPython console running inside vscode like @GPhilo suggested. I wasn't referring to Jupyter's QtConsole.

What I meant was that plenty of tools these days are ignoring IPython integration and opting instead for Jupyter Notebook. Jupyter Notebooks are great, but they require a completely different work flow.

[brettcannon]

Just to help incentivize people to leave 👍 vote on the top comment, if you look at the issues with the most 👍 votes you will see this is number 6, but number for has nearly 50 votes. So while I appreciate the comments asking for this, it's those votes that really matter when we prioritize our work (and what people say in the survey that pops up for people who are randomly selected to fill one out).

[jflam]

@GPhilo and @mcdevitts would you be interested in reviewing a proposal I have for this feature? If so, please hit contact me at jflam at {the company that makes VS Code}. Thx!

[janat08]

will this fix where autocomplete works line-by-line basis not recognizing blocks of code?

[GPhilo]

@jflam Are there any updates on this issue?

[jflam]

Yes - I had some students at ICL create a prototype of this concept, and it's available on the VS Code marketplace today:

https://marketplace.visualstudio.com/items?itemName=neuron.neuron-IPE

We're definitely going to build something like this.

[brettcannon]

We have started our first cut of this experience in microsoft/vscode-python#3034.

[JackCaster]

We have started our first cut of this experience in microsoft/vscode-python#3034.

Is microsoft/vscode-python#3034 somehow related to Neuron? I read a recent post on the Microsoft blog that says that

Microsoft has decided to integrate the work that we did in neuron into the Python extension for Visual Studio Code

[brettcannon]

@JackCaster yes

[ronglums]

Please check out the latest Python extension that enables an interactive experience to work with data: https://blogs.msdn.microsoft.com/pythonengineering/2018/11/08/data-science-with-python-in-visual-studio-code/

[bitagoras]

I am a little bit unhappy with the current solution. In my optinion the way on how we interactively work with Python should go in a slightly other direction. I programmed many years ago a hybrid of a simple editor and a console. Up to now, I could not find any comparable in the internet, so I recently wrote it again as an extension for Notepad++: https://github.com/bitagoras/PyPadPlusPlus You can have a look at the animation to see the idea behind this concept. What is special in PyPadPlusPlus compared to other interactive Python environments:

• The editor acts as the console input and you can execute single lines of code
• There is only one pure output console
• This means, there is no mixture of code and output and there is no redundant copy of the code which wasts some of the available area of your window and gives no additional information.
• The code blocks are optional. You can always execute single lines or any selected part of the code. With Ctrl+Enter you can step through your code and run it line-by-line. When you start at a #%% comment, you go through your code block-by-block. It depends on where you start. This is totaly flexible and intuitive.
• you can execute single lines or selections with the mouse middle button. With only one hand you can choose and run any part of the code within one second.
• Auto completion and code inspection is based on runtime information of the object.
• A matplotlib event handler is included, so you can plot many interactive windows from the same script.
• There is no Jupyter kernel required, so the startup is very fast.

I wrote a first simple version of this editor in 2004 and since then I used it continuously in science and industry until now. I controlled complex experimental setups in a physics lab with it. There I used Python code-lines like clickable buttons to control the experimental setup: a motor, the superconducting magnet coils, a cryostat and many other devices. The control was easy to use like a GUI and flexible like a text file. The program could grow while I was using it. This control produced great scientiffic results. The same workflow was also very useful for data analyses, numerical calculations and algorithm development.

I also use sometimes Jupyter, when I want to demonstrate mathematical concepts to others. But most of the time I need pure Python code, where my editor concept gives me the best flexibility. I realy would like to make this available in VS Code. In the Hydrogen plugin of Atom is something that goes in a similar direction. Could this maybe also fit to the concept of vscode-python?

[yzhang-gh]

I have some similar experiences as @bitagoras shared. In short, the good things are

1. Ability to execute the selected code (or the current line if no seletion) (no matter there is a #%% or not)
2. The "pure output console". IMO, the current output cells with the corresponding codes are probably useful, but most of the time they are not but distractive... I have opened a related issue microsoft/vscode-python#3266 (fold the code in the output cell by default)

These two features are already there with the former Jupyter extension (It has a command executeSelectedCode (don't remember the exact name) for feature 1)

But I have to use my own fork since that is unmaintained now.

---

As for

3. Auto completion and code inspection is based on runtime information of the object.
4. A matplotlib event handler is included, so you can plot many interactive windows from the same script.
5. There is no Jupyter kernel required, so the startup is very fast.

they look attractive

[bitagoras]

A folded cell by defaul would be a good improvement. I would go even further and would also hide the first line of code when folded. I am pretty sure that nobody would unfold it ever. Since you cannot modify this code when you want to restart the cell, you have to search the corresponding line in the editor anway.

The currenty concept tries to mimic Jupyter notebook, but it isn't it Jupyter. I don't expect that VS Code will become a replacement for Jupyter. This would require to include all kind of web techniques to make the interactive plots etc. available. PyCharm tried to do this for some time but it was never good enough, so that I used it. And currently the startup time of VS Code to print a simple "hello world" with Python takes 30 seconds at my computer. So, I would apreciate if I this plugin would focus on the properties where Jupyter is not made for, e.g. single line / selection execution, simplicity, efficiency (e.g. best use of the available window space) and fast startup.

[syagev]

I think the extension is still not addressing one of the most critical original issues - specifically, the ability to type code directly into REPL. Since we can't type in the "Results" tab of the extension we are "forced" to edit code in some "dummy" notebook file whenever we want to try out a few pieces of code. This isn't the most convenient way of working..

Think about this simple use case: I'm working on an existing complex plain python module, and while I'm doing that I need to quickly try out some code without launching the entire project. I can either (1) type it in a terminal, but without code completion, or (2) switch tabs to a "dummy" notebook and type it there - this means that on a decent monitor I can't have both the code file I'm working on and a decent REPL open in the same time.

It might sound picky, but I really think it's a crucial part of doing productive python data science development.

I've posted a related SO question (since I wasn't sure this feature existes and maybe I just missed it): https://stackoverflow.com/questions/53327894/interactive-python-console-with-autocomplete-in-vs-code/53328072#53328072

[GPhilo]

I agree. The feature request was for a "QTConsole-like experience" and we got notebooks 2.0.

[jflam]

Thanks for the feedback. The current feature is a point-in-time snapshot of where we are right now, and was chosen for engineering expediency. We are definitely thinking about the right way to provide folks with both options as well. Stay tuned!

[jrwrigh]

@jflam So if the issue has not been fully resolved (as I think we've all agreed on), should it not be re-opened?

[jflam]

@ronglums - can you please re-open and drive this? Thx!

[jamescasbon]

You can get a repl into the current kernel by running "jupyter console --existing"

This gives you access to variables defined from the notebook cells.

One possibility is to use the %connect_info stuff to automate creating a repl connected to the current notebook.

Plotting doesn't work with this setup, but it is possible to inspect and otherwise poke objects.

[mcdevitts]

Being able to automatically open an ipython shell would be fantastic.

To tag onto @GPhilo's comment, it would be even better if we could just execute the current file in a new ipython shell that stays open AFTER execution for further interaction. In other words, this would be dependent upon the ipython shell only, not Juptyer. Being able to plot from that console (these plots don't have to be inline in the terminal) would be key.

[stinos]

^ agreed; Jupyter is all nice, but not everything has to be Jupyter when a plain shell works as well, and often faster. For my workflow which isn't uncommon I think, the behaviour e.g. Spyder has would be best. And ideally improve that (e.g. Spyder has still annoying quirks like Ctrl-C not clearing REPL input, F9 only working in editor not for text from REPL output, ...). So yes that's: open an IPython terminal, run code and/or entire files in it and leave terminal open, maybe even with variables shown in watch window. And preferrably exactly the same behaviour when running under the debugger. Thinking of it, even though I dislike Matlab both as a language and IDE, it does have the REPL part nailed almost perfectly.

[ronglums]

Great discussion here! Reopening the issue and we will look into this.

[bitagoras]

I would like you to reconsider whether a shell is really the best way of interaction with Python. The main drawback for shells is in my opinion the very inconvenient way of repeating your previoulsly used commands. Imagine your commands have several lines of code. When you're lucky it consists of one indented block which you can scroll back, modify something and repeat it. But if several single lines belong together, e.g if you plot some data, it becomes very annoying. You have to scroll back to every single line, perhaps modify it, and run it in the right order. In my workflow I like to write some code in the editor, run in partially, then I want to plot some intermediate data or write some output to test my program. Then I either discard the plots and tests or I keep it for later usage or maybe make it part of the program or make a separate program out of it. But in the shell they are lost and I have to retype everything into the editor or fishing all the single command lines from the shell. How can this be a desired solution? In the solution I demonstrated with PyPadPlusPlus you have all the advantanges of a shell but none of the described disadvantages. It does not cover all the advantages of Jupyter for presenting results with formated text, rendered equations etc. But it is much more versatile compared to a shell. And Hydrogen, which has a similar mode than PyPadPlusPlus with a separate output area, demonstrates how you can have many properties of the two concepts combined into one plugin.

[GPhilo]

@bitagoras I'm not sure I understand your comment. No one here is saying "throw out the editor", the desired coding experience if very different from what you describe. You have your code, possibly with defined sections similarly to what the current data science extension allows, but these sections run in an interactive iPython console. That way, if you need to quickly run some one-liner, or if you want to inspect the value of some variable/expression, you just type it in the console and get the output.

Of course, if you have a major block of code, you'll probably want to have that in the editor (possibly in a different file). Again, the best case scenario would be "select the code, press Shift+Enter and the code runs in the interactive console". How is this inconvenient? To get a better idea of what I'm talking about here, look up how Spyder's console works. That's essentially what I personally would love to see implemented in vscode.

[bitagoras]

Don't get me wrong. I am just talking about the interactive addon to the editor.

Sure, "select the code, press Shift+Enter and the code runs in the interactive console" is basically it. But there are several things I don't like: The selection is annoying. You have to select any single line before you can execute it. If you did not select the whole line properly, you get an error. You also mess up your output with a copy of your commands and some ">>>" and "...". Why do you need the same commands twice on your screen? And why do you want to enter commands "in" the shell? Your editor should be where you type your commands (the "human" side) and the output is only the output (the "machine" side). I don't see the advantage of this mixture. You can open a new scratch editor if you do not like to enter some commands in your main program. Having a "pure input" area and a "pure output" area seems to me more cleaned up and you never have to switch your cursor. And your output will more look like the output of your finalized Python script when it runs as a whole.

The idea behind the shell was to have an interview-like dialog between the user and the interpreter. This perfectly fits into a system shell like a linux shell. But in a modern editor and IDE such a shell has no right being here any more im my opinion. There I expect something more optimized for this environment.

[ronglums]

We can start with an iPython console area in the Interactive window that is editable and results will be rendered in the Interactive window.

[mcdevitts]

@bitagoras Your points miss a major use-case many of us are arguing for: introspection and manipulation of objects after the code has been run. In the case of ndarrays, why should I re-run my entire code or a specific "cell" when all I want is to examine the shape of the array, and play with various permutations of the data to make it work.

Finally, this ability is even built into the python interpreter with the -i command. All we're asking for is an ipython shell be opened in the same way. Heck, I'd be okay if this was selectable option. I completely disagree with your comment that a shell has no place in a modern editor or IDE. Both Python Tools in Visual Studio and PyCharm have integrated shells for debugging / introspection / manipulation after the script has been run. I would even point out that interactive consoles are available in JupyterLab and for very similar reasons like we've been mentioning:

Link a code console to a notebook kernel to explore code interactively without cluttering up the notebook with temporary scratch work.

[bitagoras]

This use-case is included in what I described. After you have runned the whole code (Ctrl+A and Shift+Enter), you can inspect the variables and do some post processing, testing or plotting. Your program never ends until you shut down the kernel.

I know from PyCharm your use-case with the shell. PyCharm is great, you can debug and stop at a break point and then use the shell to investigate the current state of your variables. You also can open a scratch editor and write some analysis or plotting functions and run selected parts in the shell to inspect the current local scope of your function. But this selecting an running code in the shell could be more smooth and without displaying a copy of your code in the shell, because it is a mess. That's basically what I say. I don't want to abolish the shell, but improve it.

I think it is a question of complexity of your data whether a normal shell satifies your needs. In my use-case it is often not sufficient to check some variable contents or find some bugs in the code. I have to find numerical inaccuracies in very complex data. This can be quite challenging and requires mathematical transforms and ploting of maps or interactive 3D animations to investigate the data (I currently develop code for a satellite based navigation system, where it's all about numerical accuracy). My interactive auxillary calculations are too complex to enter it in the shell. My need for an appropriate interactive environment was so great that I developed this year this new Notepad++ addon. It helps me now a lot and it is much better than my old solution and all other Python environment I tried (and I tried everything new the last 14 years). I think that many people who work with complex data or maybe machine learning algorithms have the same need. I just still miss a debugger, therefore I really hope that this could also be realized in VS Code, which is a great and modern editor.

[rchiodo]

It would be nice to summarize what people are asking for. I think these are?

1. iPython console that is interactable.
2. iPython console that doesn't go away or that the lifetime can be controlled
3. iPython console that starts up fast

Anything else?

[bitagoras]

I would add:

4. Execution of code from the editor with intelligent selection of line(s) and advancing cursor.
5. "Silent" execution without a copy of the executed code in the console.
6. Auto completion in the editor based on run-time information.

[bat67]

I think what we need is like the same way running sections or files in SpyderIDE or MATLAB.

In Spyder we can run files or run sections(seperated by #%%) in the console by pressing shift-enter in editor, and after the file or sections finish running, all the variables remain, so it’s possible for doing some subsequent processing in the console, which is interactable(maybe watching variables, or doing something that is not maturely considered so it shouldn’t be written in this file). What running in the console is just IPython, not jupyter-notebook(in VSCode currently, if doing the same thing it will start Jupyter Server which is slow).

And in MATLAB, it is the same as in Spyder, we can run sections seperated by %%, after running this section, we can do anything we want to these variables in the MATLAB command line window.

But in Spyder, running sections is just like copying the section, and pasting in spyder ipython console to run. This does not happen in MATLAB because it will run section silently and only print the output the program should print. This is somehow like what @bitagoras said in number 5, “Silent execution without a copy of the executed code in the console”.

Maybe in VSCode, we can open a terminal and type ipython to open ipython in bash. And print run xx.py to run this file. After running, variables remain. But this method cannot run sections, only if you copy and paste this sections in ipython console and run manually. This may be the same as Spyder in  some ways, but a little bit tricky.

[ftpronk]

The last thing I want to do when debugging python code is to clutter my files with run sections or any other modifier. It's not only cumbersome, but I find that it's simply, visually, standing in the way. To find bugs, you need to be able to read your code, and anything that impedes or impacts code readability is not an acceptable solution.

[nraw]

It's great to see that there's a discussion on this as I'm baffled to see that this kind of support is not available yet. I guess coming from languages such as R and Matlab I'm more used to the interactive way of coding when it comes to data science and while I guess Spyder comes very close to this, it just feels like a very basic tool of which the main benefits could be easily replicated in VSCode.

But honestly, I think I would already be fine by just having the option of "Send line/selection to Debug console" and "Send block to Debug console". Am I missing it and this already exists?

So far everything I've found was either sending it to the terminal and/or the Interactive python window, which just feels like an inferior Jupyter notebook which if I wanted I would just be using Jupyter in the first place.

[stinos]

Am I missing it and this already exists?

Yes: while in debug mode, select text, right-click, select Debug: Evaluate

But for the rest the Debug Console isn't the most exciting environment, especially not if you're used to a full R or Matlab or IPython REPL.

[rchiodo]

People seem to be asking for an iPython REPL. Do you have an example of an iPython REPL? Are you talking spyder? What are the parts missing from the VS Code Python Interactive window?

[mcdevitts]

@rchiodo Spyder, I think, has the best implementation of an integrated IPython REPL. You can choose to run code or your file in a new REPL which then stays open for tinkering, or you can run your file in an already open and existing IPython REPL. Both use cases have their advantages and disadvantages, but not being forced into one or the other is a huge plus.

A non-python example of an ever-present REPL is Matlab, and I think a lot of the scientific code community either started in Matlab or plays in Matlab enough to see the utility of an interactive interpreter.