Make evaluation commands print output

[vspinu]

Hi. I love this new project. Thanks!

Currently all evaluation commands don't show output. As an ESS user, that's extremely annoying. I don't even know if the command finished or not.

Looking at the difference between standard evaluation commands and python-shell-print-region-or-symbol I see it's all about python-shell-buffer-substring wrapper. I haven't investigated this further. Is there a way to make python print last value automatically?

[wavexx]

On 04/02/16 16:06, Vitalie Spinu wrote:

Hi I love this new project Thanks

Currently all evaluation commands don't show output As an ESS user, that's extremely annoying I don't even know if the command finished or not

That's probably the biggest difference with ESS. I'm not against it, but I'd like to make this customizable, as I generally prefer not to see the evaluation I/O all the time (it just litters the buffer to no end).

You'd be surprised I often work with the python buffer buried. Since I display the data using gtabview (https://github.com/wavexx/gtabview) and matplotlib, I hardly need to see the output, confident that errors will pop in my face.

As a general rule, no output/window => no error. If you notice, the prompt is re-shown when the evaluation completes, although I completely agree it's hard and annoying to see.

I'd like to have something in the modeline instead (maybe just a status: Ready|Running). If you have some examples here in how other packages do it, I'd be eager to look.

Looking at the difference between standard evaluation commands and |python-shell-print-region-or-symbol| I see it's all about |python-shell-buffer-substring| wrapper I haven't investigated this further Is there a way to make python print last value automatically?

I use a different method compared to ESS. For a single statement, the output is printed. For blocks, it's not.

When I explicitly want the result or evaluate the thing at point, I use "C-c p p" (for python-print), which displays the output buffer as a result too.

For larger blocks, I use expand-region extensively.

I mainly use code sections, or single statement evaluation only. With ESS there's this tendency to crappify the code in order to keep paragraphs as a minimal workable unit, and I went to great lengths to avoid that (I'd love to backport sections to ESS whenever I'll have time).

It's a much more selective approach compared to ESS, but it has the advantage that I rarely, if ever, need to scroll the python buffer. It contains only the output I need (and exceptions). It's an hybrid between a notebook and an ESS session.

Getting the output for the last statement should be customizable nonetheless, but I encourage you to think differently™.

[vspinu]

On Thu, Feb 04 2016 08:24, wavexx wrote:

I generally prefer not to see the evaluation I/O all the time (it just litters the buffer to no end).

I see. Similar workflow is implemented in Cider and I couldn't quite get used to it. IMO inferior buffer is garbage anyways and it's convenient to have all the ouput in one place (resutls, errors and other feedback).

You'd be surprised I often work with the python buffer buried.

Me to, when it's about plots. In ESS you can also work with closed inferior and it will popup only on errors.

The problem comes to textual output. I am constantly inspecting objects and not seing output is aching.

Since I display the data using gtabview (https://github.com/wavexx/gtabview) and matplotlib,

Is there a way to hook pprint and tabvew automatically on the returned objects?

I am learning python because my new team is in python, and I find myself permanently going to terminal and typing pprint(xxx).

I use a different method compared to ESS. For a single statement, the output is printed. For blocks, it's not.

I haven't noticed this invariant so far. It makes sense. I will try to adopt it for a while.

So it's not a limitation of python interpreter. You can get the output for blocks, right? Would it be possible to make that optional without loosing correct stacktrace?

For larger blocks, I use expand-region extensively.

I tend to use regions with python much more than with ESS and it doesn't quite feel right.

I mainly use code sections, or single statement evaluation only. With ESS there's this tendency to crappify the code in order to keep paragraphs as a minimal workable unit,

That's surely true. But I am personally not bothered by this. I am using paragraphs as navigation units, so using them as evaluation units feels natural.

and I went to great lengths to avoid that (I'd love to backport sections to ESS whenever I'll have time).

Would be great! There are definitely good use cases for it.

It's a much more selective approach compared to ESS, but it has the advantage that I rarely, if ever, need to scroll the python buffer. It contains only the output I need (and exceptions). It's an hybrid between a notebook and an ESS session.

While I have no experience with folds I find it hard to believe it's more efficient approach for exploratory code. In my experience when you play with the code it becomes a big mess all around. You experiment with this and that till you fill up your file with crap. Maintaining folds seems like huge burden because it involves extra markup.

but I encourage you to think differently™.

In ESS I only use 3 commands without thinking: C-M-x which evaluates region or function or block and doesn't move the cursor, C-c C-c - same but steps, and ess-eval-region-or-line-and-step. That's it!

With python there seem to be many more incidental conditions (block vs statement vs lines, folds, sections, defns, paragraphs, print/no-print). As a result there seem to be a proliferation of commands and contexts where you can (or should) apply each. This feels much more contextual as compared to ESS.

BTW, I think it's unfortunate that C-c C-c and C-M-x do almost the same thing. IMO C-c C-c stepping in ESS is a better UI. But I guess you are really constrained by folds evaluation here. I think dwim would be better on C-M-x because no-step is consistent across all modes.

[wavexx]

Is there a way to hook pprint and tabvew automatically on the returned objects?

Tell no one I've told you this ;)

In emacs I also use:

(with-eval-after-load 'python
  (require 'python-x)
  (python-x-setup)
  (define-key python-mode-map (kbd "C-c p v") 'python-pylab-view-region-or-symbol)
  (define-key python-mode-map (kbd "C-c p l") 'python-pylab-view-locals))

(defun python-pylab-view-region-or-symbol ()
  (interactive)
  (let* ((substring (if (use-region-p)
                        (buffer-substring-no-properties (region-beginning) (region-end))
                        (symbol-name (symbol-at-point))))
         (stm (python-string-to-statement substring)))
    (python-shell-send-string (concat "__import__(\"pylab_local\").tv(" stm ")"))))

(defun python-pylab-view-locals ()
  (interactive)
  (python-shell-send-string (concat "__import__(\"pylab_local\").tv_locals()")))

Chunks of my pylab_local.py (which is in my python path):

if os.environ.get('TERM', 'dumb') == 'dumb' and 'DISPLAY' in os.environ:
    import gtabview
    tv = gtabview.view
else:
    import tabview
    tv = tabview.view

def tv_locals(private=False, modules=False, functions=False, max_width=128):
    import types, inspect
    data = []
    for k, v in inspect.currentframe().f_back.f_locals.iteritems():
        if v.__class__.__name__ == '_Feature': continue
        if not private and k[0] == '_': continue
        if not modules and type(v) == types.ModuleType: continue
        if not functions and type(v) == types.FunctionType: continue
        value = v
        if isinstance(value, pd.DataFrame):
            value = list(value.columns)
        value = repr(str(value))[1:-1]
        if len(value) > max_width:
            value = value[:max_width - 1] + '…'
        data.append([k, v.__class__.__name__, value])
    data = pd.DataFrame(data, columns=['NAME', 'TYPE', 'VALUE'])
    data.set_index(data['NAME'], inplace=True)
    data.drop('NAME', axis=1, inplace=True)
    tv(data, title='bound local variables')

Then, C-c p v (print visually) and then C-c p l (for print locals). They'll work for both the current symbol at point and for a selected statement. They'll even work in a terminal window if there's no X.

While I have no experience with folds I find it hard to believe it's more efficient approach for exploratory code. In my experience when you play with the code it becomes a big mess all around. You experiment with this and that till you fill up your file with crap. Maintaining folds seems like huge burden because it involves extra markup.

I wanted to re-use folding-mode, but I never used folds. I just use "sections", which look something like this:

#!/usr/bin/env python3
from pylab_local import *
from matplotlib import dates as mdates

#--- do something
bla bla

#--- do something else
bla bla

#--- and more
....

using C-c C-c will execute the current block (or the current selection if I want to narrow down things). My sections are typically ~5-10 lines long. The section title is also shown when evaluating, which is a nice touch.

Of course, all these things make sense if you use them together. I promised myself to write a quick tutorial on how to combine them properly, but it takes so much time to write a decent article...

[wavexx]

By the way, if you can get gtabview running, and try sections as shown above for a while, I'm very interested in feedback.

I personally find I organize code in blocks which are "self-contained", in the sense that re-evaluating the same block finishes in something ready for inspection. So what I do is:

• In the beginning of the block I make copies for temporaries (if needed), so that I can safely re-evaluate it at will.
• The code might span a few loops, maybe define and call a new function. You're not restricted anymore to a single paragraph.
• The block, as a result, makes much more sense. I generally describe them with 2-3 words just after the section header (#--- add value X to Y).
• Since there's no output by default when evaluating a block, the last few lines normally contain one print statement, or a call to tv() (which is an alias to gtabview.view as you saw before).
• If there's a bug, I can just poke around in the block using C-c p p without ever selecting anything.
• Since this leads to a number of globals, I added a "locals" browser (as you probably know from R-studio), which is bound to C-c p l in the earlier example. It's not fantastic, but I don't need more for now.

As you'll notice, gtabview has full support for pandas and numpy. I do statistical analysis in python whenever I can. After trying R-studio, ipython, inotebook and various derivatives, I really enjoy this setup.

I feel help should be more "browsable" to be better (I'd love to detect references, but it's way harder with python). Some functions are needed to re-evaluate the current file up to the current block (which would be equivalent to inotebook's "restart/reevaluate kernel").

I genuinely believe this is much more productive than ESS. Needs more polish and integration though.

[vspinu]

The block, as a result, makes much more sense. I generally describe them with 2-3 words just after the section header (#--- add value X to Y).

I like this idea. It's a simple transition from code documentation to evaluation blocks. I would be glad to push this to ESS but I am not very sure about the UI. I don't want yet another evaluation function.

Does it make sense for sections to have highest priority? Like in eval-region-or-section-or-defun-or-paragraph? If so, if you have only one ##---, when point is before the separator do you eval all the way from the begining ofthe buffer? Same for after the separator - do you eval to the end of the file? (BTW, python-shell-send-dwim doesnt work for me. No matter where I put the separator the whole buffer is executed. I will track it up tomorow).

As you'll notice, gtabview has full support for pandas and numpy.

Indeed, but it's outside emacs and that's quite inconvenient IMO.

When the the viewer is open I keep receiving:

Traceback (most recent call last):
  File "/home/vspinu/.local/lib/python2.7/site-packages/gtabview/viewer.py", line 70, in headerData
    return section if self.axis == (orientation - 1) else \
TypeError: unsupported operand type(s) for -: 'PySide.QtCore.Qt.Orientation' and 'int'
Traceback (most recent call last):
  File "/home/vspinu/.local/lib/python2.7/site-packages/gtabview/viewer.py", line 70, in headerData
    return section if self.axis == (orientation - 1) else \
TypeError: unsupported operand type(s) for -: 'PySide.QtCore.Qt.Orientation' and 'int'
Traceback (most recent call last):
  File "/home/vspinu/.local/lib/python2.7/site-packages/gtabview/viewer.py", line 70, in headerData
    return section if self.axis == (orientation - 1) else \
TypeError: unsupported operand type(s) for -: 'PySide.QtCore.Qt.Orientation' and 'int'
Traceback (most recent call last):
  File "/home/vspinu/.local/lib/python2.7/site-packages/gtabview/viewer.py", line 70, in headerData
    return section if self.axis == (orientation - 1) else \
TypeError: unsupported operand type(s) for -: 'PySide.QtCore.Qt.Orientation' and 'int'

The terminal part didn't work for me unfortunately:

>>> tv_locals()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vspinu/Dropbox/prog/python/pylab_local.py", line 35, in tv_locals
    tv(data)
  File "/home/vspinu/.local/lib/python2.7/site-packages/tabview/tabview.py", line 1315, in view
    if new_data:
  File "/home/vspinu/.local/lib/python2.7/site-packages/pandas/core/generic.py", line 731, in __nonzero__
    .format(self.__class__.__name__))
ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
>>> 

I feel help should be more "browsable" to be better (I'd love to detect references,

I will try to add some stuff. I quite like it the way it is. Much better than the Elpy stuff that I have tried. Let alone that it always gives help on object's classes. Love that.

I genuinely believe this is much more productive than ESS.

I tend to construct my code just as you described, but with paragraph blocks. There is one big advantage of sections - they force you to document pieces of code. I will definitely try it out.

[wavexx]

On 05/02/16 00:15, Vitalie Spinu wrote:

Does it make sense for sections to have highest priority? Like in eval-region-or-section-or-defun-or-paragraph? If so, if you have only one ##---, when point is before the separator do you eval all the way from the begining ofthe buffer? Same for after the separator - do you eval to the end of the file? (BTW, python-shell-send-dwim doesnt work for me. No matter where I put the separator the whole buffer is executed. I will track it up tomorow).

The separator is defined in python-section-delimiter and contains a space after #. It's normally "# ---".

The priority is normally folds, then sections. I did stuff like that in the paste:

--- bla

for ... ...

{{{ stuff to debug

...

}}}

...

---

If you eval before/after for loop, the entire block is sent as the fold is nested. But if you want to test the hot loop only, you can.

I use it rarely though. I changed style a few times before settling on what I wrote before.

As you'll notice, gtabview has full support for pandas and numpy.

Indeed, but it's outside emacs and that's quite inconvenient IMO.

I like it a lot more than ein (the emacs ipython interface). In notebooks you spend your whole time scrolling up and down cycling between one cell and the other because the output is in the middle.

Under X, you can align the plot/data/emacs window using a tiling window manager, and you get an interface which is pretty much the same as an ide.

"/home/vspinu/.local/lib/python2.7/site-packages/gtabview/viewer.py", line 70, in headerData return section if self.axis == (orientation - 1) else \ TypeError: unsupported operand type(s) for -: 'PySide.QtCore.Qt.Orientation' and 'int' |

I'm testing it using python-pyqt4 at the moment. It's a while I didn't run it using pyside.

tabview has been neglected lately, it's support for pandas is much more limited.

[vspinu]

It's normally "# ---".

Ok. I made it work. Would be good to accept #--- as well to avoid accidental confusion.

Also would be nice to mark blocks that you don't want to execute. Maybe # --- #. Then you can quickly activate deactivate whole sections. I am opening a new issue for this and related topics.

[wavexx]

On 05/02/16 00:15, Vitalie Spinu wrote:

When the the viewer is open I keep receiving:

|Traceback (most recent call last): File "/home/vspinu/.local/lib/python2.7/site-packages/gtabview/viewer.py", line 70, in headerData return section if self.axis == (orientation - 1) else \ TypeError: unsupported operand type(s) for -: 'PySide.QtCore.Qt.Orientation' and 'int' Traceback (most recent call last): File

I now should have fixed gtabview (in git) with PySide.

As for the tv_locals implementation, it was just a little fix to make it work for both python2 and python3:

from __future__ import generators
import pandas as pd

def tv_locals(private=False, modules=False, functions=False, max_width=128):
    import types, inspect
    data = []
    for k, v in inspect.currentframe().f_back.f_locals.items():
        if v.__class__.__name__ == '_Feature': continue
        if not private and k[0] == '_': continue
        if not modules and type(v) == types.ModuleType: continue
        if not functions and type(v) == types.FunctionType: continue
        value = v
        if isinstance(value, pd.DataFrame):
            value = list(value.columns)
        value = repr(str(value))[1:-1]
        if len(value) > max_width:
            value = value[:max_width - 1] + '…'
        data.append([k, v.__class__.__name__, value])
    data = pd.DataFrame(data, columns=['NAME', 'TYPE', 'VALUE'])
    data.set_index(data['NAME'], inplace=True)
    data.drop('NAME', axis=1, inplace=True)
    tv(data, title='bound local variables')