Fix #9: Support other kernels

[ilysym]

Main changes are:

• added :kernel code block argument
• added kernel launcher mode to driver; in place of ipython kernel, the driver is started as python driver.py --kernel kr --conn-file file
• request completed only after idle message is received from PUB/SUB socket; this ensures that execute_result and remaining stdout messages are received
• reply message could be received before the handler is registered, so callback waits for the handler to be added to list

[smartass101]

You might want to have a look at PR #39 and how we could make it work together with your proposals. You're concerned with starting arbitrary kernels which I thought different :session arguments already addresses. Nevertheless, I find your message handling improvements very interesting.

[acowley]

This did not work for me.

When evaluating a source block, a few messages pass by the echo area before a rather long wait (perhaps 30s or 1 minute) on the message "Sent python-eldoc-setup-code", then the debugger kicks in.

Debugger entered--Lisp error: (file-error "Writing to process" "bad file descriptor" #<process Python>)
  process-send-string(#<process Python> "import codecs, os;__pyfile = codecs.open('''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''');exec(compile(__code, '''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''', 'exec'));")
  comint-send-string(#<process Python> "import codecs, os;__pyfile = codecs.open('''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''');exec(compile(__code, '''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''', 'exec'));")
  python-shell-send-string("import codecs, os;__pyfile = codecs.open('''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''');exec(compile(__code, '''/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk''', 'exec'));" #<process Python>)
  python-shell-send-file("/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk" #<process Python> "/var/folders/n6/0j2z684n2pq_0d5kphzzvjs80000gn/T/py6123sQk" t)
  python-shell-send-string("def __PYDOC_get_help(obj):\n    try:\n        import inspect\n        if hasattr(obj, 'startswith'):\n            obj = eval(obj, globals())\n        doc = inspect.getdoc(obj)\n        if not doc and callable(obj):\n            target = None\n            if inspect.isclass(obj) and hasattr(obj, '__init__'):\n                target = obj.__init__\n                objtype = 'class'\n            else:\n                target = obj\n                objtype = 'def'\n            if target:\n                args = inspect.formatargspec(\n                    *inspect.getargspec(target)\n                )\n                name = obj.__name__\n                doc = '{objtype} {name}{args}'.format(\n                    objtype=objtype, name=name, args=args\n                )\n        else:\n            doc = doc.splitlines()[0]\n    except:\n        doc = ''\n    try:\n        exec('print doc')\n    except SyntaxError:\n        print(doc)" #<process Python>)
  python-shell-send-setup-code()
  run-hooks(change-major-mode-after-body-hook comint-mode-hook inferior-python-mode-hook)
  apply(run-hooks (change-major-mode-after-body-hook comint-mode-hook inferior-python-mode-hook))
  run-mode-hooks(inferior-python-mode-hook)
  inferior-python-mode()
  python-shell-make-comint("ipython console --existing emacs-default.json" "Python" nil)
  run-python("ipython console --existing emacs-default.json" nil nil)
  ob-ipython--create-repl("default")
  org-babel-ipython-initiate-session(nil ((:comments . "") (:shebang . "") (:cache . "no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . "code") (:results . "none") (:hlines . "no") (:session) (:result-type . value) (:result-params "none") (:rowname-names) (:colname-names)))
  org-babel-execute:ipython("%matplotlib inline\nimport matplotlib.pyplot as plt" ((:comments . "") (:shebang . "") (:cache . "no") (:padline . "") (:noweb . "no") (:tangle . "no") (:exports . "code") (:results . "none") (:hlines . "no") (:session) (:result-type . value) (:result-params "none") (:rowname-names) (:colname-names)))
  org-babel-execute-src-block(nil)
  org-babel-execute-src-block-maybe()
  org-babel-execute-maybe()
  org-babel-execute-safely-maybe()
  run-hook-with-args-until-success(org-babel-execute-safely-maybe)
  org-ctrl-c-ctrl-c(nil)
  call-interactively(org-ctrl-c-ctrl-c)
  god-mode-self-insert()
  call-interactively(god-mode-self-insert nil nil)
  command-execute(god-mode-self-insert)

The *Python* buffer says,

Jupyter Console 4.1.0

ERROR: Kernel did not respond

Shutting down kernel

Process Python finished

repeated a few times.

The master branch works fine. This is with Python 3.4.4 and Emacs 24.5.2 on OS X.

[ilysym]

Try same code in ordinary Python mode, i.e. paste code (without %matplotlib inline) to .py file and hit \C-c\C-c

[acowley]

@ilysym You're right on target! I'm having the issue with ordinary Python, too. I'll work on getting it sorted; do you have any advice?

EDIT: Just running python3 in shell revealed that python.el was likely having problems due to a shell variable leaking in and messing up PYTHONPATH. I resolved this (turn it off and back on again) so that regular Python files now properly work with the REPL.

Disappointingly, this did not resolve my issue with this branch of ob-ipython which produces the same error as above.

[acowley]

The issues were Python 3 compatibility things: mixed spaces and tabs, use of has_key, and use of filter. I've made a PR to @ilysym's fork that lets me mix python and haskell kernels in a single Org document.

[acowley]

Pinging @gregsexton

This is a great PR, please consider merging it to make installation easier for others to try it!

[chu-]

Yes it's a great PR. @acowley Did you test this PR successfully?

[gregsexton]

Hi. Sorry. Life has been extremely busy for me recently. I have a backlog of PRs/issues to review and merge. I have every intention of merging this.

[acowley]

@chu- I've used it heavily for the past few weeks. The only bad part is needing to associate ipython with whatever language you want to use for a particular block. This makes it tricky to work with a single-document-multi-kernel. Addressing that seems like a great bit of extra functionality to add after this PR is merged.

[chu-]

@acowley Cool, Thanks! I use the following code block in org-mode to test julia kernel, but it failed [TerminalIPythonApp] CRITICAL | Unrecognized flag: '--conn-file' any suggestions?

#+BEGIN_SRC ipython :session julia_test :kernel julia-0.5
sin(2.0)
#+END_SRC

Setting: (with-eval-after-load 'org (org-babel-do-load-languages 'org-babel-load-languages '( (ipython . t) )) )

[acowley]

@chu- I just tried here, but I only have julia-0.4.2.

#+PROPERTY: header-args:ipython :session :kernel julia-0.4

* Setup                                                            :noexport:
#+BEGIN_SRC elisp :results silent
(add-to-list 'org-src-lang-modes '("ipython" . julia))
#+END_SRC

* A first example

#+BEGIN_SRC ipython :results value
f(x) = x * x
f(9)
#+END_SRC

#+RESULTS:
: 81

#+BEGIN_SRC ipython :results output
code_llvm(f, (Float64,))
#+END_SRC

#+RESULTS:
: 
: define double @julia_f_21609(double) {
: top:
:   %1 = fmul double %0, %0
:   ret double %1
: }

Debugging from within emacs is nearly impossible, so make sure to give it a try in a terminal with jupyter console --kernel julia-0.5 to see if there are any problems there.

[chu-]

@acowley Great! thanks.

[hsseung]

I'd be delighted if this could be merged. I agree with others that it's an important feature.

[stardiviner]

Great feature! But seems not merged yet. Any update on this?

[gregsexton]

This causes emacs to hang for me continously. Will try and debug.

[gregsexton]

Ok. Merged! Thanks for the PR it adds a great feature and moves ob-ipython towards ob-jupyter. :)

Good catch on the race condition. I simplified the mechanism you used though. I was able to get it to deadlock with inputs that just did imports. It also backed up on messages that didn't have a handler associated.

[chu-]

Excellent!

On Sun, Apr 24, 2016 at 10:12 PM, Greg Sexton notifications@github.com wrote:

Closed #38 https://github.com/gregsexton/ob-ipython/pull/38.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/gregsexton/ob-ipython/pull/38#event-639176180

[hsseung]

Wonderful!