search

emacs-jupyter / jupyter

An interface to communicate with Jupyter kernels.
GNU General Public License v3.0
940 stars 92 forks source link

PySpark kernel support #292

Open fbielejec opened 4 years ago

fbielejec commented 4 years ago

Fist of all, hats down for making this package available to us all. I much prefer it to using ein for the simplicity it brings.

I'm evaluating using it with an AWS glue development endpoint for developing ETL scripts, which is nothing more than Livy API endpoint to an AWS managed Spark instance.

Suffice to say after following AWS instructions and installing sparkmagic and M+x jupyter-run-repl + selectign PySpark for a session I get a fully functioning REPL:

pyspark_kernel

Nice! What I can't seem to do is to start evaluating a python buffer code in the REPL.

C-c C-c gives: user-error: Start a Python process first with ‘M-x run-python’ or ‘C-c C-p’.

M+x jupyter-repl-associate-buffer No REPL formajor-mode' exists. Start one? (y or n) y` Starting one I can select the PySpark kernel and create another repl, but still cannot evaluate code in it.

M+x jupyter-eval Wrong type argument: "Need a client to read an expression", nil, jupyter-current-client

fleimgruber commented 3 years ago

I guess jupyter-repl-associate-buffer does not work as you expect because the kernel language pyspark does not correspond to the major-mode of the buffer you want to associate, see also here.

Maybe such cases can be solved by a custom variable that maps REPL kernel languages to major-modes in such cases? I don't know about the implications and problems with that approach. @nnicandro is the current behaviour there for it's simplicity or because other approaches (such as the suggested one) are problematic?

nnicandro commented 3 years ago

@fbielejec When you run M-: jupyter-repl-lang-mode RET in the PySpark REPL buffer what do you get?

@fleimgruber The current behavior is there because it's simpler. I don't think there would be any problems with mapping the kernel language to a major-mode. That is already kind of done in jupyter-kernel-language-mode-properties.