Controlling neovim from jupyter notebook: "another loop is running"

[beyarkay]

I've got a jupyter notebook, from which I want to control an instance of neovim. Following the instructions in the pynvim repo, I can setup neovim to listen like so:

NVIM_LISTEN_ADDRESS=/tmp/nvim nvim

and then I can control that instance of neovim from a regular (non-notebook) python session like so:

from pynvim import attach
nvim = attach('socket', path='/tmp/nvim')
nvim.command('vsplit')

The above all works fine from a regular python session (python3 or ipython) but it doesn't work from a jupyter notebook. Trying to run the following cell:

from pynvim import attach
nvim = attach('socket', path='/tmp/nvim')

Results in the following error:

 RuntimeError: Cannot run the event loop while another loop is running 

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [9], in <cell line: 3>()
      1 # %pip install pynvim
      2 from pynvim import attach
----> 3 nvim = attach('socket', path='/tmp/nvim')

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/__init__.py:111, in attach(session_type, address, port, path, argv, decode)
     80 def attach(session_type, address=None, port=None,
     81            path=None, argv=None, decode=None):
     82     """Provide a nicer interface to create python api sessions.
     83 
     84     Previous machinery to create python api sessions is still there. This only
   (...)
    108 
    109     """
    110     session = (tcp_session(address, port) if session_type == 'tcp' else
--> 111                socket_session(path) if session_type == 'socket' else
    112                stdio_session() if session_type == 'stdio' else
    113                child_session(argv) if session_type == 'child' else
    114                None)
    116     if not session:
    117         raise Exception('Unknown session type "%s"' % session_type)

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:35, in socket_session(path)
     33 def socket_session(path):
     34     """Create a msgpack-rpc session from a unix domain socket."""
---> 35     return session('socket', path)

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:19, in session(transport_type, *args, **kwargs)
     18 def session(transport_type='stdio', *args, **kwargs):
---> 19     loop = EventLoop(transport_type, *args, **kwargs)
     20     msgpack_stream = MsgpackStream(loop)
     21     async_session = AsyncSession(msgpack_stream)

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:93, in BaseEventLoop.__init__(self, transport_type, *args)
     91 except Exception as e:
     92     self.close()
---> 93     raise e
     94 self._start_reading()

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:90, in BaseEventLoop.__init__(self, transport_type, *args)
     88 self._init()
     89 try:
---> 90     getattr(self, '_connect_{}'.format(transport_type))(*args)
     91 except Exception as e:
     92     self.close()

File ~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/asyncio.py:96, in AsyncioEventLoop._connect_socket(self, path)
     94 else:
     95     coroutine = self._loop.create_unix_connection(self._fact, path)
---> 96 self._loop.run_until_complete(coroutine)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:622, in BaseEventLoop.run_until_complete(self, future)
    611 """Run until the Future is done.
    612 
    613 If the argument is a coroutine, it is wrapped in a Task.
   (...)
    619 Return the Future's result, or raise its exception.
    620 """
    621 self._check_closed()
--> 622 self._check_running()
    624 new_task = not futures.isfuture(future)
    625 future = tasks.ensure_future(future, loop=self)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:584, in BaseEventLoop._check_running(self)
    582     raise RuntimeError('This event loop is already running')
    583 if events._get_running_loop() is not None:
--> 584     raise RuntimeError(
    585         'Cannot run the event loop while another loop is running')

RuntimeError: Cannot run the event loop while another loop is running

Which looks like both jupyter and pynvim are trying to run async loops at the same time. I'm not sure how to get around the issue though.

How can I control neovim from a python jupyter notebook?

I'm running on a Mac, Python 3.11.3, IPython version 8.4.0, ipykernel version 6.15.1, pynvim version 0.4.3

[beyarkay]

GH doesn't support uploading ipynb files, so here's the raw text of a minimal Jupyter notebook to reproduce the issue:

Expand me for the notebook

``` { "cells": [ { "cell_type": "code", "execution_count": 5, "id": "5636335d", "metadata": { "ExecuteTime": { "end_time": "2023-09-15T10:31:53.742817Z", "start_time": "2023-09-15T10:31:42.902843Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: pynvim in /Users/brk/Library/Python/3.10/lib/python/site-packages (0.4.3)\n", "Requirement already satisfied: msgpack>=0.5.0 in /Users/brk/Library/Python/3.10/lib/python/site-packages (from pynvim) (1.0.5)\n", "Requirement already satisfied: greenlet in /Users/brk/Library/Python/3.10/lib/python/site-packages (from pynvim) (2.0.2)\n", "Note: you may need to restart the kernel to use updated packages.\n", "!Make sure you start neovim listening with: `NVIM_LISTEN_ADDRESS=/tmp/nvim nvim`. Press any key to continue\n" ] }, { "ename": "RuntimeError", "evalue": "Cannot run the event loop while another loop is running", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)", "Input \u001b[0;32mIn [5]\u001b[0m, in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpynvim\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m attach\n\u001b[1;32m 3\u001b[0m blocker \u001b[38;5;241m=\u001b[39m \u001b[38;5;28minput\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m!Make sure you start neovim listening with: `NVIM_LISTEN_ADDRESS=/tmp/nvim nvim`. Press any key to continue\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 4\u001b[0m nvim \u001b[38;5;241m=\u001b[39m \u001b[43mattach\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43msocket\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpath\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m/tmp/nvim\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5\u001b[0m nvim\u001b[38;5;241m.\u001b[39mcommand(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mvsplit\u001b[39m\u001b[38;5;124m'\u001b[39m)\n", "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/__init__.py:111\u001b[0m, in \u001b[0;36mattach\u001b[0;34m(session_type, address, port, path, argv, decode)\u001b[0m\n\u001b[1;32m 80\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mattach\u001b[39m(session_type, address\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, port\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[1;32m 81\u001b[0m path\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, argv\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, decode\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[1;32m 82\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Provide a nicer interface to create python api sessions.\u001b[39;00m\n\u001b[1;32m 83\u001b[0m \n\u001b[1;32m 84\u001b[0m \u001b[38;5;124;03m Previous machinery to create python api sessions is still there. This only\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 108\u001b[0m \n\u001b[1;32m 109\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[1;32m 110\u001b[0m session \u001b[38;5;241m=\u001b[39m (tcp_session(address, port) \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtcp\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[0;32m--> 111\u001b[0m \u001b[43msocket_session\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpath\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124msocket\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m 112\u001b[0m stdio_session() \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstdio\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m 113\u001b[0m child_session(argv) \u001b[38;5;28;01mif\u001b[39;00m session_type \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mchild\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;28;01melse\u001b[39;00m\n\u001b[1;32m 114\u001b[0m \u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[1;32m 116\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m session:\n\u001b[1;32m 117\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mUnknown session type \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m'\u001b[39m \u001b[38;5;241m%\u001b[39m session_type)\n", "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:35\u001b[0m, in \u001b[0;36msocket_session\u001b[0;34m(path)\u001b[0m\n\u001b[1;32m 33\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msocket_session\u001b[39m(path):\n\u001b[1;32m 34\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Create a msgpack-rpc session from a unix domain socket.\"\"\"\u001b[39;00m\n\u001b[0;32m---> 35\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43msession\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43msocket\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpath\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/__init__.py:19\u001b[0m, in \u001b[0;36msession\u001b[0;34m(transport_type, *args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msession\u001b[39m(transport_type\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mstdio\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m---> 19\u001b[0m loop \u001b[38;5;241m=\u001b[39m \u001b[43mEventLoop\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtransport_type\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 20\u001b[0m msgpack_stream \u001b[38;5;241m=\u001b[39m MsgpackStream(loop)\n\u001b[1;32m 21\u001b[0m async_session \u001b[38;5;241m=\u001b[39m AsyncSession(msgpack_stream)\n", "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:93\u001b[0m, in \u001b[0;36mBaseEventLoop.__init__\u001b[0;34m(self, transport_type, *args)\u001b[0m\n\u001b[1;32m 91\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 92\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclose()\n\u001b[0;32m---> 93\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m e\n\u001b[1;32m 94\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_start_reading()\n", "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/base.py:90\u001b[0m, in \u001b[0;36mBaseEventLoop.__init__\u001b[0;34m(self, transport_type, *args)\u001b[0m\n\u001b[1;32m 88\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_init()\n\u001b[1;32m 89\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 90\u001b[0m \u001b[38;5;28;43mgetattr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43m_connect_\u001b[39;49m\u001b[38;5;132;43;01m{}\u001b[39;49;00m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mformat\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtransport_type\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 91\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 92\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mclose()\n", "File \u001b[0;32m~/Library/Python/3.10/lib/python/site-packages/pynvim/msgpack_rpc/event_loop/asyncio.py:96\u001b[0m, in \u001b[0;36mAsyncioEventLoop._connect_socket\u001b[0;34m(self, path)\u001b[0m\n\u001b[1;32m 94\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 95\u001b[0m coroutine \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_loop\u001b[38;5;241m.\u001b[39mcreate_unix_connection(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_fact, path)\n\u001b[0;32m---> 96\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_loop\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_until_complete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcoroutine\u001b[49m\u001b[43m)\u001b[49m\n", "File \u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:622\u001b[0m, in \u001b[0;36mBaseEventLoop.run_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m 611\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Run until the Future is done.\u001b[39;00m\n\u001b[1;32m 612\u001b[0m \n\u001b[1;32m 613\u001b[0m \u001b[38;5;124;03mIf the argument is a coroutine, it is wrapped in a Task.\u001b[39;00m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 619\u001b[0m \u001b[38;5;124;03mReturn the Future's result, or raise its exception.\u001b[39;00m\n\u001b[1;32m 620\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 621\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_closed()\n\u001b[0;32m--> 622\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_check_running\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 624\u001b[0m new_task \u001b[38;5;241m=\u001b[39m \u001b[38;5;129;01mnot\u001b[39;00m futures\u001b[38;5;241m.\u001b[39misfuture(future)\n\u001b[1;32m 625\u001b[0m future \u001b[38;5;241m=\u001b[39m tasks\u001b[38;5;241m.\u001b[39mensure_future(future, loop\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n", "File \u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py:584\u001b[0m, in \u001b[0;36mBaseEventLoop._check_running\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 582\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mThis event loop is already running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 583\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m events\u001b[38;5;241m.\u001b[39m_get_running_loop() \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 584\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[1;32m 585\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mCannot run the event loop while another loop is running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n", "\u001b[0;31mRuntimeError\u001b[0m: Cannot run the event loop while another loop is running" ] } ], "source": [ "%pip install pynvim\n", "from pynvim import attach\n", "blocker = input(\"!Make sure you start neovim listening with: `NVIM_LISTEN_ADDRESS=/tmp/nvim nvim`. Press any key to continue\")\n", "nvim = attach('socket', path='/tmp/nvim')\n", "nvim.command('vsplit')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": true }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 5 } ```

[justinmk]

Don't have an answer to this unfortunately. Known issue: https://github.com/neovim/pynvim/issues/489

Possible workaround: https://github.com/neovim/pynvim/issues/326#issuecomment-403166302

[beyarkay]

Thanks for the swift reply

[wookayin]

A workaround is to use nest_asyncio. It is not a perfect solution as the behavior of nested asyncio loop can be quite unintuitive and strange, but works at least minimally:

[1]: import nest_asyncio
     nest_asyncio.apply()

[2]: from pynvim import attach
     nvim = attach('socket', path='/tmp/nvim.sock')

[3]: nvim.current.buffer[:] = 'foo'