search

vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.28k stars 590 forks source link

[BUG-REPORT] vaex.execution:nested execute call in Jupyterlab #902

Open ichsan2895 opened 4 years ago

ichsan2895 commented 4 years ago

Hello, I'm new user with vaex an I'm excited to learn it

Description This error happen when I try to run code from this reference Jupyter integration: interactivity

The code was

df = vaex.example()  # we create the dataframe again, to leave all the plots above 'alone'
heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
heatmap_xy

Software information

Additional information Error message:

ERROR:MainThread:vaex.execution:nested execute call
ERROR:MainThread:vaex.jupyter:Error while executing tasks
Traceback (most recent call last):
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\__init__.py", line 41, in execute_debounced
    await self.df.execute_async()
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 320, in execute_async
    await self.executor.execute_async()
  File "G:\Anaconda3\lib\site-packages\vaex\execution.py", line 105, in execute_async
    raise RuntimeError("nested execute call: %r %r\nlast trace:\n%s\ncurrent trace:\n%s" % (chunk_executor_thread, self.local.executing, self.local.last_trace, trace))
RuntimeError: nested execute call: False True
last trace:
  File "G:\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "G:\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "G:\Anaconda3\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "G:\Anaconda3\lib\site-packages\traitlets\config\application.py", line 664, in launch_instance
    app.start()
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelapp.py", line 597, in start
    self.io_loop.start()
  File "G:\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 149, in start
    self.asyncio_loop.run_forever()
  File "G:\Anaconda3\lib\asyncio\base_events.py", line 541, in run_forever
    self._run_once()
  File "G:\Anaconda3\lib\asyncio\base_events.py", line 1786, in _run_once
    handle._run()
  File "G:\Anaconda3\lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "G:\Anaconda3\lib\site-packages\tornado\ioloop.py", line 690, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "G:\Anaconda3\lib\site-packages\tornado\ioloop.py", line 743, in _run_callback
    ret = callback()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 787, in inner
    self.run()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 381, in dispatch_queue
    yield self.process_one()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 225, in wrapper
    runner = Runner(result, future, yielded)
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 714, in __init__
    self.run()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 365, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 545, in execute_request
    user_expressions, allow_stdin,
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "G:\Anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 300, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "G:\Anaconda3\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2867, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2895, in _run_cell
    return runner(coro)
  File "G:\Anaconda3\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3072, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3263, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-aca4e0325e41>", line 3, in <module>
    heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\__init__.py", line 110, in heatmap
    x, y = self._axes([x, y], limits)
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\__init__.py", line 73, in _axes
    limits = self.df.limits(expressions, limits)
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 1663, in limits
    return self._delay(delay, finish(limits_list))
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 1463, in _delay
    self.execute()
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 315, in execute
    just_run(self.execute_async())
  File "G:\Anaconda3\lib\site-packages\vaex\asyncio.py", line 35, in just_run
    return loop.run_until_complete(coro)
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 54, in run_until_complete
    self._run_once()
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 85, in _run_once
    handle._run()
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 150, in run
    ctx.run(self._callback, *self._args)
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 108, in step
    step_orig(task, exc)
  File "G:\Anaconda3\lib\asyncio\tasks.py", line 249, in __step
    result = coro.send(None)
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 320, in execute_async
    await self.executor.execute_async()
  File "G:\Anaconda3\lib\site-packages\vaex\execution.py", line 102, in execute_async
    trace = ''.join(traceback.format_stack())

current trace:
  File "G:\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "G:\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "G:\Anaconda3\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "G:\Anaconda3\lib\site-packages\traitlets\config\application.py", line 664, in launch_instance
    app.start()
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelapp.py", line 597, in start
    self.io_loop.start()
  File "G:\Anaconda3\lib\site-packages\tornado\platform\asyncio.py", line 149, in start
    self.asyncio_loop.run_forever()
  File "G:\Anaconda3\lib\asyncio\base_events.py", line 541, in run_forever
    self._run_once()
  File "G:\Anaconda3\lib\asyncio\base_events.py", line 1786, in _run_once
    handle._run()
  File "G:\Anaconda3\lib\asyncio\events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "G:\Anaconda3\lib\site-packages\tornado\ioloop.py", line 690, in <lambda>
    lambda f: self._run_callback(functools.partial(callback, future))
  File "G:\Anaconda3\lib\site-packages\tornado\ioloop.py", line 743, in _run_callback
    ret = callback()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 787, in inner
    self.run()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 381, in dispatch_queue
    yield self.process_one()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 225, in wrapper
    runner = Runner(result, future, yielded)
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 714, in __init__
    self.run()
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 748, in run
    yielded = self.gen.send(value)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 365, in process_one
    yield gen.maybe_future(dispatch(*args))
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
    yield gen.maybe_future(handler(stream, idents, msg))
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "G:\Anaconda3\lib\site-packages\ipykernel\kernelbase.py", line 545, in execute_request
    user_expressions, allow_stdin,
  File "G:\Anaconda3\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "G:\Anaconda3\lib\site-packages\ipykernel\ipkernel.py", line 300, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "G:\Anaconda3\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2867, in run_cell
    raw_cell, store_history, silent, shell_futures)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2895, in _run_cell
    return runner(coro)
  File "G:\Anaconda3\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3072, in run_cell_async
    interactivity=interactivity, compiler=compiler, result=result)
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3263, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "G:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-aca4e0325e41>", line 3, in <module>
    heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\__init__.py", line 110, in heatmap
    x, y = self._axes([x, y], limits)
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\__init__.py", line 73, in _axes
    limits = self.df.limits(expressions, limits)
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 1663, in limits
    return self._delay(delay, finish(limits_list))
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 1463, in _delay
    self.execute()
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 315, in execute
    just_run(self.execute_async())
  File "G:\Anaconda3\lib\site-packages\vaex\asyncio.py", line 35, in just_run
    return loop.run_until_complete(coro)
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 54, in run_until_complete
    self._run_once()
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 85, in _run_once
    handle._run()
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 150, in run
    ctx.run(self._callback, *self._args)
  File "G:\Anaconda3\lib\asyncio\tasks.py", line 329, in __wakeup
    self.__step()
  File "G:\Anaconda3\lib\site-packages\nest_asyncio.py", line 108, in step
    step_orig(task, exc)
  File "G:\Anaconda3\lib\asyncio\tasks.py", line 249, in __step
    result = coro.send(None)
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\utils.py", line 184, in run_async
    result = await result
  File "G:\Anaconda3\lib\site-packages\vaex\jupyter\__init__.py", line 41, in execute_debounced
    await self.df.execute_async()
  File "G:\Anaconda3\lib\site-packages\vaex\dataframe.py", line 320, in execute_async
    await self.executor.execute_async()
  File "G:\Anaconda3\lib\site-packages\vaex\execution.py", line 102, in execute_async
    trace = ''.join(traceback.format_stack())

Loading widget...
maartenbreddels commented 4 years ago

Does this happen every time?

ichsan2895 commented 4 years ago

Does this happen every time?

Thanks for respond. The error message is disappear when I run that cell again. But the widget still doesn't appear.

bug_vaex2

I try to run this command to make sure all extension are already installed. bug_vaex

*I'm sorry I accidentally clicked 'Close Issue'. Now I reopen again

maartenbreddels commented 4 years ago

That might indeed be a widget issue (maybe try jupyter notebook). Could it be that you ran the cell twice?

ichsan2895 commented 4 years ago

That might indeed be a widget issue (maybe try jupyter notebook). Could it be that you ran the cell twice?

If I ran the cell one time, the error message appear and widget didn't appear. If I ran the cell twice, the error message disappear and widget still didn't appear.

Okay I will give a try Jupyter notebook later..

ichsan2895 commented 4 years ago

Hello, the widget appeared when I used jupyter notebook instead jupyterlab. But, the error message still appears (sometimes). It will disappear if I run the cell twice

maartenbreddels commented 4 years ago

Note that if you execute:

df = vaex.example()  # we create the dataframe again, to leave all the plots above 'alone'
heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
heatmap_xy

Twice, it will trigger that exception. If you do:

df = vaex.example()  # we create the dataframe again, to leave all the plots above 'alone'
heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
await vaex.jupyter.gather()
heatmap_xy

As mentioned in the tutorial, it should work fine, does that help you?

ichsan2895 commented 4 years ago

Note that if you execute:

df = vaex.example()  # we create the dataframe again, to leave all the plots above 'alone'
heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
heatmap_xy

Twice, it will trigger that exception. If you do:

df = vaex.example()  # we create the dataframe again, to leave all the plots above 'alone'
heatmap_xy = df.widget.heatmap(df.x, df.y, selection=[None, True])
await vaex.jupyter.gather()
heatmap_xy

As mentioned in the tutorial, it should work fine, does that help you?

Hello Mr. Maarten breddels, I'm sorry to say, it still happen. But, I guest, this error happen because I use "restart kernel and run all cells" button.. I think it's not serious bug since it will disappear on second attemp. But, Jupyterlab couldn't show any widget. Jupyter notebook still rocks.

maartenbreddels commented 4 years ago

Can you please share you notebook so I can take a look?