search

vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.28k stars 590 forks source link

[question] Multicore cpu async tasks. #2080

Closed kv-gits closed 2 years ago

kv-gits commented 2 years ago

Comparing with dask (https://examples.dask.org/applications/async-web-server.html) is it possible in Vaex to mix cpu bound tasks and io bound tasks asyncronously (python asyncio module) with usage of all available cpu cores? Do I need create vaex server or just use futures is ok?

maartenbreddels commented 2 years ago

Hi,

Yes, take a look at: https://vaex.io/docs/example_async.html if you start an asyncio event loop in the main thread, and await a vaex tasks, your main thread will handle other asyncio tasks, all vaex parts will execute in other threads. No need to start a server, although it does would work fine with e.g. starlette/fastapi.

Does that answer you question?

Regards,

Maarten