search

dask / dask-kubernetes

Native Kubernetes integration for Dask
https://kubernetes.dask.org
BSD 3-Clause "New" or "Revised" License
312 stars 143 forks source link

Asynchronous processing #16

Closed mrocklin closed 4 years ago

mrocklin commented 6 years ago

I notice that the kubernetes API has an async= keyword. Do you happen to know if people use this library with Tornado?

Incidentally this library is new and pops up in web searches: https://github.com/kippandrew/client-python-tornado It's obviously very new though.

mrocklin commented 6 years ago

Or maybe swagger can give this to us magically?

mrocklin commented 5 years ago

@yuvipanda if I reall correctly you mentioned that there was an asyncio kubernetes library that was decent. Would you mind pointing out which one you meant here? (doing a web search turns up a couple)

mrocklin commented 5 years ago

Never mind, on further investigation I see that @yuvipanda has been committing to https://github.com/olitheolix/aiokubernetes

yuvipanda commented 5 years ago

@mrocklin https://github.com/kubernetes-client/python/issues/323 has discussion and links to two libraries. The one you linked to, and https://github.com/tomplus/kubernetes_asyncio. It's unclear which one to throw weight behind :) I like the design of https://github.com/olitheolix/aiokubernetes better though.

the async= parameter in kubernetes API is pretty bad - it makes a new thread and returns it! I think it has been changed to making a new threadpool and returning a thread from it, but I wouldn't consider any of these good python practices.

@minrk also has thoughts on this.

minrk commented 5 years ago

The ThreadPool approach in kubernetes >=4 is the same as the one we use in kubespawner/binderhub (return ThreadPoolExecutor.submit(blocking_func)), and it works great (we just run kubernetes calls in a background thread of our own instead of passing an async arg per call). The downsides of using Python threads are often overstated, but a native asyncio API is still nice. From the looks of the examples, I would pick kubernetes_asyncio, which seems to have the most "the same as before, but add await" API, rather than what appears to be a much more complex and verbose API in aiokubernetes.

jacobtomlinson commented 4 years ago

Closed in #162

mrocklin commented 4 years ago

Woo!

On Mon, Oct 14, 2019 at 5:21 AM Jacob Tomlinson notifications@github.com wrote:

Closed #16 https://github.com/dask/dask-kubernetes/issues/16.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-kubernetes/issues/16?email_source=notifications&email_token=AACKZTFYFD6VJXXDVI35SLTQORB2TA5CNFSM4EKP5HLKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUGC7ZQY#event-2709912771, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTFEFJLUXJJR5ZEZ4MDQORB2TANCNFSM4EKP5HLA .