search

martius-lab / cluster_utils

https://cluster-utils.readthedocs.io/stable/
Other
12 stars 1 forks source link

Cannot use pyuv as is on PyPI #80

Closed luator closed 4 months ago

luator commented 7 months ago

Currently we depend on the master-branch version of pyuv as it contains unreleased fixes for recent Python versions. Since PyPI doesn't allow dependencies on external sources, this is blocking us from publishing cluster_utils there.

pyuv does not seem to be actively worked on anymore, so it's a bit unclear if a new release will happen any time soon (the maintainer is still around, though, see related issue).

So until a new release of pyuv happens, I see the following potential workarounds

  1. Add pyuv as subpackage to cluster_utils, so we don't have an external dependency on it. Not sure how much this would complicate the build process.
  2. Fork pyuv and put our fork on PyPI. I'm not a big fan of that, though, as it pollutes PyPI with temporary workaround packages.
  3. Find an alternative to pyuv which is more actively maintained. At least during some quick research I didn't find an obvious replacement, though.
luator commented 7 months ago

Actually, getting a new official release might not be that far away, see latest comments on the issue linked above :).

mseitzer commented 7 months ago

Thanks for asking them! Let's hope pyuv gets a new release, as replacing it will not be trivial.

An alternative could be the Python asyncio event loop, or its faster implementation, uvloop. However, I think those are missing some components we would need (e.g. a UDP implementation).

luator commented 5 months ago

I looked a bit into this again. It seems implementing a UDP server with asyncio is rather trivial. See https://stackoverflow.com/a/64540509/2095383. I did a quick, hacky proof of concept, using that instead of pyuv. It currently still has a few quirks that need to be resolved (e.g. proper SIGINT handling) but at least the communication seems to be working.

So I suggest that we switch to asyncio and get rid of the third-party dependency.

mseitzer commented 5 months ago

Okay, sounds promising. With network communication, the devil is always in the detail though..