[Discussion] Splitting UCX-Py into two packages

[madsbk]

I suggest that we split UCX-Py into two Python packages: one targeting low-level system developers and one targeting high-level Python developers.

ucxpy (Python bindings for UCX)

Low-level Python wrappers of the UCX API where handlers such as ucp_worker_h, ucp_listener_h and ucp_ep_h are wrapped in Python classes.

This package uses (almost) the same semantic as UCX and will therefore be easy to migrate upstream to the UCX repository.

aucxpy (Asynchronous UCXPY)

High-level pure Python library that make use of the primitives in ucxpy to implement a new semantic that is based on asyncio where communication calls return awaitable futures instead of callback functions.

Generally, at this level we try to make the common use case as easy as possible; taking the liberty to change the API to fit Python users.

---

What are your thoughts? Any package name suggestions?

[pentschev]

Generally speaking, I have no preference either to split or leave as it is. The argument that the proposed ucxpy is closer to UCX semantics and will be easier to upstream to UCX repository makes sense to me and is very compelling too.

What's not clear to me from the proposal is whether the packages would be dependent on each other or not, it seems that aucxpy will depend on ucxpy, but I'm not totally sure my understanding is correct.

If we decide to split, could that wait for post-GTC? I'm worried this may cause delays and we already have little time to get everything done.

[madsbk]

What's not clear to me from the proposal is whether the packages would be dependent on each other or not, it seems that aucxpy will depend on ucxpy, but I'm not totally sure my understanding is correct.

Yes, aucxpy will be a pure Python library the make use of the primitives in ucxpy to implement a high-level async library. (Updated the description to make this more clear)

If we decide to split, could that wait for post-GTC?

Agree, no need to hurry this. Also, we properly need to keep the ucp package around for a while and make it forward to aucxpy.

[quasiben]

Splitting up the package makes sense to me especially if we want to upstream and even get some help maintaining bindings. Definitely +1 for a post GTC effort.

Not a big fan of aucxpy and ucxpy. I think the names are bit too close and will lead to some confusion. But I also don't have a good alternative at this time. It's worth noting that we are building some marketing around UCX-Py, so maybe keep this name and change the bindings name -- ucxpy-bind ?

[pentschev]

Yeah, perhaps a name that doesn't seem a scramble of letters would be better, ucxpy already seems like a scramble, so something like ucxpy-async would make things a little more palatable.

[madsbk]

Or maybe ucxpy-core and ucxpy ?