MostAwesomeDude
commented
1 year ago Oh, this isn't breaking anything; I don't recommend shipping this. But it is moving fast, mostly to provoke discussion.
I've been hoping for a couple weeks now that there would be a definition of the various objects which we're going to manipulate. An ontology, a glossary, and a bit of a structure. I can guess at how to do it, and if our problem space is already specified then I can derive it, but I'd prefer to have it mostly defined by consensus.
I don't particularly care about which language we're using. I think that FastAPI is a prototyping library which mainly serves to orient the API around OpenAPI's semantics from the start, emphasizing documentation and clarity of relationships. I don't think that it matters what the second version is implemented in, although I'll always point out that Python can go fast enough -- with PyPy -- to provide a decently-scaled frontend. For me, Rust is not about speed, but about memory safety, and so Python is not a worse choice.
Why cloud-native? I'm trying to imagine the boundary between us and the customer. It felt like a cloud-native approach would be the better choice since we want to transparently use OCI when possible. I genuinely don't know how you envisioned the inner structure of a GPU host or a frontend load-balancer, but I imagined them as two distinct responsibilities. In particular, I want the design constraint that upgrading the frontend won't disturb ongoing inference.
I don't care about directory structure, other than to note that Docker-style container-building tools expect everything to be in a single "build context" directory for security. Similarly, I don't care about URLs other than the standard expectations around versioning and semantic accessibility. I'll change the URLs next time I hack on this branch.
rstarmer
sdake
This is a very basic Friday-afternoon project. Where we go from here is very dependent on overall system architecture, which we haven't discussed too much.