Should we rewrite in Typescript or Elixir?

[josephjclark]

The gen project is currently written in Python. There are four layers of the project:

• Http services
• Application-specific services for different tasks - basically a pre-written prompt for eg code generation
• The core inference engine which calls up to the models
• The fined tuned llama and gpt models

The thing is, looking at the codebase, I think the only thing that has to be python is the fine tuner.

The first three layers are really just http layers, serving and calling hosted services. There's no benefit to those that I can see in python. I'm sure wrapper libraries are available for elixir and node but even if not, we could create a http wrapper.

Given that we're not a python house, are we wrong to maintain this python code base?

This is an issue to discuss the, uh, issue.

Python Advantages

• Python is pretty much the home language of ML developers and hugging face. It's where contributors will want to be working
• We're more likely to attract open-source contributions if we have a python codebase
• Fine tuning in particular basically has to be python. But this is an offline step - the fine tuner could (and probably should) be separate repo

TS/Elixir Advantages

• We don't need ML engineers to write the public web interfaces. Better that we own, configure and deploy those ourselves in languages we're comfortable with
• AI services in TS and Elixir could be called directly within lightning. Of course they're still going to call out to a server to do the hearby lifting so maybe this isn't a big benefit.
• If we decide to move the metadata and docs services into the same gen server, well those are Javascript based already, so it would make a lot of sense to wrap them into a node server

Solutions?

The obvious answer is that the web layer should node or elixir and that the AI layer should be python. But of course you have to get those layers to speak to each other and we have no mechanism for that. HTTP I guess.

If the http layer is sufficiently light then I don't mind it being in python, which biases the repo toward ML engineers, which is good! We may have to do some API key processing but perhaps we could offload that out of pythonland.

[josephjclark]

I'm convinced that the answer is here is: keep the python ML stuff, but rewrite all the server stuff in typescript

[josephjclark]

See #41: we're gonna have a Typescript web server but leave the AI stuff in python.