Restructure Web Server

[josephjclark]

I would like to restructure the gen project so that there is a single web server which hosts endpoints to each of the code generation services.

I'd also like to see a nice clean separation of concerns between "web server stuff" and "AI generation stuff" within the repo. I should be able to quickly drill down to the code generation functions without being distracted by web server code.

After this refactor, there should only be a single run.sh required to run the webserver and able all endpoints to work. This should probably live somewhere at the top level

Folder structure

I think we want a folder structure like this:

gen
├── codegen
│   ├── README.md
│   ├── src
│   │   └── main.py
│   └── test
├── inference
│   ├── README.md
│   ├── src
│   │   └── main.py
│   └── test
├── openfn_llama
│   ├── README.md
│   ├── src
│   │   └── main.py
│   └── test
├── siggen
│   ├── README.md
│   ├── src
│   │   └── main.py
│   └── test
└── web
    ├── README.md
    ├── src
    │   ├── main.py
    │   └── router.py
    └── test

Ie, the webserver is its own folder, with all the AI modules as siblings.

The top folder names could be changed (signature_generatorvs siggen vs generators/signature). I'd prefer idiomatic python to "stuff Joe likes" - this is supposed to make the project more readable and manageable.

The router.py should call out to the other modules and pass in the correct inputs. It should be very lightweight glue code.

I don't have a string opinion on whether we have a single router which calls out to the services, or if each endpoint has its own little router file.

Endpoints

The webserver should expose endpoints like this:

/generate/code
/generate/signature
/generate/test

It should be easy for devs to contribute new endpoints.

Inference

The inference engine should not be an endpoint of its own. It should be a python module which is invoked by the other endpoints

Benefits

As well as this being a cleaner architecture generally (IMO anyway), having a single web server makes life much easier in @openfn/CLI, because we can target one single gen server, rather than having to take a whole URL for each service.

Documentation

The web server should have a little readme to explain some fundamentals.

The main readme should explain this architecture and help devs understand how to contribute new endpoints.

[josephjclark]

Closing this down in favour of #42 - which just formalises this concept into a fuller design