search

EinStack / glide

🐦 A open blazing-fast simple model gateway for rapid development of production GenAI apps
https://docs.einstack.ai/glide/
Apache License 2.0
133 stars 20 forks source link

🔧 Make it possible to run Glide with no providers #227

Open gernest opened 7 months ago

gernest commented 7 months ago

I wanted to work on open telemetry tickets, as a first time contributor I faced some minor challenges (nothing major but things that can probably be improved)

Possible solution to this is to make routers configuration optional, If it is not provided glide should be able to start but without any providers loaded, this will also pave way for adding providers or removing them on demand via api.

I would still love to contribute, I currently can't afford these third party providers api's so I was hopping to help with everything else.

roma-glushko commented 7 months ago

Hey @gernest Geoffrey 👋

requiring open api to start : As I mentioned I intended to work on setting up open telemetry for traces and metrics. I was surprised there was no way for me to just start the binary without and provider and then setup/teardown otel providers and have tests to verify it is working.

I was thinking about that in scope of end-to-end testing of the gateway. There is no way real providers are used there because of the cost implications. So I thought of:

Out of these three, I like your idea more because it's simple and as you pointed out it will pave out way to the database-on setup (Glide is databaseless at this moment but there is not way we can resist sharing common state between Glide nodes for a long time. I think Kong has two modes too. Not sure about Tyk).

why module glide instead of module github.com/EinStack/glide ?

Haha good question! I think this is out of ignorance. I have seen ppl calling their libs after git repos but would there be any benefits doing that for something like Glide that is an application not supposed to be exposed to anyone via imports?

I would still love to contribute, I currently can't afford these third party providers api's so I was hopping to help with everything else.

I can try to share keys with you if that helps. Just give me a sign what else would be interesting to touch that would require access to providers (I don't have access to absolutely all of them yet but we will figure that out) 👀

Speaking about telemetry, I could not even dream of someone else helping me with (thought I would be integrating alone), so I made a bunch of placeholder tickets "linked" to the main one: https://github.com/EinStack/glide/issues/19

Will try to go over them tom morning my time to fill them with some useful information.

gernest commented 7 months ago

Haha good question! I think this is out of ignorance. I have seen ppl calling their libs after git repos but would there be any benefits doing that for something like Glide that is an application not supposed to be exposed to anyone via imports?

Not necessary , go install github.com/EinStack/glide@latest can be used to install latest binary without the need for cloning and building.

I'm interested on those otel tickets.

Are going to handle making the router optional, or do you want help with it ? I'm fine either way, but there is nothing much I can do for now until this is addressed.

gernest commented 7 months ago

random question, what is the story behind all these emoji ? I don't remember opening my ticket with emoji, Its hard to take a project seriously as I can't really understand what they mean.

roma-glushko commented 7 months ago

Are going to handle making the router optional, or do you want help with it ? I'm fine either way, but there is nothing much I can do for now until this is addressed.

@gernest yeah, I can leave it on you Geoffrey. In any case, we have a super simple workaround to keep you going without the no router setup thing in place. You can simply put invalid API key into the config like:

telemetry:
  logging:
    level: debug  # debug, info, warn, error, fatal
    encoding: console

routers:
  language:
    - id: default
      models:
        - id: openai
          openai:
            api_key: "geoffreySuperSecretKey"

That should make the trick. Although, this setup won't help us to do telemetry tickets that are connected to instrumentation of the router workflows.

Not necessary , go install github.com/EinStack/glide@latest can be used to install latest binary without the need for cloning and building.

Alright, yeah, we may want to have that. Binaries can be also installed via HomeBrew/Spancraft/Scoop(soon), so I'm too worry about missing that. But again, yeah that would nice to have. Moving that idea to a dedicated ticket (https://github.com/EinStack/glide/issues/228).

random question, what is the story behind all these emoji ? I don't remember opening my ticket with emoji, Its hard to take a project seriously as I can't really understand what they mean.

It's kind of nice to have these visual differentiations in Github Issue list and Git history, so we are using emojiis for that 😃 Here is a full list of emojiis with their meaning https://github.com/roma-glushko/gitmoji

This also helps a bit during changelog collection (as it's not yet automated).

gernest commented 7 months ago

That should make the trick. Although, this setup won't help us to do telemetry tickets that are connected to instrumentation of the router workflows.

Good observation, I think your suggestion of having a placeholder provider for testing makes more sense. I will see what I can do with this.