BOUNTY: Convert the endpoints to a smart router like litellm does in python

[sandys]

this bounty refers to the JS SDK only http://github.com/arakoodev/EdgeChains/tree/ts/JS/edgechains

we are super inspired by Litellm in python (and have a lot of respect for them) - https://github.com/BerriAI/litellm and we want to create a similar infrastructure in Edgechain.js using Typescript. We are hoping libraries like axios makes this easier.

the four features we want are these:

1. load balance between multiple deployments. It picks the deployment which is below rate-limit and has the least amount of tokens used. (only openai, google palm and cohere needed). this includes basic reliability features (https://docs.litellm.ai/docs/routing#basic-reliability) like timeouts/retries/etc. you can use axios.interceptors.response.use to achieve these.
2. streaming capability
3. token usage - https://docs.litellm.ai/docs/completion/token_usage
4. logging - https://docs.litellm.ai/docs/observability/callbacks (only sentry and posthog needed)

Once created, you will replace the Endpoint class with this new infrastructure (https://github.com/arakoodev/EdgeChains/blob/ts/JS/edgechains/lib/src/lib/endpoints/OpenAiEndpoint.ts)

project will be deemed complete when:

1. you add testcases for all the features above.
2. mock the openai, palm and cohere endpoints for testing. (you can take inspiration from here https://github.com/polly3d/mockai)
3. you have deleted the current endpoint classes and replaced it with your new classes in all examples and testcases.

NOTE: the core of edgechains is the usage of jsonnet (https://jsonnet.org/) configuration to manage all kinds of config - like prompts, etc. we will prefer that as far as possible, we use jsonnet to configure this routing.

[sandys]

/bounty 200

[algora-pbc[bot]]

💎 $200 bounty • Arakoo.ai

Steps to solve:

1. Start working: Comment /attempt #286 with your implementation plan
2. Submit work: Create a pull request including /claim #286 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to arakoodev/EdgeChains!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🔴 @rajdip-b	Dec 2, 2023, 12:50:05 PM	WIP

[rajdip-b]

/attempt #286

Options

• Cancel my attempt

[rajdip-b]

@sandys can i be assigned this issue?

[sandys]

@rajdip-b happy to. but first want to check if you understand what has to be achieved ? any doubts, etc ?

[guhitb]

@rajdip-b Any interest in collaborating on this?

[rajdip-b]

@sandys As per my understanding, the main challenge is creating the load balancer in the endpoints of OpenAiEndpoint.ts. I was going through the references you mentioned and have a clarity over what needs to be implemented. I would have to do some further digging about the how's of the implementation, but yes, I feel I can definitely give it a shot!

And not to mention, as and when issues come up, I will ping you about them.

[sandys]

@rajdip-b there are 4 capabilities needed (as mentioned in my original post). it is important that you look at all four, because the architecture will need to be inclusive. also, this architecture will REPLACE the current endpoint structure

[rajdip-b]

@guhitb, are you familiar with the project structure/implementation details? I would be happy to collaborate!

[rajdip-b]

@rajdip-b there are 4 capabilities needed (as mentioned in my original post). it is important that you look at all four, because the architecture will need to be inclusive. also, this architecture will REPLACE the current endpoint structure

Yeah, of course, I understand that. For me, I feel the challenge would be implementing load balancing, hence mentioned that. As for the rest, I feel they can be done with lesser efforts than the former.

[sandys]

very well. assigned to @rajdip-b best wishes. please feel free to drop in to discord if u want to discuss anything quickly. https://discord.gg/Avsbume2

[rajdip-b]

very well. assigned to @rajdip-b best wishes. please feel free to drop in to discord if u want to discuss anything quickly. https://discord.gg/Avsbume2

Discord is my go-to for discussion! Hopping in right now!

[guhitb]

@rajdip-b Not yet, aside from browsing through the repo today. I don't have much experience with open source, but I'm sure I can help with mocks, logging, tests, etc. The load balancing part feels a little daunting though.

[rajdip-b]

@rajdip-b Not yet, aside from browsing through the repo today. I don't have much experience with open source, but I'm sure I can help with mocks, logging, tests, etc. The load balancing part feels a little daunting though.

Hey sure thing! We can pair up!

Drop me a text on discord! _the_heisenberg

[rajdip-b]

/split @guhitb

[sandys]

Hi rajdip We are not totally in favor of splitting the bounty right now. No offense, but we are looking to reward the person who can complete the work with the best architecture to benefit end users.

We realise that there is a spirit of collaboration at play - we request you do this offline within yourself if you want.

We are requesting for this option since if yourSelf you cannot pull this off...we would like to reserve the right to assign this to someone else entirely.

At this time, we believe in YOUR abilities and we would like for the reward to reflect that

Case in point - there was this massive crash and burn on algora with the ziglang folks cos of this. And we would like to avoid a situation like that. So mentioning up front !

On Sat, Dec 2, 2023, 20:33 Rajdip Bhattacharya @.***> wrote:

/split @guhitb https://github.com/guhitb

— Reply to this email directly, view it on GitHub https://github.com/arakoodev/EdgeChains/issues/286#issuecomment-1837168552, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYUY2VCAQN47PNWAZWGTYHM7NVAVCNFSM6AAAAABADW6S3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGE3DQNJVGI . You are receiving this because you were mentioned.Message ID: @.***>

[rajdip-b]

@sandys Ah, that's bad. Sure, I'll take this issue op alone. @guhitb You can still connect with me in case you want to collab on any other issue! Hope that's fine by everyone :D

[rajdip-b]

Hey @sandys, I have started with the docs today. I have went through the code base of litellm and how they implement their stuff. I also went though openai docs. I have some questions and suggestions that I would like to share and discuss with you. I couldn't find you in discord in you channel. mind pinging you handle in here? ill open a thread in discord and tag you in it!

[sandys]

just create a new thread in the "support" channel (and mention the bug number). the right people will get back to you

On Mon, Dec 4, 2023 at 12:14 AM Rajdip Bhattacharya < @.***> wrote:

Hey @sandys https://github.com/sandys, I have started with the docs today. I have went through the code base of litellm and how they implement their stuff. I also went though openai docs. I have some questions and suggestions that I would like to share and discuss with you. I couldn't find you in discord in you channel. mind pinging you handle in here? ill open a thread in discord and tag you in it!

— Reply to this email directly, view it on GitHub https://github.com/arakoodev/EdgeChains/issues/286#issuecomment-1837565012, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYU3MZ4YDKNDAOLNWMNTYHTCCFAVCNFSM6AAAAABADW6S3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZXGU3DKMBRGI . You are receiving this because you were mentioned.Message ID: @.***>

[rajdip-b]

Created! @sandys

[Sajiyah-Salat]

Hello @rajdip-b are you still working on this. Let me know your discord handle. I would like to connect with you there

[rajdip-b]

Hey @Sajiyah-Salat, there has been a blocker, and hence I have paused my work currently.

[Sajiyah-Salat]

Okay. Thanks for the reply

[Sambit003]

@sandys @Sajiyah-Salat can I work on this?

[rajdip-b]

@sandys I'm dropping my attempt on this issue. For anyone who attempts next, feel free to pick up from my PR.

[snigdha510]

can I work on this?

[sandys]

hi snigdha thanks for your interest. can u please join discord and message please ? we will be able to assign this (or other projects) if you are interested https://discord.gg/cby9pQKT

On Fri, May 17, 2024 at 12:04 PM Snigdha Parashar @.***> wrote:

can I work on this?

— Reply to this email directly, view it on GitHub https://github.com/arakoodev/EdgeChains/issues/286#issuecomment-2116851122, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYU7NWXSWJDDGRYUBFTDZCWQGZAVCNFSM6AAAAABADW6S3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJWHA2TCMJSGI . You are receiving this because you were mentioned.Message ID: @.***>

[Sadaf-A]

/attempt https://github.com/arakoodev/EdgeChains/issues/286

[Sadaf-A]

Hey @sandys ! Is this up for contribution?