feat: support a local custom cortex engine

RyanMetcalfeInt8 commented 1 month ago

Hello, I have a custom cortex engine (compiled into an engine.dll). Right now I am able to make use of it in a super hacky way, by replacing the default cortex.llamacpp engine.dll with my custom version. But obviously I don't want to have to do that..

So, I am wondering -- What the correct way to make use of my custom engine? I see in model.json that there is an 'engine' field, which is typically set to 'nitro'. My confusion here is that 'nitro' seems equivalent to 'cortex' (which I also use), but I don't see any field that can specify which specific engine to use (cortex.llamacpp vs. mine).

It's very possible that I missed / overlooked something..

Perhaps there is also some feature needed in cortex.cpp layer as well?

Thanks! Ryan

louis-jan commented 1 month ago

Hi @RyanMetcalfeInt8, that's awesome! You're right, existing models have defined nitro as the engine, but the extension will actually route to the cortex.cpp server underneath, it would run on the llama.cpp engine (version 0.5.5). We're working on multiple engines support, which is tracked here. #3690

RyanMetcalfeInt8 commented 1 month ago

Hey @louis-jan -- Thanks for getting back to me so quickly! Makes sense.. I'll subscribe to that issue.. and happy to help test new features as they become available. Cheers!

louis-jan commented 1 month ago

Hey @RyanMetcalfeInt8, I think this is a valid issue to track for improving the extension dev experience. I encountered the same issue while working on MLX support. I will reopen it for tracking.

0xSage commented 1 month ago

related #3773

imtuyethan commented 1 week ago

janhq / jan

feat: support a local custom cortex engine #3764