Support Local LLMs & custom API-Endpoints

chymian commented 11 months ago

I just found out about that impressive work, which has be done here.

Is there a plan to support Local LLMs and custom API-Points or something like openrouter.ai or even better litellm.ai, while the latter is a API-Proxy/loadblancer/router/cache?

kreneskyp commented 11 months ago

local LLMs

Two options for local models are available:

ollama
llama-cpp

OpenAI Proxy

Open AI proxies also work by setting the proxy url and/or api base.

Routers

openrouter.ai: not familiar with this one so I would need to research it more
litellm.ai: supported by Langchain so it should be easy to add a component config to integrate.

chymian commented 11 months ago

@kreneskyp

ah, all already there.

so this server-tab is in the settings, I assume? settings it never opens for me.

when I try to add a secret, I see in the nginx logs:

ix-nginx-1   | 10.11.11.1 - - [05/Dec/2023:11:20:22 +0000] "GET /api/secrets/?limit=90000&offset=0 HTTP/1.1" 200 99 "http://pai:8000/chat/a700887b-514b-4213-a2c6-986040d61b31" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"

when I click settings: nothing happens in the GUI, nothing in the log.

about litellm: it is a library to simplify LLM completion and emebbings calls and gives access to the rest of the world of LLMs/APIs.

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

chymian commented 11 months ago

the browser console reports:

00887b-514b-4213-a2c6-986040d61b31:1 The Cross-Origin-Opener-Policy header has been ignored, because the URL's origin was untrustworthy. It was defined either in the final response or a redirect. Please deliver the response using the HTTPS protocol. You can also use the 'localhost' origin instead. See https://www.w3.org/TR/powerful-features/#potentially-trustworthy-origin and https://html.spec.whatwg.org/#the-cross-origin-opener-policy-header.
react-dom.development.js:29840 Download the React DevTools for a better development experience: https://reactjs.org/link/react-devtools
react-dom.development.js:86 Warning: validateDOMNesting(...): <tbody> cannot appear as a child of <div>.
    at tbody
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/table/dist/chunk-J4QO5GAJ.mjs:16:85)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at table
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/table/dist/chunk-GEJVU65N.mjs:25:88)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at SecretsTable (webpack-internal:///./frontend/secrets/SecretsTable.js:30:19)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at LeftSidebarPopupContent (webpack-internal:///./frontend/site/LeftMenuPopover.js:31:24)
    at section
    at MotionComponent (webpack-internal:///./node_modules/framer-motion/dist/es/motion/index.mjs:49:65)
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at PopoverTransition2 (webpack-internal:///./node_modules/@chakra-ui/popover/dist/chunk-F4GPNG57.mjs:53:11)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at PopoverContent2 (webpack-internal:///./node_modules/@chakra-ui/popover/dist/chunk-24I2HV4N.mjs:21:13)
    at Popover (webpack-internal:///./node_modules/@chakra-ui/popover/dist/chunk-UUZWUER5.mjs:21:88)
    at LeftMenuPopover (webpack-internal:///./frontend/site/LeftMenuPopover.js:35:24)
    at SecretsMenuItem (webpack-internal:///./frontend/secrets/SecretsMenuItem.js:27:21)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/layout/dist/chunk-ZHMYA64R.mjs:25:5)
    at nav
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at Navigation (webpack-internal:///./frontend/site/Navigation.js:26:90)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/layout/dist/chunk-ZHMYA64R.mjs:25:5)
    at VStack
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at Flex2 (webpack-internal:///./node_modules/@chakra-ui/layout/dist/chunk-KRPLQIP4.mjs:14:11)
    at SidebarProvider (webpack-internal:///./frontend/site/sidebar/context.js:45:25)
    at Layout (webpack-internal:///./frontend/site/Layout.js:38:24)
    at ChatInputProvider (webpack-internal:///./frontend/chat/input/ChatInputProvider.js:45:21)
    at ChatViewProvider (webpack-internal:///./frontend/chat/ChatView.js:116:24)
    at ChatView (webpack-internal:///./frontend/chat/ChatView.js:120:80)
    at RenderedRoute (webpack-internal:///./node_modules/react-router/dist/index.js:554:5)
    at RenderErrorBoundary (webpack-internal:///./node_modules/react-router/dist/index.js:501:5)
    at DataRoutes (webpack-internal:///./node_modules/react-router/dist/index.js:1011:5)
    at Router (webpack-internal:///./node_modules/react-router/dist/index.js:1123:15)
    at RouterProvider (webpack-internal:///./node_modules/react-router/dist/index.js:951:5)
    at EnvironmentProvider (webpack-internal:///./node_modules/@chakra-ui/react-env/dist/chunk-VMD3UMGK.mjs:34:11)
    at ColorModeProvider (webpack-internal:///./node_modules/@chakra-ui/color-mode/dist/chunk-AMBGAKG2.mjs:29:5)
    at ThemeProvider (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:123:50)
    at ThemeProvider (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-UBPTU7WB.mjs:30:11)
    at ChakraProvider (webpack-internal:///./node_modules/@chakra-ui/provider/dist/chunk-3DDHO3UN.mjs:21:5)
    at ChakraProvider2 (webpack-internal:///./node_modules/@chakra-ui/react/dist/chunk-QAITB7GG.mjs:19:5)
    at RelayEnvironmentProvider (webpack-internal:///./node_modules/react-relay/lib/relay-hooks/RelayEnvironmentProvider.js:18:24)
    at App (webpack-internal:///./frontend/App.js:22:21)
printWarning @ react-dom.development.js:86
error @ react-dom.development.js:60
validateDOMNesting @ react-dom.development.js:10847
createInstance @ react-dom.development.js:10930
completeWork @ react-dom.development.js:22187
completeUnitOfWork @ react-dom.development.js:26593
performUnitOfWork @ react-dom.development.js:26568
workLoopSync @ react-dom.development.js:26466
renderRootSync @ react-dom.development.js:26434
performConcurrentWorkOnRoot @ react-dom.development.js:25738
workLoop @ scheduler.development.js:266
flushWork @ scheduler.development.js:239
performWorkUntilDeadline @ scheduler.development.js:533
react-dom.development.js:86 Warning: validateDOMNesting(...): <div> cannot appear as a child of <table>.
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at table
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/table/dist/chunk-GEJVU65N.mjs:25:88)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at SecretsTable (webpack-internal:///./frontend/secrets/SecretsTable.js:30:19)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at LeftSidebarPopupContent (webpack-internal:///./frontend/site/LeftMenuPopover.js:31:24)
    at section
    at MotionComponent (webpack-internal:///./node_modules/framer-motion/dist/es/motion/index.mjs:49:65)
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at PopoverTransition2 (webpack-internal:///./node_modules/@chakra-ui/popover/dist/chunk-F4GPNG57.mjs:53:11)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at PopoverContent2 (webpack-internal:///./node_modules/@chakra-ui/popover/dist/chunk-24I2HV4N.mjs:21:13)
    at Popover (webpack-internal:///./node_modules/@chakra-ui/popover/dist/chunk-UUZWUER5.mjs:21:88)
    at LeftMenuPopover (webpack-internal:///./frontend/site/LeftMenuPopover.js:35:24)
    at SecretsMenuItem (webpack-internal:///./frontend/secrets/SecretsMenuItem.js:27:21)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/layout/dist/chunk-ZHMYA64R.mjs:25:5)
    at nav
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at Navigation (webpack-internal:///./frontend/site/Navigation.js:26:90)
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at eval (webpack-internal:///./node_modules/@chakra-ui/layout/dist/chunk-ZHMYA64R.mjs:25:5)
    at VStack
    at div
    at eval (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:60:66)
    at ChakraComponent (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-5PL47M24.mjs:51:102)
    at Flex2 (webpack-internal:///./node_modules/@chakra-ui/layout/dist/chunk-KRPLQIP4.mjs:14:11)
    at SidebarProvider (webpack-internal:///./frontend/site/sidebar/context.js:45:25)
    at Layout (webpack-internal:///./frontend/site/Layout.js:38:24)
    at ChatInputProvider (webpack-internal:///./frontend/chat/input/ChatInputProvider.js:45:21)
    at ChatViewProvider (webpack-internal:///./frontend/chat/ChatView.js:116:24)
    at ChatView (webpack-internal:///./frontend/chat/ChatView.js:120:80)
    at RenderedRoute (webpack-internal:///./node_modules/react-router/dist/index.js:554:5)
    at RenderErrorBoundary (webpack-internal:///./node_modules/react-router/dist/index.js:501:5)
    at DataRoutes (webpack-internal:///./node_modules/react-router/dist/index.js:1011:5)
    at Router (webpack-internal:///./node_modules/react-router/dist/index.js:1123:15)
    at RouterProvider (webpack-internal:///./node_modules/react-router/dist/index.js:951:5)
    at EnvironmentProvider (webpack-internal:///./node_modules/@chakra-ui/react-env/dist/chunk-VMD3UMGK.mjs:34:11)
    at ColorModeProvider (webpack-internal:///./node_modules/@chakra-ui/color-mode/dist/chunk-AMBGAKG2.mjs:29:5)
    at ThemeProvider (webpack-internal:///./node_modules/@emotion/react/dist/emotion-element-c39617d8.browser.esm.js:123:50)
    at ThemeProvider (webpack-internal:///./node_modules/@chakra-ui/system/dist/chunk-UBPTU7WB.mjs:30:11)
    at ChakraProvider (webpack-internal:///./node_modules/@chakra-ui/provider/dist/chunk-3DDHO3UN.mjs:21:5)
    at ChakraProvider2 (webpack-internal:///./node_modules/@chakra-ui/react/dist/chunk-QAITB7GG.mjs:19:5)
    at RelayEnvironmentProvider (webpack-internal:///./node_modules/react-relay/lib/relay-hooks/RelayEnvironmentProvider.js:18:24)
    at App (webpack-internal:///./frontend/App.js:22:21)
printWarning @ react-dom.development.js:86
error @ react-dom.development.js:60
validateDOMNesting @ react-dom.development.js:10847
createInstance @ react-dom.development.js:10930
completeWork @ react-dom.development.js:22187
completeUnitOfWork @ react-dom.development.js:26593
performUnitOfWork @ react-dom.development.js:26568
workLoopSync @ react-dom.development.js:26466
renderRootSync @ react-dom.development.js:26434
performConcurrentWorkOnRoot @ react-dom.development.js:25738
workLoop @ scheduler.development.js:266
flushWork @ scheduler.development.js:239
performWorkUntilDeadline @ scheduler.development.js:533

the ix-docker runs in a LXC on my laptop.

kreneskyp commented 11 months ago

ah, all already there.

so this server-tab is in the settings, I assume? settings it never opens for me.

It's set individually for each LLM node.

Drag an OpenAI node into the chain
Click on it
Settings will open in the right side bar

kreneskyp / ix