Open sundaraa-deshaw opened 10 months ago
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
This behavior is observed when using the Jupyternaut chat.. The prompt template is honored when using the magic %%ai in a cell. However I find it inconsistent that it does not use the prompt template when used from the chat,
Yes, as of now the prompt templates apply to the magic only. There are two issues tracking customisation of prompts for chat (although it is not obvious from the titles):
I think ultimately per-model prompts for inline completions (https://github.com/jupyterlab/jupyter-ai/pull/465) would come in handy too; these will need separate prompt for code and text/markdown generation.
Hi @sundaraa-deshaw, I am also currently trying to setup connection between jupyter-ai and a local LLM. In my case the LlaMA2 sits on a local GPU server and the jupyter-ai is setup on a different development server. Please can you let me know if you have tried to build a similar setup and got some leads.
Hi @sundaraa-deshaw, I am also currently trying to setup connection between jupyter-ai and a local LLM. In my case the LlaMA2 sits on a local GPU server and the jupyter-ai is setup on a different development server. Please can you let me know if you have tried to build a similar setup and got some leads.
Hi, I did something similar previously. i.e. run the llama engine on a GPU and expose the inference/chat completion as a server (you get this for free with llama.cpp/server) and then implement a local LLM provider that hits the completion endpoint. This worked for me as a PoC
Description
I am using the jupyter ai extension with a custom model provider as per steps in https://jupyter-ai.readthedocs.io/en/latest/users/index.html#custom-model-providers
However the custom prompt template is not being used.
Reproduce
write code to transpose a numpy matrix
Generated prompt:
Human: write code to transpose a numpy matrix AI:
"Produce output as source code only, with no text or explanation before or after it. Produce the output in Markdown format