Using Granite Code models in VS Code with Continue plugin

[adampingel]

See Gabe Goodhart's blog post

https://developer.ibm.com/tutorials/awb-local-ai-copilot-ibm-granite-code-ollama-continue/

Some additional resources:

Continue

• https://docs.continue.dev/quickstart - Installing in VSCode
• https://marketplace.visualstudio.com/items?itemName=Continue.continue

[fayvor]

@adampingel how would I get access to this link? https://github.ibm.com/ghart/writing/blob/main/blog-posts/granite-continue-copilot.md

It's asking me to sign on with w3id: https://login.w3.ibm.com/idaas/mtfim/sps/idaas/login?runtime=true&Target=https%3A%2F%2Flogin.w3.ibm.com%2Fsaml%2Fsps%2Fauth

[hickeyma]

@fayvor The link in the description was incorrect. I have updated it now to: https://developer.ibm.com/tutorials/awb-local-ai-copilot-ibm-granite-code-ollama-continue/

[deanwampler]

It has also been republished in two parts here:

• https://allthingsopen.org/articles/build-a-local-ai-co-pilot
• https://allthingsopen.org/articles/extend-local-ai-co-pilot-framework

[fayvor]

Thanks guys!

[fayvor]

Looks like I need additional permissions to claim or change the status of this ticket, but pls consider it claimed.

[fayvor]

I've put up a draft PR here for feedback.

[fayvor]

PR up for review; doesn't look like I can assign it.

[deanwampler]

I'll review it shortly...

[fayvor]

PR has been merged.

[adampingel]

Tagging @gabe-l-hart for awareness

[gabe-l-hart]

Thanks for getting this all together @fayvor! The one comment I have on the notebook in the PR is that it doesn't address the code-completion model. I've found that the 3b 128k context model (which I'm in the process of getting up on ollama) has very solid auto-completion performance.

[fayvor]

Hi @gabe-l-hart! Thanks for pointing that out. I'll try it with ollama. Here is a ticket for that.

We'll also want to provide Continue configs to point these at remote models on Replicate (once they are kept hot) and WatsonX.

[gabe-l-hart]

Ah, yeah, that will be really nice to show how the same models can be used running elsewhere. Do we have a plan for usage limiting in Replicate, or is it more of "if you deploy it in Replicate, here's how you can configure Continue?"

[fayvor]

We have a plan for hosting those models publicly on Replicate.

When you say 3b 128k, what does the 128k describe? And what is the 3b model that is in the ollama library?

[fayvor]

Claude says 128k is the context window :) What is the default context window? 1024 tokens?

[gabe-l-hart]

🤦 Yep, it's a terrible way to version them. You're correct that 128k is the context length. The models currently on ollama have 2k (3b), 4k (8b), and 8k (20b) context lengths which can be very limiting as a code assistant. The new models vastly extend this to support much larger context windows.