Using Anthropic API renders some issues

[dirksierd]

Thanks for this project, I'd love to use this with helix.

Doing some attempts I ran into two issues, the first I was able to track down.

2024-06-08T21:55:06.687 helix_lsp::transport [ERROR] lsp-ai <- InternalError: "{\"message\":\"messages.0.tool_calls: Extra inputs are not permitted\",\"type\":\"invalid_request_error\"}"

I made a fork and removed both references to tool_calls in config.rs; this fixed that issue. I'm no Rustacean, so not sure how to implement that nicely, but I hope it points you in some useful direction.

---

The second issue I'm not sure how to tackle… I get results from the Anthropic-API, but they are cut off at (I believe…) about 155 tokens.

For example, if I enter

const monthNames = <CURSOR>

…it returns a (cut-off) list of monthNames.

The following, however, works fine.

const monthNamesAbbreviated = <CURSOR>

[SilasMarvin]

Thanks for checking it out! I think we needed that tool call reference for Mistral FIM, I'm not 100% sure I will double check it tomorrow and get back to you here with a fix.

Can you share your config for Anthropic? You probably need to increase the max_tokens parameter.

It's also worth noting that these LLMs are not deterministic and do have some strange behaviour. We may also need to adjust the prompt you are using.

[SilasMarvin]

Fixed the Anthropic tool errors, thank you for catching those! https://github.com/SilasMarvin/lsp-ai/pull/7

[dirksierd]

Ah, you're right about the max_tokens… I wasn't sure as I followed the recommended config. Could've tried though.

Anyway… it's now at 4096 and this fixes the aforementioned issue.

[dirksierd]

Hm… I suppose not.

When I started helix with the old setting (at 2048 max_tokens) it didn't work. I then changed the config to 4096 and did a :config-reload and a :lsp-restart

Then it worked.

Now after restarting the editor it returned the cropped results again and I can't reproduce the above steps.

[SilasMarvin]

Can you share your config?

[dirksierd]

[language-server.lsp-ai]
command = "lsp-ai"
args = ["--stdio"]

[language-server.lsp-ai.config.memory]
file_store = {}

[language-server.lsp-ai.config.models.model1]
type = "anthropic"
chat_endpoint = "https://api.anthropic.com/v1/messages"
model = "claude-3-sonnet-20240229"
auth_token_env_var_name = "ANTHROPIC_API_KEY"

[language-server.lsp-ai.config.completion]
model = "model1"

[language-server.lsp-ai.config.completion.parameters]
max_context = 2048
max_new_tokens = 128
system = "Instructions:\n- You are an AI programming assistant.\n- Given a piece of code with the cursor location marked by \"<CURSOR>\", replace \"<CURSOR>\" with the correct code or comment.\n- First, think step-by-step.\n- Describe your plan for what to build in pseudocode, written out in great detail.\n- Then output the code replacing the \"<CURSOR>\"\n- Ensure that your completion fits within the language context of the provided code snippet (e.g., Python, JavaScript, Rust).\n\nRules:\n- Only respond with code or comments.\n- Only replace \"<CURSOR>\"; do not include any previously written code.\n- Never include \"<CURSOR>\" in your response\n- If the cursor is within a comment, complete the comment meaningfully.\n- Handle ambiguous cases by providing the most contextually appropriate completion.\n- Be consistent with your responses."

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "user"
content = "def greet(name):\n    print(f\"Hello, {<CURSOR>}\")"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "assistant"
content = "name"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "user"
content = "function sum(a, b) {\n    return a + <CURSOR>;\n}"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "assistant"
content = "b"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "user"
content = "fn multiply(a: i32, b: i32) -> i32 {\n    a * <CURSOR>\n}"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "assistant"
content = "b"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "user"
content = "# <CURSOR>\ndef add(a, b):\n    return a + b"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "assistant"
content = "Adds two numbers"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "user"
content = "# This function checks if a number is even\n<CURSOR>"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "assistant"
content = "def is_even(n):\n    return n % 2 == 0"

[[language-server.lsp-ai.config.completion.parameters.messages]]
role = "user"
content = "{CODE}"

[language-server.tailwindcss-vue]
command = "tailwindcss-language-server"
args = ["--stdio"]
language-id = "vue"
timeout = 2

[language-server.typescript-language-server.config]
plugins = [
  { name = "@vue/typescript-plugin", location = "node_modules/@vue/language-server", languages = ["vue"] }
]

[language-server.vue-language-server]
command = "vue-language-server"
args = ["--stdio"]
config = { vue = { hybridMode = true }, typescript = { tsdk = "node_modules/typescript/lib" } }

[[language]]
name = "vue"
auto-format = true
formatter = { command = "prettier", args = ["--parser", "vue"] }
language-servers = ["typescript-language-server", "lsp-ai", "vue-language-server", "tailwindcss-vue"]
indent = { tab-width = 2, unit = "  " }

[[language]]
name = "typescript"
auto-format = true
formatter = { command = "prettier", args = ["--parser", "typescript"] }
language-servers = ["typescript-language-server"]

[[language]]
name = "elixir"
auto-format = true
language-servers = ["elixir-ls"]

[SilasMarvin]

Your max_new_tokens param should just be max_tokens. That should fix your problem! By default we set a max_tokens of 64, so right now with your current configuration, you are only able to generate up to 64 tokens.

Did you get this config from the docs somewhere? If so, I need to go update it, I apologize for the bad documentation.

[dirksierd]

That did it, thanks!

I used this config to start out: https://github.com/SilasMarvin/lsp-ai/wiki/Configuration#chat-4 – it uses max_new_tokens.

[SilasMarvin]

Ah got it! Sorry about that, just updated the docs!