Function calling (using food-search.ts)

[beddows]

I'm submitting a ... [ x] bug report

Summary I'm getting errors with:

Anthropic

``` @ax-llm/ax/build/module/src/util/apicall.js:39 throw new Error(`API Error: ${apiUrl.href}, ${e}`); ^ Error: API Error: https://api.anthropic.com/v1/messages, Error: API Error: https://api.anthropic.com/v1/messages, 400, Bad Request { "type": "error", "error": { "type": "invalid_request_error", "message": "Your API request included an `assistant` message in the final position, which would pre-fill the `assistant` response. When using tools, pre-filling the `assistant` response is not supported." } } ``` Also: - Anthropic just released "claude-3-5-sonnet-20240620" 🎉

Groq

``` @ax-llm/ax/build/module/src/ai/openai/api.js:166 ({ delta: { content, role, tool_calls }, finish_reason }) => { ^ TypeError: Cannot read properties of undefined (reading 'content') ``` Also: - ai/groq/index.ts is missing "export * from './types.js';" - is 'AxAxGroqArgs' a typo?

Cohere

``` @ax-llm/ax/build/module/src/util/apicall.js:39 throw new Error(`API Error: ${apiUrl.href}, ${e}`); ^ Error: API Error: https://api.cohere.ai/v1/generate, Error: API Error: https://api.cohere.ai/v1/generate, 400, Bad Request { "message": "invalid request: prompt must be at least 1 token long." } ```

Google

``` @ax-llm/ax/build/module/src/util/apicall.js:39 throw new Error(`API Error: ${apiUrl.href}, ${e}`); ^ Error: API Error: https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:streamGenerateContent?alt=sse&key=, Error: API Error: https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:streamGenerateContent?alt=sse&key=, 400, Bad Request { "error": { "code": 400, "message": "Invalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field.\nInvalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[1].parameters.properties[1].value': Cannot find field.", "status": "INVALID_ARGUMENT", "details": [ { "@type": "type.googleapis.com/google.rpc.BadRequest", "fieldViolations": [ { "field": "tools[0].function_declarations[0].parameters.properties[1].value", "description": "Invalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field." }, { "field": "tools[0].function_declarations[1].parameters.properties[1].value", "description": "Invalid JSON payload received. Unknown name \"default\" at 'tools[0].function_declarations[1].parameters.properties[1].value': Cannot find field." } ] } ] } } ```

Other observations:

• DeepSeek worked quite well, except when I changed "sushi" to "sashimi", then it failed... But it's a coding model, not a foodie.
• None of the Together function calling models worked (they're either confused or hallucinating): mistralai/Mixtral-8x7B-Instruct-v0.1 mistralai/Mistral-7B-Instruct-v0.1 togethercomputer/CodeLlama-34b-Instruct

[dosco]

will take a look

[dosco]

can you confirm you're testing with the latest version we fixed a bunch of stuff today and also added the new 3.5 sonnet model.

[beddows]

@dosco Anthropic and Groq still throwing the same errors. I'm having different issues with Cohere and Google, but I need to double check that I'm not missing something.

I like the new AxAI syntax, easier to follow!

[dosco]

great will take a look and fix this today. thanks as the api was growing the proper prefix helps with autocomplete etc

[dosco]

latest release https://github.com/ax-llm/ax/releases/tag/9.0.9 has fixes for anthropic, cohere, gemini, i'm looking into groq but i suspect its more a model issue there i might bump up the default model choice to a bigger model

[beddows]

@dosco Cool, I'll check it out! Regarding Groq, one thing to keep in mind - they limit llama3-70b to 6k tokens/min and llama3-8b to 30k t/m. I found myself hitting the limit with the larger model quite often, within one run.

[dosco]

probably need to add a rate limiter we support those in the library maybe groq needs one by default

[dosco]

in the latest release i added a default token bucket bases rate limiter to groq by default to slow it down when needed.