Using local inference results in "Failed generating code: result was truncated: length"

gofireflyio / aiac

Artificial Intelligence Infrastructure-as-Code Generator.

Apache License 2.0

3.52k stars 276 forks source link

Using local inference results in "Failed generating code: result was truncated: length" #121

Closed remmen-io closed 4 months ago

remmen-io commented 6 months ago

I'm using a local inference service using huggingface tgi, which is compatible with the openai api.

I get the following error using it with aiac

aiac get --url="https://deepseek.myinference.ch/v1" --api-key="none" terraform code for a s3 bucket on aws
Failed generating code: result was truncated: length

[R/r]: retry same prompt
[Y/y]: copy to clipboard
[Q/q]: quit

Using the inference api with curl or other AI tools works flawless

ido50 commented 5 months ago

Hi, sorry for the delay. Can you post the full JSON output you get when running the same exact query via curl? It seems the API is telling us it did not return the full generated text.

ido50 commented 4 months ago

Version 5.0.0 no longer fails on truncated output from the API. Instead, the library returns the stop reason for the user to decide how to proceed.