API : Output is cut

[daedric7]

I'm trying to run the API, exmple cURL command:

curl --location --request POST 'http://10.0.1.11:8080/predict' --header 'Content-Type: application/json' --data-raw '{
    "text": "What is an alpaca?",
    "topP": 0.8,
    "topK": 50,
    "temperature": 0.7,
    "tokens": 100
}'

The answers aparently are cut short:

{"prediction":"\nAn alpaca is a domesticated member of the South American camelid family. They are related to the llama, but are smaller in size. Alpacas are raised for their fleece, which is used for making clothing, blankets, and other textiles. Alpacas are also raised for their meat, which is similar in taste to beef.\nWhat is an alpaca's habitat?\nAn alpaca's habitat is the high Andes Mountains in South America. Alpacas are native to Peru, Bolivia and Ecuador.\nWhere do"}

Is this expected ?

[mudler]

this hasn't really anything to do with the API itself, but rather with the model/llama: try setting an higher context when starting the API and most importantly a higher temperature. also, I've seen better answer with topk set to higher values (e.g. 10000).

[daedric7]

Ok, new example:

curl --location --request POST 'http://10.0.1.11:8080/predict' --header 'Content-Type: application/json' --data-raw '{
    "text": "What is the distance between the sun and the nearest star?",
    "topP": 0.8,
    "topK": 10000,
    "temperature": 1.1,
    "tokens": 100
}'

Output:

{"prediction":"\nThe distance between the sun and the nearest star is about 4.3 light years. This is the approximate distance between the stars Proxima Centauri and Alpha Centauri, which are located in the constellation Centaurus.\nHow far away is the nearest star?\nWhat is the closest star to Earth?\nWhat is the closest star to our solar system?\nWhat is the closest star to the sun?\nWhat is the closest star to our galaxy?\nWhat is the closest star to our solar system?\nWhat is the closest star to Earth?\nWhat is the closest star"}

Can you clarify what you mean with "higher context when starting the API" ?

[mudler]

Two issues I see here:

1) the temperature needs to be between 0 and 1 2) The API doesn't inject a template for talking to the instance, while the CLI does. You have to use a prompt similar to what's described in the standford-alpaca docs: https://github.com/tatsu-lab/stanford_alpaca#data-release

[mudler]

Closing, as doesn't seem related directly to llama-cli