triton-inference-server / tensorrtllm_backend

The Triton TensorRT-LLM Backend
Apache License 2.0
657 stars 94 forks source link

About CUM_LOG_PROBS and OUTPUT_LOG_PROBS #291

Open callmezhangchenchenokay opened 8 months ago

callmezhangchenchenokay commented 8 months ago
{
    name: "OUT_CUM_LOG_PROBS"
    data_type: TYPE_FP32
    dims: [ -1 ]
  },
  {
    name: "OUT_OUTPUT_LOG_PROBS"
    data_type: TYPE_FP32
    dims: [ -1, -1 ]
  }

I get the output this way and print it, and the output is all zeros. cum_log_pros = result.as_numpy('OUT_CUM_LOG_PROBS')

schetlur-nv commented 8 months ago

Can you try result.as_numpy('cum_log_probs')?

callmezhangchenchenokay commented 8 months ago

ok, I know what you mean Whether to use cum_log_probs or OUT_CUM_LOG_PROBS depends on your config information

No matter how you write it, the output is 0.

schetlur-nv commented 8 months ago

Interesting - which model are you running? This seems to be working fine in our CI testing.

callmezhangchenchenokay commented 8 months ago

I feel that my version is too low, I will try the results of the latest version

callmezhangchenchenokay commented 8 months ago

I got it; I need to set "return_log_probs" to True, But why are the returned values so small [[1]], the following content

image
tstar589 commented 5 months ago

@callmezhangchenchenokay What was the version you used? Running into same issue and it is emitting 0s?