Open lanking520 opened 1 month ago
the rel_attn_table
warning can be ignored. We're trying to reproduce the error. In theory this shouldn't happen because we run flan-t5 test everyday and they're passing. And your commands look good to me
So the error reporting is because the following value in the tritonserver is not set
parameters: {
key: "max_tokens_in_paged_kv_cache"
value: {
string_value: "${max_tokens_in_paged_kv_cache}"
}
}
parameters: {
key: "max_attention_window_size"
value: {
string_value: "${max_attention_window_size}"
}
}
After setting both of them to 4096, the model start to work. But wonder why not setting them doesn't work
Hi @lanking520,
I'm not able to reproduce your error with the latest main (2fa86dc). I build the model with your commands (without --use_custom_all_reduce disable
, --tp_size
, --pp_size
due to deprecation of some features), and run the tritonserver
following the encoder-decoder tutorial. There is no problem with or without max_tokens_in_paged_kv_cache
and max_attention_window_size
.
By default, max_attention_window_size
would be set to min(max_attention_window_size, max_seq_len)
and max_tokens_in_paged_kv_cache
would be set according to current free memory on the device if kv_cache_free_gpu_mem_fraction
has value. There should not be a problem if those values are not set.
Please share your tritonserver
config so that we can further investigate this problem, thanks!!
_note: one parameter needed to be added to tensorrt_llm/config.pbtxt
if using the latest main: max_queuesize:0
System Info
L4 GPU (AWS G6.12xl) with TensorRTLLM 0.11.0, running with Tritonbackends
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Convert checkpoint
Build Encoder
Build Decoder
Expected behavior
Should be running fine
actual behavior
During the decoder compilation, seeing:
During the inference, getting
additional notes
N/A