Closed VibhuJawa closed 8 months ago
Looks like a known limitation, could you please try latest TRT 9.2? Thanks!
Looks like a known limitation, could you please try latest TRT 9.2? Thanks!
Can you link me to on how do I get access to latest TRT 9.2 , please?
I dont see nightly wheels.
I tested it with tensorrt==9.0.1.post12.dev4
and i can create the engine now.
I want to support dynamic batch and sequence sizes, I am running into below warnings which based on my understanding will mean that we will fail there. Can you suggest how to get that working ?
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
/datasets/vjawa/miniconda3/envs/TensorRT_Transformers/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:554: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
torch.tensor(mid - 1).type_as(relative_pos),
/datasets/vjawa/miniconda3/envs/TensorRT_Transformers/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:558: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
torch.ceil(torch.log(abs_pos / mid) / torch.log(torch.tensor((max_position - 1) / mid)) * (mid - 1)) + mid
/datasets/vjawa/miniconda3/envs/TensorRT_Transformers/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:717: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
scale = torch.sqrt(torch.tensor(query_layer.size(-1), dtype=torch.float) * scale_factor)
/datasets/vjawa/miniconda3/envs/TensorRT_Transformers/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:717: UserWarning:
.................
if key_layer.size(-2) != query_layer.size(-2):
/datasets/vjawa/miniconda3/envs/TensorRT_Transformers/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py:112: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
output = input.masked_fill(rmask, torch.tensor(torch.finfo(input.dtype).min))
it's a warning from transformers(pytorch), I think it means use dynamic shape may caused unexpected behavior when export onnx(e.g. some tensor still become constant) during export onnx. You can ask for help in the transformer repo.
Closing since no activity for more than 3 weeks, thanks all!
is there any release or tag of TensorRT 8.6.1 that resolves this issue ?
Because I could built it with TRT 9.2.0, but I want to deploy this model on nvidia triton inference server.
However, triton tensorRT backend does not yet support newer version of tensorRT than 8.6.1, and the tensorRT runtime version should be the same than the one used to build the engine.
What would be my best solution for this problem ?
Description
I am running into conversion issue while trying to convert
deberta-v3-base
into a TensorRT engine. We run intoMore Trace is present next to MRE.
Environment
TensorRT Version:
8.6
NVIDIA GPU:
V100
NVIDIA Driver Version: 525.105.17
CUDA Version: 12.0
CUDNN Version:
Operating System:
Python Version (if applicable): 3.10
Steps To Reproduce
Also tried following https://github.com/NVIDIA/TensorRT/issues/3124 but to no avail.