Open 0xSage opened 4 months ago
Hi @0xSage! I suggest only detecting the .engine files for now, they seem like the more popular format on the hub right now (450 models contain a .engine
file, vs. 0 model repo contain a .plans
file)
We can auto-tag those repos with a tensorrt
tag, i think it'd be the easiest!
Problem
isTensorrtModel Rules
.engine
.config.json
. Caveat: By design, model builders can actually rename this file.Engine compatibility rules
For context, TensorRT models are specific to:
GPU architectures
, i.e. models compiled for Ada will only run on AdaTRT-LLM release
, i.e. models compiled on release version v0.9.0 will need to run on 0.9.0OS
(optional), though as of v0.9.0, models are cross OS compatible. We're still testing as it could be flaky.n GPUs
, i.e. GPU topology. This can be detected by counting the # of engine files actually.Unfortunately, afaik
config.json
and other metadata files do not track the hardware/build-time configurations once the models are built, so model authors will have to specify this info.^ We'll update this info as it changes, and as we learn more 😄 .
Naming
.plans
or.onnx
.engines
isTensorrtEngine
vsisTensorrtPlan
?