Test CausalLM generate & pipeline

huggingface / optimum-nvidia

Apache License 2.0

887 stars 86 forks source link

Closed fxmarty closed 5 months ago

fxmarty commented 6 months ago

Tests that TensorRT-based generate method matches Transformers API and generation.

Current status:

[x] TRT Llama batch size = 1 matches exactly with Transformers
[x] Mistral generation is completely different with TRT, although not incoherent (missing config arg)
[x] Gemma does not generate EOS with TRT while it does with Transformers (transformers truncates to max_length = 20)

fxmarty commented 6 months ago

My ruff is ignoring

[tool.ruff.lint.isort]
lines-after-imports = 2
known-first-party = ["optimum"]

and I have no idea why.