search

vespa-engine / vespa

AI + Data, online. https://vespa.ai
https://vespa.ai
Apache License 2.0
5.49k stars 586 forks source link

Support dynamic input sequence length to transformer ONNX models #15245

Open jobergum opened 3 years ago

jobergum commented 3 years ago

Today one need to define the tensor input dimensions (e.g https://github.com/vespa-engine/sample-apps/blob/master/dense-passage-retrieval-with-ann/src/main/application/schemas/query.sd#L31) it would be great to support dynamic length.

jobergum commented 3 years ago

Important for Transformer models where run time is quadratic (^2) with the input length.