Closed masahi closed 2 years ago
cc @driazati as there's some overlap with tlc-pack/ci-data
IIUC, you want to add the serialized binary files of these models to this repo. In terms of the license, I'd like to confirm that did you generate these binary files by yourself based on the FasterTransformer, or are these directly cloned from somewhere in that repo?
I checked the FastTransformer repo and it is Apache-2.0 license, so it is fine for us to use any code from that repo. We only need to add one line saying this is modified from FastTransformer repo. In the case of binary files, I think a separate README.md under the same directory also works.
Yes I generated both of them myself. I added ugly hack in one of scripts in FasterTransformer
to manually export the model to ONNX. A new README is there already, I can add more details on the export process if desired.
I was not aware of ci-data
, but these models are not for CI. So I think this repo is better.
Got it. Then IMHO we just need to explicitly saying like generated from XXX under Apache-2.0 license
in README.
Ok, added more details on the export process and licensed under Apache-2.0
blurb. README is here https://github.com/masahi/TLCBench/blob/bench-models/models/README.md
I think it is good to go.
An example on how to use quantized BERT (Running requires https://github.com/apache/tvm/pull/10596):
Example output:
One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.
Evaluate inference time cost with target cuda ...
Execution time summary:
mean (ms) median (ms) max (ms) min (ms) std (ms)
18.4024 18.2218 19.6602 18.1031 0.4598
One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.
Evaluate inference time cost with target cuda -libs=cublas ...
Execution time summary:
mean (ms) median (ms) max (ms) min (ms) std (ms)
9.2887 9.2200 9.7776 9.1559 0.2160
The motivation is that, there has been growing interest in testing and benchmarking int8 BERT. But BERT and other transformer models are hard to quantize or import to TVM properly, so until recently they were not available to us for benchmarking.
Recently I found that NVIDIA
FasterTransformer
repo has an example of quantizing BERT by PTQ or QAT using TensorRT'spytorch_quantization
tool. And thanks to the recent work on https://github.com/apache/tvm/pull/10239, we can now import those "fake-quantized" QAT BERT models into Relay and convert them into a fully-integer model.Example usages will be provided soon in the tvm repo under
python/tvm/meta_schedule/testing/XXX.py
. Also see https://github.com/tlc-pack/TLCBench/pull/5#issuecomment-1067431710I wonder if we need to worry about license issues?
cc @junrushao1994 @areusch @tqchen @comaniac