I followed instructions to use the Iterative Product Quantization provided here: (https://github.com/pytorch/fairseq/tree/master/examples/quant_noise)
I succeeded to create a transformer model using the commands below
although it seems that not all parameters are actually quantized.
But the main problem arises when I tried generation, because I get a dummy output.
To Reproduce
Steps to reproduce the behavior:
preprocessing
fairseq-preprocess --source-lang en --target-lang it --trainpref fairseq_bpefied/EPv8_en__it.train --validpref fairseq_bpefied/EPv8_en__it.valid --testpref fairseq_bpefied/EPv8_en__it.test --destdir fairseq_data-bin/EPv8.en__it --joined-dictionary
During generation I got this dummy output, when I enter this input My name is Nicola
My name is Nicola
S-0 My name is Nic@@ ola
W-0 0.062 seconds
H-0 -5.2093610763549805 in in in in in
D-0 -5.2093610763549805 in in in in in
P-0 -5.1419 -5.1419 -5.1419 -5.1419 -5.1419 -5.5468
### Expected behavior
Translating with a non-quantized transformer model, trained in a comparable way (same parameters, but quantization ones), I got the "right" output
My name is Nicola
S-0 My name is Nic@@ ola
W-0 0.087 seconds
H-0 -0.2353324592113495 Il mio nome è Nic@@ ola .
D-0 -0.2353324592113495 Il mio nome è Nicola .
P-0 -0.4744 -0.0869 -0.0734 -0.2561 -0.4708 -0.0012 -0.4113 -0.1085
🐛 Bug
I followed instructions to use the Iterative Product Quantization provided here: (https://github.com/pytorch/fairseq/tree/master/examples/quant_noise) I succeeded to create a transformer model using the commands below although it seems that not all parameters are actually quantized.
But the main problem arises when I tried generation, because I get a dummy output.
To Reproduce
Steps to reproduce the behavior:
preprocessing
training
I run these two commands sequentially:
Note: the
transformer_quantization_config
configuration file is the default provided in the instructions. Note: here are thetrain_options
:interactive generation
Errors:
Error 1
During training only the following layers are quantized
whereas in the config file self_attn parameters should be quantized as you can see here:
Error 2
During generation I got this dummy output, when I enter this input
My name is Nicola
My name is Nicola S-0 My name is Nic@@ ola W-0 0.087 seconds H-0 -0.2353324592113495 Il mio nome è Nic@@ ola . D-0 -0.2353324592113495 Il mio nome è Nicola . P-0 -0.4744 -0.0869 -0.0734 -0.2561 -0.4708 -0.0012 -0.4113 -0.1085
Environment
pip
, source): pip