no speedup using ort

[housebaby]

I have tried using ort in training transformer . But it seems that no speed up is got. I wonder whether i have missed someting in configuration.

[baijumeswani]

Please share with us your model code if possible?

[housebaby]

Please share with us your model code if possible?

model = ORTModule(init_asr_model(configs)) this is how I use the ort.

when I print the model, it is as follows: ORTModule(` (encoder): ConformerEncoder( (global_cmvn): GlobalCMVN() (embed): Conv2dSubsampling4( (conv): Sequential( (0): Conv2d(1, 512, kernel_size=(3, 3), stride=(2, 2)) (1): ReLU() (2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2)) (3): ReLU() ) (out): Sequential( (0): Linear(in_features=9728, out_features=512, bias=True) ) (pos_enc): RelPositionalEncoding( (dropout): Dropout(p=0.1, inplace=False) ) ) (after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (encoders): ModuleList( (0): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (1): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (2): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (3): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (4): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (5): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (6): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (7): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (8): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (9): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (10): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) (11): ConformerEncoderLayer( (self_attn): RelPositionMultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) (linear_pos): Linear(in_features=512, out_features=512, bias=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (feed_forward_macaron): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): SiLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (conv_module): ConvolutionModule( (pointwise_conv1): Conv1d(512, 1024, kernel_size=(1,), stride=(1,)) (depthwise_conv): Conv1d(512, 512, kernel_size=(15,), stride=(1,), groups=512) (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True) (pointwise_conv2): Conv1d(512, 512, kernel_size=(1,), stride=(1,)) (activation): SiLU() ) (norm_ff): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_mha): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_ff_macaron): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_conv): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm_final): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear): Linear(in_features=1024, out_features=512, bias=True) ) ) ) (decoder): TransformerDecoder( (embed): Sequential( (0): Embedding(5002, 512) (1): PositionalEncoding( (dropout): Dropout(p=0.1, inplace=False) ) ) (after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (output_layer): Linear(in_features=512, out_features=5002, bias=True) (decoders): ModuleList( (0): DecoderLayer( (self_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (src_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): ReLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear1): Linear(in_features=1024, out_features=512, bias=True) (concat_linear2): Linear(in_features=1024, out_features=512, bias=True) ) (1): DecoderLayer( (self_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (src_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): ReLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear1): Linear(in_features=1024, out_features=512, bias=True) (concat_linear2): Linear(in_features=1024, out_features=512, bias=True) ) (2): DecoderLayer( (self_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (src_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): ReLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear1): Linear(in_features=1024, out_features=512, bias=True) (concat_linear2): Linear(in_features=1024, out_features=512, bias=True) ) (3): DecoderLayer( (self_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (src_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): ReLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear1): Linear(in_features=1024, out_features=512, bias=True) (concat_linear2): Linear(in_features=1024, out_features=512, bias=True) ) (4): DecoderLayer( (self_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (src_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): ReLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear1): Linear(in_features=1024, out_features=512, bias=True) (concat_linear2): Linear(in_features=1024, out_features=512, bias=True) ) (5): DecoderLayer( (self_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (src_attn): MultiHeadedAttention( (linear_q): Linear(in_features=512, out_features=512, bias=True) (linear_k): Linear(in_features=512, out_features=512, bias=True) (linear_v): Linear(in_features=512, out_features=512, bias=True) (linear_out): Linear(in_features=512, out_features=512, bias=True) (dropout): Dropout(p=0.0, inplace=False) ) (feed_forward): PositionwiseFeedForward( (w_1): Linear(in_features=512, out_features=2048, bias=True) (activation): ReLU() (dropout): Dropout(p=0.1, inplace=False) (w_2): Linear(in_features=2048, out_features=512, bias=True) ) (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True) (dropout): Dropout(p=0.1, inplace=False) (concat_linear1): Linear(in_features=1024, out_features=512, bias=True) (concat_linear2): Linear(in_features=1024, out_features=512, bias=True) ) ) ) (ctc): CTC( (ctc_lo): Linear(in_features=512, out_features=5002, bias=True) (ctc_loss): CTCLoss() ) (criterion_att): LabelSmoothingLoss( (criterion): KLDivLoss() ) )

[housebaby]

Please share with us your model code if possible?

class ASRModel(torch.nn.Module):
    """CTC-attention hybrid Encoder-Decoder model"""
    def __init__(
        self,
        vocab_size: int,
        encoder: TransformerEncoder,
        decoder: TransformerDecoder,
        ctc: CTC,
        ctc_weight: float = 0.5,
        ignore_id: int = IGNORE_ID,
        reverse_weight: float = 0.0,
        lsm_weight: float = 0.0,
        length_normalized_loss: bool = False,
    ):
        assert 0.0 <= ctc_weight <= 1.0, ctc_weight

        super().__init__()
        # note that eos is the same as sos (equivalent ID)
        self.sos = vocab_size - 1
        self.eos = vocab_size - 1
        self.vocab_size = vocab_size
        self.ignore_id = ignore_id
        self.ctc_weight = ctc_weight
        self.reverse_weight = reverse_weight

        self.encoder = encoder
        self.decoder = decoder
        self.ctc = ctc
        self.criterion_att = LabelSmoothingLoss(
            size=vocab_size,
            padding_idx=ignore_id,
            smoothing=lsm_weight,
            normalize_length=length_normalized_loss,
        )

    def forward(
        self,
        speech: torch.Tensor,
        speech_lengths: torch.Tensor,
        text: torch.Tensor,
        text_lengths: torch.Tensor,
    ) -> Tuple[Optional[torch.Tensor], Optional[torch.Tensor],
               Optional[torch.Tensor]]:
        """Frontend + Encoder + Decoder + Calc loss

        Args:
            speech: (Batch, Length, ...)
            speech_lengths: (Batch, )
            text: (Batch, Length)
            text_lengths: (Batch,)
        """
        assert text_lengths.dim() == 1, text_lengths.shape
        # Check that batch_size is unified
        assert (speech.shape[0] == speech_lengths.shape[0] == text.shape[0] ==
                text_lengths.shape[0]), (speech.shape, speech_lengths.shape,
                                         text.shape, text_lengths.shape)
        # 1. Encoder
        encoder_out, encoder_mask = self.encoder(speech, speech_lengths)
        encoder_out_lens = encoder_mask.squeeze(1).sum(1)

        # 2a. Attention-decoder branch
        if self.ctc_weight != 1.0:
            loss_att, acc_att = self._calc_att_loss(encoder_out, encoder_mask,
                                                    text, text_lengths)
        else:
            loss_att = None

        # 2b. CTC branch
        if self.ctc_weight != 0.0:
            loss_ctc = self.ctc(encoder_out, encoder_out_lens, text,
                                text_lengths)
        else:
            loss_ctc = None

        if loss_ctc is None:
            loss = loss_att
        elif loss_att is None:
            loss = loss_ctc
        else:
            loss = self.ctc_weight * loss_ctc + (1 -
                                                 self.ctc_weight) * loss_att
        return loss, loss_att, loss_ctc

[ytaous]

Hi, would you please provide steps to repro the issue? including sample data and run scripts? Also what's your runtime env? i.e., installations, ort version, etc. Older version of ORT may not have obvious gain, you may try our latest release + upgraded torch and it makes difference before getting back to us. https://download.onnxruntime.ai/ Thanks.