fine tuning in korean conformer-transducer

dykim3 commented 2 years ago

How can I get a space(blank) it self in tokenizer vocab? I am trying to finetune the stt_en_conformer_transducer_large model with grapheme in korean. I changed some code to load labels with chardataset.

in finetuning the prediction log shows blank in a unk tag. (??)

[NeMo I 2022-05-23 06:02:11 rnnt_wer_bpe:231] reference :그래서 ⁇ 내가 ⁇ 오글 ⁇ 막 ⁇ 이랬는데
[NeMo I 2022-05-23 06:02:11 rnnt_wer_bpe:232] predicted :그래서 ⁇ 다 ⁇ 얼 ⁇ 말인데

this is my tokenizer vocab.

<unk>   0
<b> 0
▁   -0.693147
ᄌ   -4.18965
ᄀ   -4.8828
ᄁ   -4.8828
ᄂ   -4.8828
ᄃ   -4.8828
ᄄ   -4.8828
ᄅ   -4.8828
ᄆ   -4.8828
ᄇ   -4.8828
ᄈ   -4.8828
ᄉ   -4.8828
ᄊ   -4.8828
ᄋ   -4.8828
ᄍ   -4.8828
ᄎ   -4.8828
ᄏ   -4.8828
ᄐ   -4.8828
ᄑ   -4.8828
ᄒ   -4.8828
ᅡ   -4.8828
ᅢ   -4.8828
ᅣ   -4.8828
ᅤ   -4.8828
ᅥ   -4.8828
ᅦ   -4.8828
ᅧ   -4.8828
ᅨ   -4.8828
ᅩ   -4.8828
ᅪ   -4.8828
ᅫ   -4.8828
ᅬ   -4.8828
ᅭ   -4.8828
ᅮ   -4.8828
ᅯ   -4.8828
ᅰ   -4.8828
ᅱ   -4.8828
ᅲ   -4.8828
ᅳ   -4.8828
ᅴ   -4.8828
ᅵ   -4.8828
ᆨ   -4.8828
ᆩ   -4.8828
ᆪ   -4.8828
ᆫ   -4.8828
ᆬ   -4.8828
ᆭ   -4.8828
ᆮ   -4.8828
ᆯ   -4.8828
ᆰ   -4.8828
ᆱ   -4.8828
ᆲ   -4.8828
ᆴ   -4.8828
ᆶ   -4.8828
ᆷ   -4.8828
ᆸ   -4.8828
ᆹ   -4.8828
ᆺ   -4.8828
ᆻ   -4.8828
ᆼ   -4.8828
ᆽ   -4.8828
ᆾ   -4.8828
ᆿ   -4.8828
ᇀ   -4.8828
ᇁ   -4.8828
ᇂ   -4.8828

this is my config.

name: "Conformer-Transducer-BPE"
init_from_nemo_model:
  model0:
    path: ??
    exclude: ["joint", "decoder"]
    include: ["encoder"]

model:
  sample_rate: &sample_rate 16000
  compute_eval_loss: false # eval samples can be very long and exhaust memory. Disable computation of transducer loss during validation/testing with this flag.
  log_prediction: true # enables logging sample predictions in the output during training
  skip_nan_grad: false
  change_vocab: True # added
  use_cer: True # added

  model_defaults:
    enc_hidden: ${model.encoder.d_model}
    pred_hidden: 640
    joint_hidden: 640

  train_ds:
    manifest_filepath: ???
    sample_rate: 16000 #${model.sample_rate}
    batch_size: 16 # you may increase batch_size if your memory allows
    shuffle: true
    num_workers: 8
    pin_memory: true
    # use_start_end_token: True
    trim_silence: false
    max_duration: 16.7 # it is set for LibriSpeech, you may need to update it for your dataset
    min_duration: 0.1
    # tarred datasets
    is_tarred: false
    tarred_audio_filepaths: null
    shuffle_n: 2048
    # bucketing params
    bucketing_strategy: "synced_randomized"
    bucketing_batch_size: null
    use_char: true
    unk_index: 0

  validation_ds:
    manifest_filepath: ???
    sample_rate: 16000 #${model.sample_rate}
    batch_size: 16
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_char: true
    unk_index: 0

    # use_start_end_token: True

  test_ds:
    manifest_filepath: null
    sample_rate: 16000 #${model.sample_rate}
    batch_size: 16
    shuffle: false
    num_workers: 8
    pin_memory: true
    use_char: true

    # use_start_end_token: True

  # You may find more detail on how to train a tokenizer at: /scripts/tokenizers/process_asr_text_tokenizer.py
  tokenizer:
    dir: ???  # path to directory which contains either tokenizer.model (bpe) or vocab.txt (for wpe)
    type: bpe  # Can be either bpe (SentencePiece tokenizer) or wpe (WordPiece tokenizer)

  preprocessor:
    _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
    sample_rate: *sample_rate
    normalize: "per_feature"
    window_size: 0.025
    window_stride: 0.01
    window: "hann"
    features: 80
    n_fft: 512
    frame_splicing: 1
    dither: 0.00001
    pad_to: 0

  spec_augment:
    _target_: nemo.collections.asr.modules.SpectrogramAugmentation
    freq_masks: 2 # set to zero to disable it
    time_masks: 10 # set to zero to disable it
    freq_width: 27
    time_width: 0.05

  encoder:
    _target_: nemo.collections.asr.modules.ConformerEncoder
    feat_in: ${model.preprocessor.features}
    feat_out: -1 # you may set it if you need different output size other than the default d_model
    n_layers: 17
    d_model: 512

    # Sub-sampling params
    subsampling: striding # vggnet or striding
    subsampling_factor: 4 # must be power of 2
    subsampling_conv_channels: -1 # set to -1 to make it equal to the d_model

    # Feed forward module's params
    ff_expansion_factor: 4

    # Multi-headed Attention Module's params
    self_attention_model: rel_pos # rel_pos or abs_pos
    n_heads: 8 # may need to be lower for smaller d_models
    # [left, right] specifies the number of steps to be seen from left and right of each step in self-attention
    att_context_size: [-1, -1] # -1 means unlimited context
    xscaling: true # scales up the input embeddings by sqrt(d_model)
    untie_biases: true # unties the biases of the TransformerXL layers
    pos_emb_max_len: 5000

    # Convolution module's params
    conv_kernel_size: 31
    conv_norm_type: 'batch_norm' # batch_norm or layer_norm

    ### regularization
    dropout: 0.1 # The dropout used in most of the Conformer Modules
    dropout_emb: 0.0 # The dropout used for embeddings
    dropout_att: 0.1 # The dropout for multi-headed attention modules

  decoder:
    _target_: nemo.collections.asr.modules.RNNTDecoder
    normalization_mode: null # Currently only null is supported for export.
    random_state_sampling: false # Random state sampling: https://arxiv.org/pdf/1910.11455.pdf
    blank_as_pad: true # This flag must be set in order to support exporting of RNNT models + efficient inference.

    prednet:
      pred_hidden: ${model.model_defaults.pred_hidden}
      pred_rnn_layers: 1
      t_max: null
      dropout: 0.1

  joint:
    _target_: nemo.collections.asr.modules.RNNTJoint
    log_softmax: null  # 'null' would set it automatically according to CPU/GPU device
    preserve_memory: false  # dramatically slows down training, but might preserve some memory

    # Fuses the computation of prediction net + joint net + loss + WER calculation
    # to be run on sub-batches of size `fused_batch_size`.
    # When this flag is set to true, consider the `batch_size` of *_ds to be just `encoder` batch size.
    # `fused_batch_size` is the actual batch size of the prediction net, joint net and transducer loss.
    # Using small values here will preserve a lot of memory during training, but will make training slower as well.
    # An optimal ratio of fused_batch_size : *_ds.batch_size is 1:1.
    # However, to preserve memory, this ratio can be 1:8 or even 1:16.
    # Extreme case of 1:B (i.e. fused_batch_size=1) should be avoided as training speed would be very slow.
    fuse_loss_wer: true
    fused_batch_size: 16

    jointnet:
      joint_hidden: ${model.model_defaults.joint_hidden}
      activation: "relu"
      dropout: 0.1

  decoding:
    strategy: "greedy_batch" # can be greedy, greedy_batch, beam, tsd, alsd.

    # greedy strategy config
    greedy:
      max_symbols: 30

    # beam strategy config
    beam:
      beam_size: 2
      return_best_hypothesis: False
      score_norm: true
      tsd_max_sym_exp: 50  # for Time Synchronous Decoding
      alsd_max_target_len: 2.0  # for Alignment-Length Synchronous Decoding

  loss:
    loss_name: "default"

    warprnnt_numba_kwargs:
      # FastEmit regularization: https://arxiv.org/abs/2010.11148
      # You may enable FastEmit to reduce the latency of the model for streaming
      fastemit_lambda: 0.0  # Recommended values to be in range [1e-4, 1e-2], 0.001 is a good start.
      clamp: -1.0  # if > 0, applies gradient clamping in range [-clamp, clamp] for the joint tensor only.

  # Adds Gaussian noise to the gradients of the decoder to avoid overfitting
  variational_noise:
    start_step: 0
    std: 0.0

  optim:
    name: adamw
    lr: 5.0
    # optimizer arguments
    betas: [0.9, 0.98]
    weight_decay: 1e-3

    # scheduler setup
    sched:
      name: NoamAnnealing
      d_model: ${model.encoder.d_model}
      # scheduler config override
      warmup_steps: 10000
      warmup_ratio: null
      min_lr: 1e-6

trainer:
  devices: -1 # number of GPUs, -1 would use all available GPUs
  num_nodes: 1
  max_epochs: 500
  max_steps: null # computed at runtime if not set
  val_check_interval: 1.0 # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
  accelerator: auto
  strategy: ddp
  accumulate_grad_batches: 1
  gradient_clip_val: 0.0
  precision: 32 # Should be set to 16 for O1 and O2 to enable the AMP.
  log_every_n_steps: 50  # Interval of logging.
  progress_bar_refresh_rate: 10
  resume_from_checkpoint: null # The path to a checkpoint file to continue the training, restores the whole state including the epoch, step, LR schedulers, apex, etc.
  num_sanity_val_steps: 0 # number of steps to perform validation steps for sanity check the validation process before starting the training, setting to 0 disables it
  check_val_every_n_epoch: 1 # number of evaluations on validation every n epochs
  sync_batchnorm: true
  enable_checkpointing: False  # Provided by exp_manager
  logger: false  # Provided by exp_manager

exp_manager:
  exp_dir: null
  name: ${name}
  create_tensorboard_logger: true
  create_checkpoint_callback: true
  checkpoint_callback_params:
    # in case of multiple validation sets, first one is used
    monitor: "val_wer"
    mode: "min"
    save_top_k: 1
    always_save_nemo: True # saves the checkpoints as nemo files instead of PTL checkpoints
  resume_if_exists: False
  resume_ignore_no_checkpoint: false

  create_wandb_logger: false
  wandb_logger_kwargs:
    name: null
    project: null

titu1994 commented 2 years ago

You can't add tokens after the tokenizer has been built, so the text that builds the manifest must contain some special Unicode character that represents blank. What does <b> represent ? If you mean CTC/RNNT blank, in Nemo it is the final token in the vocab.

titu1994 commented 2 years ago

Also when finetuning model, change tokenizer / vocabulary before setting up data loaders. The diagram of order of steps is here - https://github.com/NVIDIA/NeMo/tree/main/examples/asr/asr_transducer

dykim3 commented 2 years ago

got it. so i've changed the token to ' ' and loaded EncDecRNNTModel instead of EncDecRNNTBPEModel. then it has an error with loading optimizer..

@hydra_runner(config_path="experimental/contextnet_rnnt", config_name="config_rnnt")
def main(cfg):
    # logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')

    trainer = pl.Trainer(**cfg.trainer)
    exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecRNNTModel(cfg=cfg.model, trainer=trainer)

    asr_model.maybe_init_from_pretrained_checkpoint(cfg)

    # Initialize the weights of the model from another model, if provided via config
    if cfg.model.change_vocab:
        asr_model.change_vocabulary(new_vocabulary=list(korean_vocab))
        asr_model.cfg.labels = list(korean_vocab)
        asr_model.setup_training_data(cfg.model.train_ds)
        asr_model.setup_multiple_validation_data(cfg.model.validation_ds)
        asr_model.setup_test_data(cfg.model.test_ds)
        asr_model.set_trainer(trainer)

        # asr_model.model.preprocessor = asr_model.from_config_dict(cfg.model.preprocessor)
        asr_model.setup_optimization(optim_config=cfg.model.optim)
        asr_model.spec_augment = asr_model.from_config_dict(cfg.model.spec_augment)
        asr_model.cfg = cfg

    logging.info(f'Hydra config: {OmegaConf.to_yaml(asr_model.cfg)}')

    trainer.fit(asr_model)
    asr_model.save_to('conformer_test.nemo')

    if hasattr(cfg.model, 'test_ds') and cfg.model.test_ds.manifest_filepath is not None:
        if asr_model.prepare_test(trainer):
            trainer.test(asr_model)

if __name__ == '__main__':
    main()  # noqa pylint: disable=no-value-for-parameter

nohup: ignoring input
[NeMo W 2022-05-24 08:10:46 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py:52: LightningDeprecationWarning: Setting `max_steps = None` is deprecated in v1.5 and will no longer be supported in v1.7. Use `max_steps = -1` instead.
      rank_zero_deprecation(

[NeMo W 2022-05-24 08:10:46 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:96: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=10)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.
      rank_zero_deprecation(

GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[NeMo I 2022-05-24 08:10:46 exp_manager:287] Experiments will be logged at /mnt/STT/NeMo/examples/asr/asr_transducer/nemo_experiments/Conformer-Transducer-Char/2022-05-24_08-10-46
[NeMo I 2022-05-24 08:10:46 exp_manager:661] TensorboardLogger has been set up
[NeMo W 2022-05-24 08:10:46 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:2302: LightningDeprecationWarning: `Trainer.weights_save_path` has been deprecated in v1.6 and will be removed in v1.8.
      rank_zero_deprecation("`Trainer.weights_save_path` has been deprecated in v1.6 and will be removed in v1.8.")

[NeMo W 2022-05-24 08:10:46 exp_manager:895] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to -1. Please ensure that max_steps will run for at least 1 epochs to ensure that checkpointing will not error out.
Created a temporary directory at /tmp/tmphete5x3l
Writing /tmp/tmphete5x3l/_remote_module_non_sriptable.py
[NeMo I 2022-05-24 08:11:07 collections:186] Dataset loaded with 592074 files totalling 824.40 hours
[NeMo I 2022-05-24 08:11:07 collections:187] 19459 files were filtered totalling 113.86 hours
[NeMo I 2022-05-24 08:11:07 collections:186] Dataset loaded with 2519 files totalling 3.88 hours
[NeMo I 2022-05-24 08:11:07 collections:187] 0 files were filtered totalling 0.00 hours
[NeMo I 2022-05-24 08:11:07 collections:186] Dataset loaded with 54 files totalling 0.04 hours
[NeMo I 2022-05-24 08:11:07 collections:187] 0 files were filtered totalling 0.00 hours
[NeMo I 2022-05-24 08:11:07 features:200] PADDING: 0
[NeMo W 2022-05-24 08:11:08 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torch/nn/modules/rnn.py:62: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.1 and num_layers=1
      warnings.warn("dropout option adds dropout after all but last "

[NeMo I 2022-05-24 08:11:08 rnnt_models:203] Using RNNT Loss : warprnnt_numba
    Loss warprnnt_numba_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0}
[NeMo I 2022-05-24 08:11:09 modelPT:1085] Model checkpoint restored from pytorch lightning chackpoint with path : `/mnt/STT/NeMo/save_checkpoint/Conformer-Transducer-BPE--val_wer0.4882-epoch19-last.ckpt`
[NeMo I 2022-05-24 08:11:09 rnnt_models:203] Using RNNT Loss : warprnnt_numba
    Loss warprnnt_numba_kwargs: {'fastemit_lambda': 0.0, 'clamp': -1.0}
[NeMo I 2022-05-24 08:11:09 rnnt_models:391] Changed decoder to output to ['<unk>', ' ', '▁', 'ᄌ', 'ᄀ', 'ᄁ', 'ᄂ', 'ᄃ', 'ᄄ', 'ᄅ', 'ᄆ', 'ᄇ', 'ᄈ', 'ᄉ', 'ᄊ', 'ᄋ', 'ᄍ', 'ᄎ', 'ᄏ', 'ᄐ', 'ᄑ', 'ᄒ', 'ᅡ', 'ᅢ', 'ᅣ', 'ᅤ', 'ᅥ', 'ᅦ', 'ᅧ', 'ᅨ', 'ᅩ', 'ᅪ', 'ᅫ', 'ᅬ', 'ᅭ', 'ᅮ', 'ᅯ', 'ᅰ', 'ᅱ', 'ᅲ', 'ᅳ', 'ᅴ', 'ᅵ', 'ᆨ', 'ᆩ', 'ᆪ', 'ᆫ', 'ᆬ', 'ᆭ', 'ᆮ', 'ᆯ', 'ᆰ', 'ᆱ', 'ᆲ', 'ᆴ', 'ᆶ', 'ᆷ', 'ᆸ', 'ᆹ', 'ᆺ', 'ᆻ', 'ᆼ', 'ᆽ', 'ᆾ', 'ᆿ', 'ᇀ', 'ᇁ', 'ᇂ'] vocabulary.
[NeMo I 2022-05-24 08:11:30 collections:186] Dataset loaded with 592074 files totalling 824.40 hours
[NeMo I 2022-05-24 08:11:30 collections:187] 19459 files were filtered totalling 113.86 hours
[NeMo I 2022-05-24 08:11:30 collections:186] Dataset loaded with 2519 files totalling 3.88 hours
[NeMo I 2022-05-24 08:11:30 collections:187] 0 files were filtered totalling 0.00 hours
[NeMo I 2022-05-24 08:11:30 collections:186] Dataset loaded with 54 files totalling 0.04 hours
[NeMo I 2022-05-24 08:11:30 collections:187] 0 files were filtered totalling 0.00 hours
[NeMo I 2022-05-24 08:11:30 modelPT:579] Optimizer config = AdamW (
    Parameter Group 0
        amsgrad: False
        betas: [0.9, 0.999]
        eps: 1e-08
        foreach: None
        lr: 0.001
        maximize: False
        weight_decay: 0.0001
    )
[NeMo I 2022-05-24 08:11:30 lr_scheduler:833] Scheduler "<nemo.core.optim.lr_scheduler.NoamAnnealing object at 0x7efae0ac9f40>" 
    will be used during training (effective maximum steps = 246700) - 
    Parameters : 
    (d_model: 512
    warmup_steps: 2000
    warmup_ratio: null
    min_lr: 1.0e-09
    max_steps: 246700
    )
[NeMo I 2022-05-24 08:11:30 speech_to_text_rnnt:105] Hydra config: name: Conformer-Transducer-Char
    model:
      sample_rate: 16000
      compute_eval_loss: false
      log_prediction: true
      skip_nan_grad: false
      change_vocab: true
      use_cer: true
      labels:
      - <unk>
      - <pad>
      - ▁
      - ᄌ
      - ᄀ
      - ᄁ
      - ᄂ
      - ᄃ
      - ᄄ
      - ᄅ
      - ᄆ
      - ᄇ
      - ᄈ
      - ᄉ
      - ᄊ
      - ᄋ
      - ᄍ
      - ᄎ
      - ᄏ
      - ᄐ
      - ᄑ
      - ᄒ
      - ᅡ
      - ᅢ
      - ᅣ
      - ᅤ
      - ᅥ
      - ᅦ
      - ᅧ
      - ᅨ
      - ᅩ
      - ᅪ
      - ᅫ
      - ᅬ
      - ᅭ
      - ᅮ
      - ᅯ
      - ᅰ
      - ᅱ
      - ᅲ
      - ᅳ
      - ᅴ
      - ᅵ
      - ᆨ
      - ᆩ
      - ᆪ
      - ᆫ
      - ᆬ
      - ᆭ
      - ᆮ
      - ᆯ
      - ᆰ
      - ᆱ
      - ᆲ
      - ᆴ
      - ᆶ
      - ᆷ
      - ᆸ
      - ᆹ
      - ᆺ
      - ᆻ
      - ᆼ
      - ᆽ
      - ᆾ
      - ᆿ
      - ᇀ
      - ᇁ
      - ᇂ
      model_defaults:
        enc_hidden: ${model.encoder.d_model}
        pred_hidden: 640
        joint_hidden: 640
      train_ds:
        manifest_filepath: /mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_train.json
        sample_rate: 16000
        batch_size: 16
        shuffle: true
        num_workers: 8
        pin_memory: true
        trim_silence: true
        max_duration: 16.7
        min_duration: 0.1
        is_tarred: false
        tarred_audio_filepaths: null
        shuffle_n: 2048
        bucketing_strategy: synced_randomized
        bucketing_batch_size: null
        labels:
        - <unk>
        - ' '
        - ▁
        - ᄌ
        - ᄀ
        - ᄁ
        - ᄂ
        - ᄃ
        - ᄄ
        - ᄅ
        - ᄆ
        - ᄇ
        - ᄈ
        - ᄉ
        - ᄊ
        - ᄋ
        - ᄍ
        - ᄎ
        - ᄏ
        - ᄐ
        - ᄑ
        - ᄒ
        - ᅡ
        - ᅢ
        - ᅣ
        - ᅤ
        - ᅥ
        - ᅦ
        - ᅧ
        - ᅨ
        - ᅩ
        - ᅪ
        - ᅫ
        - ᅬ
        - ᅭ
        - ᅮ
        - ᅯ
        - ᅰ
        - ᅱ
        - ᅲ
        - ᅳ
        - ᅴ
        - ᅵ
        - ᆨ
        - ᆩ
        - ᆪ
        - ᆫ
        - ᆬ
        - ᆭ
        - ᆮ
        - ᆯ
        - ᆰ
        - ᆱ
        - ᆲ
        - ᆴ
        - ᆶ
        - ᆷ
        - ᆸ
        - ᆹ
        - ᆺ
        - ᆻ
        - ᆼ
        - ᆽ
        - ᆾ
        - ᆿ
        - ᇀ
        - ᇁ
        - ᇂ
      validation_ds:
        manifest_filepath: /mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_dev.json
        sample_rate: 16000
        batch_size: 16
        shuffle: false
        num_workers: 8
        pin_memory: true
      test_ds:
        manifest_filepath: /mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_test.json
        sample_rate: 16000
        batch_size: 16
        shuffle: false
        num_workers: 8
        pin_memory: true
        labels:
        - <unk>
        - ' '
        - ▁
        - ᄌ
        - ᄀ
        - ᄁ
        - ᄂ
        - ᄃ
        - ᄄ
        - ᄅ
        - ᄆ
        - ᄇ
        - ᄈ
        - ᄉ
        - ᄊ
        - ᄋ
        - ᄍ
        - ᄎ
        - ᄏ
        - ᄐ
        - ᄑ
        - ᄒ
        - ᅡ
        - ᅢ
        - ᅣ
        - ᅤ
        - ᅥ
        - ᅦ
        - ᅧ
        - ᅨ
        - ᅩ
        - ᅪ
        - ᅫ
        - ᅬ
        - ᅭ
        - ᅮ
        - ᅯ
        - ᅰ
        - ᅱ
        - ᅲ
        - ᅳ
        - ᅴ
        - ᅵ
        - ᆨ
        - ᆩ
        - ᆪ
        - ᆫ
        - ᆬ
        - ᆭ
        - ᆮ
        - ᆯ
        - ᆰ
        - ᆱ
        - ᆲ
        - ᆴ
        - ᆶ
        - ᆷ
        - ᆸ
        - ᆹ
        - ᆺ
        - ᆻ
        - ᆼ
        - ᆽ
        - ᆾ
        - ᆿ
        - ᇀ
        - ᇁ
        - ᇂ
      preprocessor:
        _target_: nemo.collections.asr.modules.AudioToMelSpectrogramPreprocessor
        sample_rate: 16000
        normalize: per_feature
        window_size: 0.025
        window_stride: 0.01
        window: hann
        features: 80
        n_fft: 512
        frame_splicing: 1
        dither: 1.0e-05
        pad_to: 0
      spec_augment:
        _target_: nemo.collections.asr.modules.SpectrogramAugmentation
        freq_masks: 2
        time_masks: 10
        freq_width: 27
        time_width: 0.05
      encoder:
        _target_: nemo.collections.asr.modules.ConformerEncoder
        feat_in: ${model.preprocessor.features}
        feat_out: -1
        n_layers: 17
        d_model: 512
        subsampling: striding
        subsampling_factor: 4
        subsampling_conv_channels: -1
        ff_expansion_factor: 4
        self_attention_model: rel_pos
        n_heads: 8
        att_context_size:
        - -1
        - -1
        xscaling: true
        untie_biases: true
        pos_emb_max_len: 5000
        conv_kernel_size: 31
        conv_norm_type: layer_norm
        dropout: 0.1
        dropout_emb: 0.0
        dropout_att: 0.1
      decoder:
        _target_: nemo.collections.asr.modules.RNNTDecoder
        normalization_mode: null
        random_state_sampling: false
        blank_as_pad: true
        prednet:
          pred_hidden: ${model.model_defaults.pred_hidden}
          pred_rnn_layers: 1
          t_max: null
          dropout: 0.1
      joint:
        _target_: nemo.collections.asr.modules.RNNTJoint
        log_softmax: null
        preserve_memory: false
        fuse_loss_wer: true
        fused_batch_size: 16
        jointnet:
          joint_hidden: ${model.model_defaults.joint_hidden}
          activation: relu
          dropout: 0.1
      decoding:
        strategy: greedy_batch
        greedy:
          max_symbols: 30
        beam:
          beam_size: 2
          return_best_hypothesis: false
          score_norm: true
          tsd_max_sym_exp: 50
          alsd_max_target_len: 2.0
      loss:
        loss_name: default
        warprnnt_numba_kwargs:
          fastemit_lambda: 0.0
          clamp: -1.0
      variational_noise:
        start_step: 0
        std: 0.0
      optim:
        name: adamw
        lr: 0.001
        betas:
        - 0.9
        - 0.999
        weight_decay: 0.0001
        sched:
          name: NoamAnnealing
          d_model: ${model.encoder.d_model}
          warmup_steps: 2000
          warmup_ratio: null
          min_lr: 1.0e-09
    trainer:
      devices: -1
      num_nodes: 1
      max_epochs: 20
      max_steps: null
      val_check_interval: 1.0
      accelerator: gpu
      strategy: ddp
      accumulate_grad_batches: 1
      gradient_clip_val: 0.0
      precision: 32
      log_every_n_steps: 20
      progress_bar_refresh_rate: 10
      resume_from_checkpoint: null
      num_sanity_val_steps: 0
      check_val_every_n_epoch: 1
      sync_batchnorm: true
      enable_checkpointing: false
      logger: false
    exp_manager:
      exp_dir: null
      name: ${name}
      create_tensorboard_logger: true
      create_checkpoint_callback: true
      checkpoint_callback_params:
        monitor: val_wer
        mode: min
        save_top_k: 1
        always_save_nemo: true
      resume_if_exists: false
      resume_ignore_no_checkpoint: false
      create_wandb_logger: false
      wandb_logger_kwargs:
        name: null
        project: null

Created a temporary directory at /tmp/tmp8qcslmlj
Writing /tmp/tmp8qcslmlj/_remote_module_non_sriptable.py
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/3
Created a temporary directory at /tmp/tmprpg9r0ft
Writing /tmp/tmprpg9r0ft/_remote_module_non_sriptable.py
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/3
Added key: store_based_barrier_key:1 to store for rank: 1
Initializing distributed: GLOBAL_RANK: 2, MEMBER: 3/3
Added key: store_based_barrier_key:1 to store for rank: 2
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 3 processes
----------------------------------------------------------------------------------------------------

Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 3 nodes.
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [1,2,3]
LOCAL_RANK: 2 - CUDA_VISIBLE_DEVICES: [1,2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1,2,3]
**_[NeMo I 2022-05-24 08:12:24 modelPT:457] No optimizer config provided, therefore no optimizer was created_**

titu1994 commented 2 years ago

Weird - it says your optimizer was setup. Also, <unk> and <pad> have no relevance in character based models. They will never be used unless your text actually includes those exact tokens. If you are using tokenizer then use such tokens, otherwise remove them from your char models. Mixing of both may cause errors in training.

[NeMo I 2022-05-24 08:11:30 modelPT:579] Optimizer config = AdamW (
    Parameter Group 0
        amsgrad: False
        betas: [0.9, 0.999]
        eps: 1e-08
        foreach: None
        lr: 0.001
        maximize: False
        weight_decay: 0.0001
    )
[NeMo I 2022-05-24 08:11:30 lr_scheduler:833] Scheduler "<nemo.core.optim.lr_scheduler.NoamAnnealing object at 0x7efae0ac9f40>" 
    will be used during training (effective maximum steps = 246700) - 
    Parameters : 
    (d_model: 512
    warmup_steps: 2000
    warmup_ratio: null
    min_lr: 1.0e-09
    max_steps: 246700
    )

Your config seems fine, and code is correct. Does it just stop after showing that message? What about 1 node, does it work ?

dykim3 commented 2 years ago

I've struggled with optimizer issue, it didn't load optimizer when I load checkpoint.. but after I've change the code to this it works fine..

@hydra_runner(config_path="experimental/contextnet_rnnt", config_name="config_rnnt")
def main(cfg):
    # logging.info(f'Hydra config: {OmegaConf.to_yaml(cfg)}')

    trainer = pl.Trainer(**cfg.trainer)
    exp_manager(trainer, cfg.get("exp_manager", None))
    asr_model = EncDecRNNTModel(cfg=cfg.model, trainer=trainer)

    # Initialize the weights of the model from another model, if provided via config
    if cfg.model.change_vocab:
        asr_model.change_vocabulary(new_vocabulary=list(korean_vocab))
        asr_model.cfg.labels = list(korean_vocab)
        asr_model.setup_training_data(cfg.model.train_ds)
        asr_model.setup_multiple_validation_data(cfg.model.validation_ds)
        asr_model.setup_test_data(cfg.model.test_ds)
        asr_model.set_trainer(trainer)
        # asr_model.cfg = asr_model._cfg
        asr_model.preprocessor = asr_model.from_config_dict(cfg.model.preprocessor)
        asr_model.setup_optimization(optim_config=cfg.model.optim)
        asr_model.spec_augment = asr_model.from_config_dict(cfg.model.spec_augment)

    asr_model.maybe_init_from_pretrained_checkpoint(cfg)

    logging.info(f'Hydra config: {asr_model.cfg}')

    trainer.fit(asr_model)

but I came with another problem...after 10% training the code shows embedding error.. which wasn't occured with the same manifest used to train rnnt_bpe model.

/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [576,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [575,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [574,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [5,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [6,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [7,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [8,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [9,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [10,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [11,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [12,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [13,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [14,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [15,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [16,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [17,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [18,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [19,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [20,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [21,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [22,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [23,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [24,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [25,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [26,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [67,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [68,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [69,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [70,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [71,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [72,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [73,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [74,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [75,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [76,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [77,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [78,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [79,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [80,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [81,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [82,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [83,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [84,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [85,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [86,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [87,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [88,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [89,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [90,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [91,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [92,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [93,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [94,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [95,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [96,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [97,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [98,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [99,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [100,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [101,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [102,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [103,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [104,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [105,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [106,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [107,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [108,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [109,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [110,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [111,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [112,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [113,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [114,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [115,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [116,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [117,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [118,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [119,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [120,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [121,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [122,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [123,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [124,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [572,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [34,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [36,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [37,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [38,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [39,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [40,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [41,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [42,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [43,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [44,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [45,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [46,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [47,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [48,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [49,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [50,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [51,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [52,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [53,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [54,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [55,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [56,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [57,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [58,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [59,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [60,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [61,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [62,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/pytorch/pytorch/aten/src/ATen/native/cuda/Indexing.cu:703: indexSelectLargeIndex: block: [573,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Error executing job with overrides: ['init_from_ptl_ckpt=/mnt/STT/NeMo/save_checkpoint/Conformer-Transducer-BPE--val_wer0.4882-epoch19-last.ckpt', 'model.train_ds.manifest_filepath=/mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_train.json', 'model.validation_ds.manifest_filepath=/mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_dev.json', 'model.test_ds.manifest_filepath=/mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_test.json', 'model.use_cer=True', 'model.log_prediction=True', 'model.change_vocab=True', 'trainer.devices=-1', 'trainer.accelerator=gpu', 'trainer.strategy=ddp', 'trainer.log_every_n_steps=20', 'trainer.max_epochs=20', 'model.optim.name=adamw', 'model.optim.lr=0.001', 'model.optim.betas=[0.9,0.999]', 'model.optim.weight_decay=0.0001', 'model.optim.sched.warmup_steps=2000', 'model.optim.sched.min_lr=1e-9']
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 194, in forward
    g, states = self.predict(y, state=states, add_sos=add_sos)  # (B, U, D)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 267, in predict
    y = self.prediction["embed"](y)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2148, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train
    self.fit_loop.run()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 268, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance
    batch_output = self.batch_loop.run(batch, batch_idx)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
    outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance
    result = self._run_optimization(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step
    self.trainer._call_lightning_module_hook(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1593, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1644, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 278, in optimizer_step
    optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step
    return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 155, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/optimizer.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/adamw.py", line 112, in step
    loss = closure()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 140, in _wrap_closure
    closure_result = closure()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure
    step_output = self._step_fn()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step
    training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1763, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 341, in training_step
    return self.model(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 992, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
    output = self.module.training_step(*inputs, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/utils/model_utils.py", line 364, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/rnnt_models.py", line 674, in training_step
    decoder, target_length, states = self.decoder(targets=transcript, target_length=transcript_len)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 961, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 201, in forward
    logging.info(f'targets: {targets} target_length: {target_length} y: {y.shape[-1]} states: {states} add_sos: {add_sos}')
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 664, in __format__
    return object.__format__(self, format_spec)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 341, in __repr__
    return torch._tensor_str._str(self)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 434, in _str
    return _str_intern(self)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 409, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 264, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 296, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 296, in <listcomp>
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 289, in get_summarized_data
    return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/STT/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt.py", line 106, in main
    trainer.fit(asr_model)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 734, in _call_and_handle_interrupt
    self.strategy.reconciliate_processes(traceback.format_exc())
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 443, in reconciliate_processes
    raise DeadlockDetectedException(f"DeadLock detected from rank: {self.global_rank} \n {trace}")
pytorch_lightning.utilities.exceptions.DeadlockDetectedException: DeadLock detected from rank: 1 
 Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 194, in forward
    g, states = self.predict(y, state=states, add_sos=add_sos)  # (B, U, D)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 267, in predict
    y = self.prediction["embed"](y)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
    return F.embedding(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py", line 2148, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: CUDA error: device-side assert triggered

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train
    self.fit_loop.run()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 268, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance
    batch_output = self.batch_loop.run(batch, batch_idx)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance
    outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance
    result = self._run_optimization(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step
    self.trainer._call_lightning_module_hook(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1593, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1644, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step
    step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 278, in optimizer_step
    optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step
    return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 155, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/optimizer.py", line 88, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/optim/adamw.py", line 112, in step
    loss = closure()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 140, in _wrap_closure
    closure_result = closure()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in __call__
    self._result = self.closure(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure
    step_output = self._step_fn()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step
    training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values())
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1763, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 341, in training_step
    return self.model(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 992, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
    output = self.module.training_step(*inputs, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/utils/model_utils.py", line 364, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/rnnt_models.py", line 674, in training_step
    decoder, target_length, states = self.decoder(targets=transcript, target_length=transcript_len)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1111, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 961, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/modules/rnnt.py", line 201, in forward
    logging.info(f'targets: {targets} target_length: {target_length} y: {y.shape[-1]} states: {states} add_sos: {add_sos}')
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 664, in __format__
    return object.__format__(self, format_spec)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 341, in __repr__
    return torch._tensor_str._str(self)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 434, in _str
    return _str_intern(self)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 409, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 264, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 296, in get_summarized_data
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 296, in <listcomp>
    return torch.stack([get_summarized_data(x) for x in (start + end)])
  File "/opt/conda/lib/python3.8/site-packages/torch/_tensor_str.py", line 289, in get_summarized_data
    return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
RuntimeError: CUDA error: device-side assert triggered

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
script/run_conformer_transducer_char.sh: line 26: 344835 Killed                  CUDA_LAUNCH_BLOCKING=1 CUDA_VISIBLE_DEVICES="1,2,3" python speech_to_text_rnnt.py --config-path=/mnt/STT/NeMo/examples/asr/conf/conformer --config-name=conformer_transducer_char_kr init_from_ptl_ckpt='/mnt/STT/NeMo/save_checkpoint/Conformer-Transducer-BPE--val_wer0.4882-epoch19-last.ckpt' model.train_ds.manifest_filepath=/mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_train.json model.validation_ds.manifest_filepath=/mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_dev.json model.test_ds.manifest_filepath=/mnt/STT/NeMo/tutorials/asr/manifests/kr/ksponspeech_manifest_test.json model.use_cer=True model.log_prediction=True model.change_vocab=True trainer.devices=-1 trainer.accelerator="gpu" trainer.strategy="ddp" trainer.log_every_n_steps=20 trainer.max_epochs=20 model.optim.name="adamw" model.optim.lr=0.001 model.optim.betas=[0.9,0.999] model.optim.weight_decay=0.0001 model.optim.sched.warmup_steps=2000 model.optim.sched.min_lr=1e-9
[W CUDAGuardImpl.h:113] Warning: CUDA warning: device-side assert triggered (function destroyEvent)
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/pytorch/pytorch/c10/cuda/CUDACachingAllocator.cpp:1230 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6c (0x7fc10956a1dc in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1fa26 (0x7fc1095c1a26 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x247 (0x7fc1095c7ec7 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x43d67c (0x7fc13e0df67c in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #4: c10::TensorImpl::release_resources() + 0x175 (0x7fc109553035 in /opt/conda/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #5: <unknown function> + 0xf2a5f9 (0x7fc1359125f9 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::Reducer::~Reducer() + 0x1ef (0x7fc13852269f in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x16 (0x7fc13e65bb96 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #8: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x48 (0x7fc13dfdc658 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x9bd8d5 (0x7fc13e65f8d5 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0x342ea2 (0x7fc13dfe4ea2 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #11: <unknown function> + 0x343db3 (0x7fc13dfe5db3 in /opt/conda/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #12: <unknown function> + 0x1169ad (0x55d1147cc9ad in /opt/conda/bin/python)
frame #13: <unknown function> + 0x14057c (0x55d1147f657c in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1169ad (0x55d1147cc9ad in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1fbfe1 (0x55d1148b1fe1 in /opt/conda/bin/python)
frame #16: <unknown function> + 0x10e907 (0x55d1147c4907 in /opt/conda/bin/python)
frame #17: <unknown function> + 0x1d529e (0x55d11488b29e in /opt/conda/bin/python)
frame #18: PyGC_Collect + 0x56 (0x55d1148b14f6 in /opt/conda/bin/python)
frame #19: Py_FinalizeEx + 0x75 (0x55d1148b0b05 in /opt/conda/bin/python)
frame #20: Py_Exit + 0x8 (0x55d1147474a1 in /opt/conda/bin/python)
frame #21: <unknown function> + 0x91400 (0x55d114747400 in /opt/conda/bin/python)
frame #22: <unknown function> + 0x910d3 (0x55d1147470d3 in /opt/conda/bin/python)
frame #23: PyRun_SimpleFileExFlags + 0x390 (0x55d11474fa4f in /opt/conda/bin/python)
frame #24: <unknown function> + 0x8d0ac (0x55d1147430ac in /opt/conda/bin/python)
frame #25: Py_BytesMain + 0x39 (0x55d114884219 in /opt/conda/bin/python)
frame #26: __libc_start_main + 0xf3 (0x7fc17488d083 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #27: <unknown function> + 0x1ce125 (0x55d114884125 in /opt/conda/bin/python)

NVIDIA / NeMo

fine tuning in korean conformer-transducer #4240