padding_idx=masking_idx in ByteNetLMTime instantiation arguments

[cmarkak]

The following code in train.py assigns padding_idx=masking_idx in the model initiation. This is conflicting with the definition above for padding_idx which is different from masking_idx. Is this an oversight or there is a particular reason for this assignment?

padding_idx = tokenizer.pad_id # PROTEIN_ALPHABET.index(PAD) masking_idx = tokenizer.mask_id print('Using {} as padding index'.format(padding_idx)) print('Using {} as masking index'.format(masking_idx)) #if args.model_type == 'ByteNet': model = ByteNetLMTime(n_tokens, d_embed, d_model, n_layers, kernel_size, r, causal=causal, padding_idx=masking_idx, rank=weight_rank, dropout=args.dropout, tie_weights=args.tie_weights, final_ln=args.final_norm, slim=slim, activation=activation, timesteps=diffusion_timesteps)

Thank you in advance

[sarahalamdari]

This is done on purpose. We follow how ESM handles mask tokens. Padding is handled with input_mask