search

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
MIT License
30.48k stars 6.41k forks source link

AttributeError: 'NoneType' object has no attribute 'transpose' during GAN training for wav2vec-U 2.0 #5301

Closed EomSooHwan closed 1 year ago

EomSooHwan commented 1 year ago

❓ Questions and Help

What is your question?

(This is from #5283. I though it is better to separate since it is the new kind of error)

I tried to reproduce wav2vec-U 2.0 with python 3.8. When I ran the GAN training script from README, however, I got the following message. I want to know how to fix this error (either by changing to code or preparing data again etc.)

Traceback (most recent call last):
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/bin/fairseq-hydra-train", line 8, in <module>
    sys.exit(cli_main())
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq_cli/hydra_train.py", line 87, in cli_main
    hydra_main()
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/main.py", line 37, in decorated_main
    strict=strict,
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/utils.py", line 355, in _run_hydra
    lambda: hydra.multirun(
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/utils.py", line 201, in run_and_report
    raise ex
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/utils.py", line 358, in <lambda>
    overrides=args.overrides,
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 136, in multirun
    return sweeper.sweep(arguments=task_overrides)
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 154, in sweep
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/_internal/core_plugins/basic_launcher.py", line 80, in launch
    job_subdir_key="hydra.sweep.subdir",
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/hydra/core/utils.py", line 129, in run_job
    ret.return_value = task_function(task_cfg)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq_cli/hydra_train.py", line 27, in hydra_main
    _hydra_main(cfg)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq_cli/hydra_train.py", line 56, in _hydra_main
    distributed_utils.call_main(cfg, pre_main, **kwargs)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq/distributed/utils.py", line 404, in call_main
    main(cfg, **kwargs)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq_cli/train.py", line 205, in main
    valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq_cli/train.py", line 331, in train
    log_output = trainer.train_step(samples)
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq/trainer.py", line 850, in train_step
    **extra_kwargs,
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq/tasks/fairseq_task.py", line 532, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/fairseq/criterions/model_criterion.py", line 56, in forward
    net_output = model(**sample["net_input"])
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/models/wav2vec_u.py", line 602, in forward
    token_y = self.discriminator(token_x, token_padding_mask)
  File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/models/wav2vec_u.py", line 271, in forward
    x = x.transpose(1, 2)  # BTC -> BCT
AttributeError: 'NoneType' object has no attribute 'transpose'

The log looks like:

[2023-08-29 00:23:31,543][HYDRA] Launching 12 jobs locally
[2023-08-29 00:23:31,543][HYDRA]        #0 : task.data=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/audiofiles/preprocessed_v2/train-clean-100 task.text_data=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/texts/phones task.kenlm_path=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/texts/train-clean-360/phones/lm.phones.filtered.04.bin common.user_dir=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised model.code_penalty=2 model.gradient_penalty=1.5 model.smoothness_weight=0.5 common.seed=0
[2023-08-29 00:23:33,341][fairseq_cli.train][INFO] - {'_name': None, 'common': {'_name': None, 'no_progress_bar': False, 'log_interval': 100, 'log_format': 'json', 'log_file': None, 'aim_repo': None, 'aim_run_hash': None, 'tensorboard_logdir': 'tb', 'wandb_project': None, 'azureml_logging': False, 'seed': 0, 'cpu': False, 'tpu': False, 'bf16': False, 'memory_efficient_bf16': False, 'fp16': False, 'memory_efficient_fp16': False, 'fp16_no_flatten_grads': True, 'fp16_init_scale': 128, 'fp16_scale_window': None, 'fp16_scale_tolerance': 0.0, 'on_cpu_convert_precision': False, 'min_loss_scale': 0.0001, 'threshold_loss_scale': None, 'amp': False, 'amp_batch_retries': 2, 'amp_init_scale': 128, 'amp_scale_window': None, 'user_dir': '/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised', 'empty_cache_freq': 0, 'all_gather_list_size': 16384, 'model_parallel_size': 1, 'quantization_config_path': None, 'profile': False, 'reset_logging': False, 'suppress_crashes': False, 'use_plasma_view': False, 'plasma_path': '/tmp/plasma'}, 'common_eval': {'_name': None, 'path': None, 'post_process': None, 'quiet': False, 'model_overrides': '{}', 'results_path': None}, 'distributed_training': {'_name': None, 'distributed_world_size': 1, 'distributed_num_procs': 4, 'distributed_rank': 0, 'distributed_backend': 'nccl', 'distributed_init_method': None, 'distributed_port': -1, 'device_id': 0, 'distributed_no_spawn': False, 'ddp_backend': 'pytorch_ddp', 'ddp_comm_hook': 'none', 'bucket_cap_mb': 25, 'fix_batches_to_gpus': False, 'find_unused_parameters': False, 'gradient_as_bucket_view': False, 'fast_stat_sync': False, 'heartbeat_timeout': -1, 'broadcast_buffers': False, 'slowmo_momentum': None, 'slowmo_base_algorithm': 'localsgd', 'localsgd_frequency': 3, 'nprocs_per_node': 4, 'pipeline_model_parallel': False, 'pipeline_balance': None, 'pipeline_devices': None, 'pipeline_chunks': 0, 'pipeline_encoder_balance': None, 'pipeline_encoder_devices': None, 'pipeline_decoder_balance': None, 'pipeline_decoder_devices': None, 'pipeline_checkpoint': 'never', 'zero_sharding': 'none', 'fp16': False, 'memory_efficient_fp16': False, 'tpu': False, 'no_reshard_after_forward': False, 'fp32_reduce_scatter': False, 'cpu_offload': False, 'use_sharded_state': False, 'not_fsdp_flatten_parameters': False}, 'dataset': {'_name': None, 'num_workers': 6, 'skip_invalid_size_inputs_valid_test': True, 'max_tokens': None, 'batch_size': 160, 'required_batch_size_multiple': 8, 'required_seq_len_multiple': 1, 'dataset_impl': None, 'data_buffer_size': 10, 'train_subset': 'train', 'valid_subset': 'valid', 'combine_valid_subsets': None, 'ignore_unused_valid_subsets': False, 'validate_interval': 1000, 'validate_interval_updates': 1000, 'validate_after_updates': 0, 'fixed_validation_seed': None, 'disable_validation': False, 'max_tokens_valid': None, 'batch_size_valid': 160, 'max_valid_steps': None, 'curriculum': 0, 'gen_subset': 'test', 'num_shards': 1, 'shard_id': 0, 'grouped_shuffling': False, 'update_epoch_batch_itr': False, 'update_ordered_indices_seed': False}, 'optimization': {'_name': None, 'max_epoch': 0, 'max_update': 150000, 'stop_time_hours': 0.0, 'clip_norm': 5.0, 'sentence_avg': False, 'update_freq': [1], 'lr': [0.0], 'stop_min_lr': -1.0, 'use_bmuf': False, 'skip_remainder_batch': False, 'debug_param_names': False}, 'checkpoint': {'_name': None, 'save_dir': '.', 'restore_file': 'checkpoint_last.pt', 'continue_once': None, 'finetune_from_model': None, 'reset_dataloader': False, 'reset_lr_scheduler': False, 'reset_meters': False, 'reset_optimizer': False, 'optimizer_overrides': '{}', 'save_interval': 1000, 'save_interval_updates': 1000, 'keep_interval_updates': -1, 'keep_interval_updates_pattern': -1, 'keep_last_epochs': -1, 'keep_best_checkpoints': -1, 'no_save': False, 'no_epoch_checkpoints': True, 'no_last_checkpoints': False, 'no_save_optimizer_state': False, 'best_checkpoint_metric': 'weighted_lm_ppl', 'maximize_best_checkpoint_metric': False, 'patience': -1, 'checkpoint_suffix': '', 'checkpoint_shard_count': 1, 'load_checkpoint_on_all_dp_ranks': False, 'write_checkpoints_asynchronously': False, 'model_parallel_size': 1}, 'bmuf': {'_name': None, 'block_lr': 1.0, 'block_momentum': 0.875, 'global_sync_iter': 50, 'warmup_iterations': 500, 'use_nbm': False, 'average_sync': False, 'distributed_world_size': 1}, 'generation': {'_name': None, 'beam': 5, 'beam_mt': 0, 'nbest': 1, 'max_len_a': 0.0, 'max_len_b': 200, 'max_len_a_mt': 0.0, 'max_len_b_mt': 200, 'min_len': 1, 'match_source_len': False, 'unnormalized': False, 'no_early_stop': False, 'no_beamable_mm': False, 'lenpen': 1.0, 'lenpen_mt': 1.0, 'unkpen': 0.0, 'replace_unk': None, 'sacrebleu': False, 'score_reference': False, 'prefix_size': 0, 'no_repeat_ngram_size': 0, 'sampling': False, 'sampling_topk': -1, 'sampling_topp': -1.0, 'constraints': None, 'temperature': 1.0, 'diverse_beam_groups': -1, 'diverse_beam_strength': 0.5, 'diversity_rate': -1.0, 'print_alignment': None, 'print_step': False, 'lm_path': None, 'lm_weight': 0.0, 'iter_decode_eos_penalty': 0.0, 'iter_decode_max_iter': 10, 'iter_decode_force_max_iter': False, 'iter_decode_with_beam': 1, 'iter_decode_with_external_reranker': False, 'retain_iter_history': False, 'retain_dropout': False, 'retain_dropout_modules': None, 'decoding_format': None, 'no_seed_provided': False, 'eos_token': None}, 'eval_lm': {'_name': None, 'output_word_probs': False, 'output_word_stats': False, 'context_window': 0, 'softmax_batch': 9223372036854775807}, 'interactive': {'_name': None, 'buffer_size': 0, 'input': '-'}, 'model': {'_name': 'wav2vec_u', 'discriminator_kernel': 8, 'discriminator_dilation': 1, 'discriminator_dim': 384, 'discriminator_causal': True, 'discriminator_linear_emb': False, 'discriminator_depth': 2, 'discriminator_max_pool': False, 'discriminator_act_after_linear': False, 'discriminator_dropout': 0.0, 'discriminator_spectral_norm': False, 'discriminator_weight_norm': False, 'generator_kernel': 9, 'generator_dilation': 1, 'generator_stride': 3, 'generator_pad': -1, 'generator_bias': False, 'generator_dropout': 0.1, 'generator_batch_norm': 30, 'generator_residual': True, 'blank_weight': 0.0, 'blank_mode': 'add', 'blank_is_sil': False, 'no_softmax': False, 'smoothness_weight': 0.5, 'smoothing': 0.0, 'smoothing_one_sided': False, 'gradient_penalty': 1.5, 'probabilistic_grad_penalty_slicing': False, 'code_penalty': 2.0, 'mmi_weight': 0.5, 'target_dim': 64, 'target_downsample_rate': 2, 'gumbel': False, 'hard_gumbel': False, 'temp': [2.0, 0.1, 0.99995], 'input_dim': 1024, 'segmentation': {'_name': None, 'type': <SegmentationType.JOIN: 5>, 'subsample_rate': 0.25, 'mean_pool': True, 'mean_pool_join': False, 'remove_zeros': False}}, 'task': {'_name': 'unpaired_audio_text', 'data': '/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/audiofiles/preprocessed_v2/train-clean-100', 'text_data': '/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/texts/phones', 'max_length': None, 'labels': 'phn', 'aux_target_postfix': 'km', 'unfiltered': False, 'ctc_eval': False, 'sort_by_length': False, 'shuffle': True, 'append_eos': False, 'uppercase': False, 'skipwords': '', 'kenlm_path': '/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/texts/train-clean-360/phones/lm.phones.filtered.04.bin', 'vocab_usage_power': 2.0, 'word_decoder_config': None, 'word_kenlm_path': None, 'decoding_config': {'_name': None, 'kenlm_path': None, 'lm_weight': 0.0, 'blank_weight': 0.0}}, 'criterion': {'_name': 'model', 'loss_weights': {}, 'log_keys': ['accuracy_dense', 'accuracy_token', 'temp', 'code_ppl'], 'can_sum': True}, 'optimizer': {'_name': 'composite', 'groups': {'generator': {'lr': [5e-05], 'lr_float': None, 'optimizer': {'_name': 'adam', 'adam_betas': [0.5, 0.98], 'adam_eps': 1e-06, 'lr': [5e-05], 'weight_decay': 0}, 'lr_scheduler': {'_name': 'fixed', 'warmup_updates': 0}}, 'discriminator': {'lr': [0.0003], 'lr_float': None, 'optimizer': {'_name': 'adam', 'adam_betas': [0.5, 0.98], 'adam_eps': 1e-06, 'lr': [0.0003], 'weight_decay': 0.0001}, 'lr_scheduler': {'_name': 'fixed', 'warmup_updates': 0}}}, 'dynamic_groups': False}, 'lr_scheduler': {'_name': 'pass_through'}, 'scoring': None, 'bpe': None, 'tokenizer': None, 'ema': {'_name': None, 'store_ema': False, 'ema_decay': 0.9999, 'ema_start_update': 0, 'ema_seed_model': None, 'ema_update_freq': 1, 'ema_fp32': False}, 'job_logging_cfg': {'version': 1, 'formatters': {'simple': {'format': '[%(asctime)s][%(name)s][%(levelname)s] - %(message)s'}}, 'handlers': {'console': {'class': 'logging.StreamHandler', 'formatter': 'simple', 'stream': 'ext://sys.stdout'}, 'file': {'class': 'logging.FileHandler', 'formatter': 'simple', 'filename': 'hydra_train.log'}}, 'root': {'level': 'INFO', 'handlers': ['console', 'file']}, 'disable_existing_loggers': False}}
[2023-08-29 00:23:33,395][fairseq_cli.train][INFO] - Wav2vec_U(
  (discriminator): Discriminator(
    (net): Sequential(
      (0): Conv1d(44, 384, kernel_size=(8,), stride=(1,), padding=(7,))
      (1): SamePad()
      (2): Dropout(p=0.0, inplace=False)
      (3): Sequential(
        (0): Conv1d(384, 384, kernel_size=(8,), stride=(1,), padding=(7,))
        (1): SamePad()
        (2): Dropout(p=0.0, inplace=False)
        (3): GELU(approximate='none')
      )
      (4): Conv1d(384, 1, kernel_size=(8,), stride=(1,), padding=(7,))
      (5): SamePad()
    )
  )
  (segmenter): JoinSegmenter()
  (generator): Generator(
    (dropout): Dropout(p=0.1, inplace=False)
    (proj): Sequential(
      (0): TransposeLast()
      (1): Conv1d(1024, 44, kernel_size=(9,), stride=(3,), padding=(4,), bias=False)
      (2): TransposeLast()
    )
    (bn): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (in_proj): Linear(in_features=1024, out_features=1024, bias=True)
  )
  (decoder): Linear(in_features=1024, out_features=64, bias=True)
)
[2023-08-29 00:23:33,395][fairseq_cli.train][INFO] - task: UnpairedAudioText
[2023-08-29 00:23:33,395][fairseq_cli.train][INFO] - model: Wav2vec_U
[2023-08-29 00:23:33,395][fairseq_cli.train][INFO] - criterion: ModelCriterion
[2023-08-29 00:23:33,395][fairseq_cli.train][INFO] - num. shared model params: 2,841,409 (num. trained: 2,841,409)
[2023-08-29 00:23:33,395][fairseq_cli.train][INFO] - num. expert model params: 0 (num. trained: 0)
[2023-08-29 00:23:33,398][unsupervised.data.extracted_features_dataset][INFO] - auxaliry target for valid missing
[2023-08-29 00:23:33,398][unsupervised.data.extracted_features_dataset][INFO] - loaded 296, skipped 0 samples
[2023-08-29 00:23:33,398][unsupervised.tasks.unpaired_audio_text][INFO] - split valid has unpaired text? False
[2023-08-29 00:23:35,072][fairseq.utils][INFO] - ***********************CUDA enviroments for all 1 workers***********************
[2023-08-29 00:23:35,072][fairseq.utils][INFO] - rank   0: capabilities =  7.5  ; total memory = 47.462 GB ; name = Quadro RTX 8000
[2023-08-29 00:23:35,072][fairseq.utils][INFO] - ***********************CUDA enviroments for all 1 workers***********************
[2023-08-29 00:23:35,072][fairseq_cli.train][INFO] - training on 1 devices (GPUs/TPUs)
[2023-08-29 00:23:35,072][fairseq_cli.train][INFO] - max tokens per device = None and max sentences per device = 160
[2023-08-29 00:23:35,074][fairseq.trainer][INFO] - Preparing to load checkpoint ./checkpoint_last.pt
[2023-08-29 00:23:35,074][fairseq.trainer][INFO] - No existing checkpoint found ./checkpoint_last.pt
[2023-08-29 00:23:35,074][fairseq.trainer][INFO] - loading train data for epoch 1
[2023-08-29 00:23:42,677][unsupervised.data.extracted_features_dataset][INFO] - loaded 28243, skipped 0 samples
[2023-08-29 00:23:42,677][unsupervised.tasks.unpaired_audio_text][INFO] - split train has unpaired text? False
[2023-08-29 00:23:42,679][fairseq.tasks.fairseq_task][INFO] - can_reuse_epoch_itr = True
[2023-08-29 00:23:42,680][fairseq.tasks.fairseq_task][INFO] - reuse_dataloader = True
[2023-08-29 00:23:42,680][fairseq.tasks.fairseq_task][INFO] - rebuild_batches = False
[2023-08-29 00:23:42,680][fairseq.tasks.fairseq_task][INFO] - creating new batches for epoch 1
[2023-08-29 00:23:43,020][fairseq.trainer][INFO] - NOTE: your device may support faster training with --fp16 or --amp
[2023-08-29 00:23:43,225][fairseq.optim.composite][INFO] - Optimizers for different groups are as below
[2023-08-29 00:23:43,226][fairseq.optim.composite][INFO] - Group : discriminator:{'_name': 'adam', 'adam_betas': [0.5, 0.98], 'adam_eps': 1e-06, 'lr': [0.0003], 'weight_decay': 0.0001}
[2023-08-29 00:23:43,226][fairseq.optim.composite][INFO] - Group : generator:{'_name': 'adam', 'adam_betas': [0.5, 0.98], 'adam_eps': 1e-06, 'lr': [5e-05], 'weight_decay': 0}
[2023-08-29 00:23:43,228][fairseq_cli.train][INFO] - begin dry-run validation on "valid" subset
[2023-08-29 00:23:43,229][fairseq.tasks.fairseq_task][INFO] - can_reuse_epoch_itr = True
[2023-08-29 00:23:43,229][fairseq.tasks.fairseq_task][INFO] - reuse_dataloader = True
[2023-08-29 00:23:43,229][fairseq.tasks.fairseq_task][INFO] - rebuild_batches = False
[2023-08-29 00:23:43,229][fairseq.tasks.fairseq_task][INFO] - creating new batches for epoch 1
[2023-08-29 00:23:45,282][fairseq.data.iterators][INFO] - grouped total_num_itrs = 177
[2023-08-29 00:23:45,287][fairseq.logging.progress_bar][WARNING] - tensorboard not found, please install with: pip install tensorboard
[2023-08-29 00:23:45,295][fairseq.trainer][INFO] - begin training epoch 1
[2023-08-29 00:23:45,296][fairseq_cli.train][INFO] - Start iterating over samples
[2023-08-29 00:23:53,190][fairseq.trainer][INFO] - Saving checkpoint to /mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/checkpoint/w2v_unsup_gan_xp/w2vu2/common.seed:0__model.code_penalty:2__model.gradient_penalty:1.5__model.smoothness_weight:0.5/0/crash.pt
[2023-08-29 00:23:53,260][fairseq.trainer][INFO] - Finished saving checkpoint to /mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/checkpoint/w2v_unsup_gan_xp/w2vu2/common.seed:0__model.code_penalty:2__model.gradient_penalty:1.5__model.smoothness_weight:0.5/0/crash.pt

Code

PREFIX=w2v_unsup_gan_xp
FAIRSEQ_ROOT=/mnt/hdd/esyoon/workspace/ICLR24/fairseq

# For wav2vec-U 2.0, use raw audio features
CONFIG_NAME=w2vu2
TASK_DATA=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/audiofiles/preprocessed_v2/train-clean-100

# Unpaired text input
TEXT_DATA=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/texts/phones  # path to fairseq-preprocessed GAN data (phones dir)
KENLM_PATH=/mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/texts/train-clean-360/phones/lm.phones.filtered.04.bin  # KenLM 4-gram phoneme language model (LM data = GAN data here)

PYTHONPATH=${FAIRSEQ_ROOT} PREFIX=${PREFIX} fairseq-hydra-train \
    -m --config-dir /mnt/hdd/esyoon/workspace/ICLR24/fairseq/examples/wav2vec/unsupervised/config/gan \
    --config-name $CONFIG_NAME \
    task.data=${TASK_DATA} \
    task.text_data=${TEXT_DATA} \
    task.kenlm_path=${KENLM_PATH} \
    common.user_dir=${FAIRSEQ_ROOT}/examples/wav2vec/unsupervised \
    model.code_penalty=2,4 model.gradient_penalty=1.5,2.0 \
    model.smoothness_weight=0.5,0.75,1.0 'common.seed=range(0,1)'

What have you tried?

I found that the same issue was occurred at #3693. Although the issue is closed, it is due to a prolonged period of inactivity and I still do not have a clear clue on how to fix this error. I do find that random_label=None in the forward pass is causing such error, but still I do not get the resolution.

What's your environment?

EomSooHwan commented 1 year ago

I figured out. train.idx was in the wrong directory for some reason.