Decoder Training with Audio only

[ouwen18]

When I train the decoder with audio only, I encounter an AssertionError: Could not infer the task type from the following configuration {'_name': 'av_hubert_pretraining', 'is_s2s': True, 'data': '/usr/zzs/data/mvlrs_v1/29h_data', 'label_dir': '/usr/zzs/data/mvlrs_v1/29h_data', 'tokenizer_bpe_model': '/usr/zzs/data/mvlrs_v1/spm1000/spm_unigram1000.model', 'normalize': True, 'labels': ['wrd'], 'single_target': True, 'fine_tuning': True, 'stack_order_audio': 4, 'tokenizer_bpe_name': 'sentencepiece', 'max_sample_size': 500, 'modalities': ['audio'], 'image_aug': True, 'pad_audio': True, 'random_crop': False, 'noise_prob': 0.25, 'noise_snr': 0, 'noise_wav': '???'}.

Available argparse tasks: dict_keys(['translation', 'translation_lev', 'speech_to_text', 'hubert_pretraining', 'speech_unit_modeling', 'multilingual_translation', 'multilingual_masked_lm', 'text_to_speech', 'frm_text_to_speech', 'denoising', 'multilingual_denoising', 'legacy_masked_lm', 'semisupervised_translation', 'translation_multi_simple_epoch', 'simul_speech_to_text', 'simul_text_to_text', 'sentence_prediction', 'sentence_prediction_adapters', 'cross_lingual_lm', 'translation_from_pretrained_bart', 'sentence_ranking', 'speech_to_speech', 'online_backtranslation', 'audio_pretraining', 'audio_finetuning', 'multilingual_language_modeling', 'language_modeling', 'masked_lm', 'translation_from_pretrained_xlm', 'dummy_lm', 'dummy_masked_lm', 'dummy_mt']).

Available hydra tasks: dict_keys(['translation', 'translation_lev', 'hubert_pretraining', 'speech_unit_modeling', 'simul_text_to_text', 'sentence_prediction', 'sentence_prediction_adapters', 'audio_pretraining', 'audio_finetuning', 'multilingual_language_modeling', 'language_modeling', 'masked_lm', 'translation_from_pretrained_xlm', 'dummy_lm', 'dummy_masked_lm']).

[ouwen18]

there is not a type named av_hubert_pretraining

[Exgc]

This task [av_huebert_pretraining] has been registered in hubert_pretraining.py. Refer to [https://github.com/Exgc/OpenSR/blob/main/opensr/hubert_pretraining.py#L165C2-L165C15]

This error may occur because you did not go into the OpenSR/opensr directory before running the code. Please check again the directory where you ran the code.

[ouwen18]

Thank you for your guidance, but there are still some issues. When I use the command:

fairseq-hydra-train --config-dir /home/aa/OpenSR/opensr/conf --config-name opensr_large_vox_audio.yaml \ task.data=/usr/zzs/data/mvlrs_v1/29h_data task.label_dir=/usr/zzs/data/mvlrs_v1/29h_data \ task.tokenizer_bpe_model=/usr/zzs/data/mvlrs_v1/spm1000/spm_unigram1000.model model.w2v_path=/usr/zzs/model/large_vox_iter5.pt hydra.run.dir=/usr/zzs/model/env common.user_dir=pwd

for training the decoder via the command line, I encounter an error:

Could not override 'task.data'. To append to your config use +task.data=/usr/zzs/data/mvlrs_v1/29h_data. Key 'data' is not in struct. full key: task.data reference type=Any object type=dict Set the environment variable HYDRA FULL ERROR=1 for a complete stack trace.

However, when I use +task.data, I face the problem:

AssertionError: full_key: task.data reference_type=Optional[FairseqConfig] object_type=FairseqConfig

[Exgc]

Please check if these two packages meet the version requirements:

• hydra-core<1.1
• omegaconf<2.1

We have a preliminary suspicion that this might result from this.

[ouwen18]

the vision of hydra-core is 1.0.7 and the vision omegaconf is 2.0.6

[Exgc]

Can you provide more details about this? It is also recommended that you can give some running screenshots.

[ouwen18]

this is the traceback of the run, (openSR) aa@aa-Super-Server:~/OpenSR/opensr$ fairseq-hydra-train --config-dir /home/aa/OpenSR/opensr/conf --config-name opensr/opensr_large_vox_audio.yaml \

task.data=usr/zzs/data/mvlrs_v1/29h_data task.label_dir=usr/zzs/data/mvlrs_v1/29h_data \ task.tokenizer_bpe_model=/usr/zzs/data/mvlrs_v1/spm1000/spm_unigram1000.model \ hydra.run.dir=/usr/zzs/model/env common.user_dir=pwd

Traceback (most recent call last): File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report return func() File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in lambda: hydra.run( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 100, in run cfg = self.compose_config( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 507, in compose_config cfg = self.config_loader.load_configuration( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration return self._load_configuration( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 224, in _load_configuration job_cfg, job_cfg_load_trace = self._load_primary_config( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 819, in _load_primary_config ret, load_trace = self._load_config_impl( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 589, in _load_config_impl ret = self.repository.load_config( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/config_repository.py", line 45, in load_config ret = source.load_config( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/_internal/core_plugins/file_config_source.py", line 35, in load_config self._update_package_in_header( File "/home/aa/anaconda3/envs/openSR/lib/python3.8/site-packages/hydra/plugins/config_source.py", line 176, in _update_package_in_header raise HydraException( hydra.errors.HydraException: Primary config 'opensr/opensr_large_vox_audio' must be in the global package; effective package : 'opensr'

[Exgc]

So, is this a new issue? I don't see what you are describing:

AssertionError: full_key: task.data reference_type=Optional[FairseqConfig] object_type=FairseqConfig

[ouwen18]

I'm sorry, I just identified the issue. It was a problem with one of the separators. I apologize for my oversight and any inconvenience it may have caused you

[Exgc]

Good Luck！Also you can find some solutions to your issues in https://github.com/facebookresearch/av_hubert/issues.

If you find OpenSR useful in your research, please cite our work. Thanks！