Open aprzez opened 1 year ago
To add on, I am seeing the same issue running this in a conda environment and have not been able to solve it yet.
Hi @aprzez, If you move the + in front of cstm
, will it work? For instance,
!python main.py exp_dir=exp \
exp_name=trainaudio \
data.modality=audio \
ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \
data/dataset=cstm \ trainer.num_nodes=1
If not, it would be great if you could provide the package versions of hydra and fairseq. I will try to reproduce the error you have.
Hi @mpc001, I added the plus because when I run
python main.py exp_dir=exp \ exp_name=LRS3_A_WER1 \ data.modality=audio \ ckpt_path='/crex/proj/uppmax2023-2-2/agnie/auto_avsr/models/LRS3_A_WER1.0/model.pth' \ data/dataset=cstm \ trainer.num_nodes=1 > nchlt_LRS3_A_WER1.txt 2>&1
I get: hydra.errors.ConfigCompositionException: Could not override 'data/dataset'. No match in the defaults list. To append to your default list use +data/dataset=cstm
Just now trying to run with "+" again: hydra.errors.ConfigCompositionException: In 'data/dataset/cstm': ConfigKeyError raised while composing config: Key 'data' not in 'FairseqConfig' full_key: data object_type=FairseqConfig
In both cases I renamed the "config" file to to "cf".
Package versions in my conda environment: fairseq 0.12.3 hydra-core 1.3.0
Thank you!
It appears that the main hydra configuration file is conflicting with the configuration file for fairseq (similar to #3). Can you please verify the version of omegaconf as well? I have included a list of my packages below. Also make sure that fairseq
folder is placed at the root directory of this repository.
omegaconf 2.3.0 pypi_0 pypi
hydra-core 1.3.0 pypi_0 pypi
fairseq 0.12.2 pypi_0 pypi
Hi, confirming the packages I have:
omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi
I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects
And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12
Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.
Thank you for looking into this.
Hi, confirming the packages I have:
omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi
I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects
And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12
Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.
Thank you for looking into this.
I also happen this problem. Do you have solved it ?
Hi, confirming the packages I have: omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12 Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr. Thank you for looking into this.
I also happen this problem. Do you have solved it ?
Install the pip==23.3.1 version :)
Hello,
I left the below comment under issue #3 but since it's closed I am not sure the comment will be seen.
I am trying to run the training on a custom dataset and also experiencing this issue. The file cstm.yaml is placed in here auto_avsr/conf/data/dataset and looks like this: `defaults:
self root: "/content/drive/MyDrive/sepedi/data/preprocess_datasets" label_dir: "labels" train_file: "train_labels.csv" val_file: "val_labels.csv" test_file: "test_labels.csv"`
As suggested above, I tried renaming the conf/config.yaml file. However, when I run:
!python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1
I get:
Error merging data/dataset=cstm Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
When running:
%env HYDRA_FULL_ERROR=1 !python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1
I get: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config ret = OmegaConf.merge(cfg, loaded_cfg) File "/usr/local/lib/python3.10/dist-packages/omegaconf/omegaconf.py", line 321, in merge target.merge_with(others[1:]) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 331, in merge_with self._format_and_raise(key=None, value=None, cause=e) File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise _raise(ex, cause) File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise raise ex # set end OC_CAUSE=1 for full backtrace File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 329, in merge_with self._merge_with(others) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 347, in _merge_with BaseContainer._map_merge(self, other) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 314, in _map_merge dest[key] = src._get_node(key) File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 258, in setitem self._format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise _raise(ex, cause) File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise raise ex # set end OC_CAUSE=1 for full backtrace omegaconf.errors.ConfigKeyError: Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/content/auto_avsr/main.py", line 74, in main() File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 32, in decorated_main _run_hydra( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 346, in _run_hydra run_and_report( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 201, in run_and_report raise ex File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 198, in run_and_report return func() File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 347, in lambda: hydra.run( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 100, in run cfg = self.compose_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 507, in compose_config cfg = self.config_loader.load_configuration( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration return self._load_configuration( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 256, in _load_configuration cfg = self._merge_defaults_into_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 805, in _merge_defaults_into_config hydra_cfg = merge_defaults_list_into_config(hydra_cfg, user_list) File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 777, in merge_defaults_list_into_config merged_cfg = self._merge_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 724, in _merge_config raise ConfigCompositionException( hydra.errors.ConfigCompositionException: Error merging data/dataset=cstm
I am running this on colab due to issues installing fairseq editable locally.
Thank you in advance!