Issue with hydra - Error merging data/dataset=cstm Key 'defaults' not in 'FairseqConfig'

[aprzez]

Hello,

I left the below comment under issue #3 but since it's closed I am not sure the comment will be seen.

I am trying to run the training on a custom dataset and also experiencing this issue. The file cstm.yaml is placed in here auto_avsr/conf/data/dataset and looks like this: `defaults:

self root: "/content/drive/MyDrive/sepedi/data/preprocess_datasets" label_dir: "labels" train_file: "train_labels.csv" val_file: "val_labels.csv" test_file: "test_labels.csv"`

As suggested above, I tried renaming the conf/config.yaml file. However, when I run:

!python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1

I get:

Error merging data/dataset=cstm Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

When running:

%env HYDRA_FULL_ERROR=1 !python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1

I get: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config ret = OmegaConf.merge(cfg, loaded_cfg) File "/usr/local/lib/python3.10/dist-packages/omegaconf/omegaconf.py", line 321, in merge target.merge_with(others[1:]) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 331, in merge_with self._format_and_raise(key=None, value=None, cause=e) File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise _raise(ex, cause) File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise raise ex # set end OC_CAUSE=1 for full backtrace File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 329, in merge_with self._merge_with(others) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 347, in _merge_with BaseContainer._map_merge(self, other) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 314, in _map_merge dest[key] = src._get_node(key) File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 258, in setitem self._format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise _raise(ex, cause) File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise raise ex # set end OC_CAUSE=1 for full backtrace omegaconf.errors.ConfigKeyError: Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/content/auto_avsr/main.py", line 74, in main() File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 32, in decorated_main _run_hydra( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 346, in _run_hydra run_and_report( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 201, in run_and_report raise ex File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 198, in run_and_report return func() File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 347, in lambda: hydra.run( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 100, in run cfg = self.compose_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 507, in compose_config cfg = self.config_loader.load_configuration( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration return self._load_configuration( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 256, in _load_configuration cfg = self._merge_defaults_into_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 805, in _merge_defaults_into_config hydra_cfg = merge_defaults_list_into_config(hydra_cfg, user_list) File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 777, in merge_defaults_list_into_config merged_cfg = self._merge_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 724, in _merge_config raise ConfigCompositionException( hydra.errors.ConfigCompositionException: Error merging data/dataset=cstm

I am running this on colab due to issues installing fairseq editable locally.

Thank you in advance!

[aprzez]

To add on, I am seeing the same issue running this in a conda environment and have not been able to solve it yet.

[mpc001]

Hi @aprzez, If you move the + in front of cstm, will it work? For instance,

!python main.py exp_dir=exp \
exp_name=trainaudio \
data.modality=audio \
ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \
data/dataset=cstm \ trainer.num_nodes=1

If not, it would be great if you could provide the package versions of hydra and fairseq. I will try to reproduce the error you have.

[aprzez]

Hi @mpc001, I added the plus because when I run

python main.py exp_dir=exp \ exp_name=LRS3_A_WER1 \ data.modality=audio \ ckpt_path='/crex/proj/uppmax2023-2-2/agnie/auto_avsr/models/LRS3_A_WER1.0/model.pth' \ data/dataset=cstm \ trainer.num_nodes=1 > nchlt_LRS3_A_WER1.txt 2>&1

I get: hydra.errors.ConfigCompositionException: Could not override 'data/dataset'. No match in the defaults list. To append to your default list use +data/dataset=cstm

Just now trying to run with "+" again: hydra.errors.ConfigCompositionException: In 'data/dataset/cstm': ConfigKeyError raised while composing config: Key 'data' not in 'FairseqConfig' full_key: data object_type=FairseqConfig

In both cases I renamed the "config" file to to "cf".

Package versions in my conda environment: fairseq 0.12.3 hydra-core 1.3.0

Thank you!

[mpc001]

It appears that the main hydra configuration file is conflicting with the configuration file for fairseq (similar to #3). Can you please verify the version of omegaconf as well? I have included a list of my packages below. Also make sure that fairseq folder is placed at the root directory of this repository.

omegaconf                 2.3.0                    pypi_0    pypi
hydra-core                1.3.0                    pypi_0    pypi
fairseq                   0.12.2                   pypi_0    pypi

[aprzez]

Hi, confirming the packages I have:

omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi

I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects

And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12

Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.

Thank you for looking into this.

[zyjcsf]

Hi, confirming the packages I have:

omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi

I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects

And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12

Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr.

Thank you for looking into this.

I also happen this problem. Do you have solved it ?

[SUNDULE]

Hi, confirming the packages I have: omegaconf 2.3.0 pyhd8ed1ab_0 conda-forge fairseq 0.12.3 py38h668f88f_3 conda-forge hydra-core 1.3.0 pypi_0 pypi I when installing fairseq editable I was getting this error: ERROR: Failed building wheel for fairseq Failed to build fairseq ERROR: Could not build wheels for fairseq, which is required to install pyproject.toml-based projects And the only way I managed to install it was by running: conda install -c conda-forge fairseq=0.12 Not sure if that makes a difference. The fairseq folder is placed inside auto_asvr. Thank you for looking into this.

I also happen this problem. Do you have solved it ?

Install the pip==23.3.1 version :)