Something went wrong with hydra and omegaconf

jinchiniao commented 1 year ago

When I run

python main.py exp_dir=exp \
               exp_name=train_24_scratch \
               data.modality=vsr \
               optimizer.lr=3e-4 \

for training, something error with hydra. Error occus:

Error merging 'config' with schema
Key 'exp_name' not in 'FairseqConfig'
        full_key: exp_name
        reference_type=Optional[Dict[Union[str, Enum], Any]]
        object_type=FairseqConfig

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

So I run

HYDRA_FULL_ERROR=1 python main.py exp_dir=exp \
               exp_name=train_24_scratch \
               data.modality=vsr \
               optimizer.lr=3e-4 \

and it shows:

Traceback (most recent call last):
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 618, in _load_config_impl
    merged = OmegaConf.merge(schema.config, ret.config)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 321, in merge
    target.merge_with(*others[1:])
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 331, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
    format_and_raise(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 329, in merge_with
    self._merge_with(*others)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 347, in _merge_with
    BaseContainer._map_merge(self, other)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 314, in _map_merge
    dest[key] = src._get_node(key)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 258, in __setitem__
    self._format_and_raise(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
    format_and_raise(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigKeyError: Key 'exp_name' not in 'FairseqConfig'
        full_key: exp_name
        reference_type=Optional[Dict[Union[str, Enum], Any]]
        object_type=FairseqConfig
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "main.py", line 74, in <module>
    main()
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 346, in _run_hydra
    run_and_report(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 201, in run_and_report
    raise ex
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 198, in run_and_report
    return func()
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/utils.py", line 347, in <lambda>
    lambda: hydra.run(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 100, in run
    cfg = self.compose_config(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 507, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration
    return self._load_configuration(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 224, in _load_configuration
    job_cfg, job_cfg_load_trace = self._load_primary_config(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 819, in _load_primary_config
    ret, load_trace = self._load_config_impl(
  File "/home/luosongtao/miniconda3/envs/autoavsr/lib/python3.8/site-packages/hydra/_internal/config_loader_impl.py", line 628, in _load_config_impl
    raise ConfigCompositionException(
hydra.errors.ConfigCompositionException: Error merging 'config' with schema

Any help to resolve this would be greatly appreciated!

jinchiniao commented 1 year ago

The latest version of fairseq (0.12.2) conflicts with the configuration files of Hydra because fairseq has its own configuration file. To resolve this, you can rename the "conf/config.yaml" file to a different name.

mpc001 commented 1 year ago

Hi @jinchiniao, thank you for letting me know the conflicts. I tested the development version. I was wondering if you installed it using pip install?

jinchiniao commented 1 year ago

I followed the instructions in the README to set up the environment. This means that I compiled fairseq from source code and also installed Hydra using pip.

aprzez commented 1 year ago

Hello! I am trying to run the training on a custom dataset and also experiencing this issue. The file cstm.yaml is placed in here auto_avsr/conf/data/dataset and looks like this: `defaults:

self root: "/content/drive/MyDrive/sepedi/data/preprocess_datasets" label_dir: "labels" train_file: "train_labels.csv" val_file: "val_labels.csv" test_file: "test_labels.csv"`

As suggested above, I tried renaming the conf/config.yaml file. However, when I run:

!python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1

I get:

Error merging data/dataset=cstm Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

When running:

%env HYDRA_FULL_ERROR=1 !python main.py exp_dir=exp \ exp_name=trainaudio \ data.modality=audio \ ckpt_path='content/drive/MyDrive/LRS3_A_WER1.0/model.pth' \ +data/dataset=cstm \ trainer.num_nodes=1

I get: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 720, in _merge_config ret = OmegaConf.merge(cfg, loaded_cfg) File "/usr/local/lib/python3.10/dist-packages/omegaconf/omegaconf.py", line 321, in merge target.merge_with(others[1:]) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 331, in merge_with self._format_and_raise(key=None, value=None, cause=e) File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise _raise(ex, cause) File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise raise ex # set end OC_CAUSE=1 for full backtrace File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 329, in merge_with self._merge_with(others) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 347, in _merge_with BaseContainer._map_merge(self, other) File "/usr/local/lib/python3.10/dist-packages/omegaconf/basecontainer.py", line 314, in _map_merge dest[key] = src._get_node(key) File "/usr/local/lib/python3.10/dist-packages/omegaconf/dictconfig.py", line 258, in setitem self._format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/base.py", line 95, in _format_and_raise format_and_raise( File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 629, in format_and_raise _raise(ex, cause) File "/usr/local/lib/python3.10/dist-packages/omegaconf/_utils.py", line 610, in _raise raise ex # set end OC_CAUSE=1 for full backtrace omegaconf.errors.ConfigKeyError: Key 'defaults' not in 'FairseqConfig' full_key: defaults reference_type=Optional[FairseqConfig] object_type=FairseqConfig

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/content/auto_avsr/main.py", line 74, in main() File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 32, in decorated_main _run_hydra( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 346, in _run_hydra run_and_report( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 201, in run_and_report raise ex File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 198, in run_and_report return func() File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 347, in lambda: hydra.run( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 100, in run cfg = self.compose_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 507, in compose_config cfg = self.config_loader.load_configuration( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 151, in load_configuration return self._load_configuration( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 256, in _load_configuration cfg = self._merge_defaults_into_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 805, in _merge_defaults_into_config hydra_cfg = merge_defaults_list_into_config(hydra_cfg, user_list) File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 777, in merge_defaults_list_into_config merged_cfg = self._merge_config( File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/config_loader_impl.py", line 724, in _merge_config raise ConfigCompositionException( hydra.errors.ConfigCompositionException: Error merging data/dataset=cstm

I am running this on colab due to issues installing fairseq editable locally.

Thank you in advance!

SarahSalimpour commented 1 year ago

Hi Thank you for sharing the code.

For Installing additional packages in step 3.4 I got this error: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. ipython 7.34.0 requires jedi>=0.16, which is not installed. arviz 0.15.1 requires setuptools>=60.0.0, but you have setuptools 59.5.0 which is incompatible. cvxpy 1.3.2 requires setuptools>65.5.1, but you have setuptools 59.5.0 which is incompatible. fairseq 0.12.2 requires hydra-core<1.1,>=1.0.7, but you have hydra-core 1.3.0 which is incompatible. fairseq 0.12.2 requires omegaconf<2.1, but you have omegaconf 2.3.0 which is incompatible. Successfully installed GitPython-3.1.32 antlr4-python3-runtime-4.9.3 av-10.0.0 docker-pycreds-0.4.0 gitdb-4.0.10 hydra-core-1.3.0 lightning-utilities-0.9.0 omegaconf-2.3.0 pathtools-0.1.2 pyDeprecate-0.3.1 pytorch-lightning-1.5.10 sentencepiece-0.1.99 sentry-sdk-1.29.2 setproctitle-1.3.2 setuptools-59.5.0 smmap-5.0.0 torchmetrics-1.0.3 wandb-0.15.8

Can you help me about this?

mpc001 / auto_avsr

Something went wrong with hydra and omegaconf #3