search

omry / omegaconf

Flexible Python configuration system. The last one you will ever need.
BSD 3-Clause "New" or "Revised" License
1.94k stars 106 forks source link

fails on `Dict[Any, str]` if passed in a list of strings #974

Open jieru-hu opened 2 years ago

jieru-hu commented 2 years ago 
from dataclasses import dataclass
from typing import Optional, Any, Dict
from omegaconf import OmegaConf

@dataclass
class Config:
    test_dict: Optional[Dict[Any, str]] = None

c = Config(test_dict={"hello": ["word"]})

OmegaConf.structured(c)

oc 2.1.1

% python /Users/jieru/workspace/hydra/repro/repro.py
<class 'omegaconf.listconfig.ListConfig'>

oc 2.2.2

% python /Users/jieru/workspace/hydra/repro/repro.py
Traceback (most recent call last):
  File "/Users/jieru/workspace/hydra/repro/repro.py", line 11, in <module>
    c=OmegaConf.structured(c)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 123, in structured
    return OmegaConf.create(obj, parent, flags)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 176, in create
    return OmegaConf._create_impl(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 882, in _create_impl
    format_and_raise(node=None, key=None, value=None, msg=str(e), cause=e)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 793, in format_and_raise
    _raise(ex, cause)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 843, in _create_impl
    return DictConfig(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 111, in __init__
    format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex))
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 793, in format_and_raise
    _raise(ex, cause)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 94, in __init__
    self._set_value(content, flags=flags)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 649, in _set_value
    raise e
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 646, in _set_value
    self._set_value_impl(value, flags)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 676, in _set_value_impl
    data = get_structured_config_data(value, allow_objects=ao)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 481, in get_structured_config_data
    return get_dataclass_data(obj, allow_objects=allow_objects)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 409, in get_dataclass_data
    format_and_raise(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 793, in format_and_raise
    _raise(ex, cause)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 401, in get_dataclass_data
    d[name] = _maybe_wrap(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 1070, in _maybe_wrap
    return _node_wrap(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 986, in _node_wrap
    node = DictConfig(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 111, in __init__
    format_and_raise(node=None, key=key, value=None, cause=ex, msg=str(ex))
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 793, in format_and_raise
    _raise(ex, cause)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 109, in __init__
    self._set_value(content, flags=flags)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 649, in _set_value
    raise e
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 646, in _set_value
    self._set_value_impl(value, flags)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 692, in _set_value_impl
    self.__setitem__(k, v)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 316, in __setitem__
    self._format_and_raise(key=key, value=value, cause=e)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 793, in format_and_raise
    _raise(ex, cause)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 310, in __setitem__
    self.__set_impl(key=key, value=value)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/dictconfig.py", line 320, in __set_impl
    self._set_item_impl(key, value)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 618, in _set_item_impl
    self._wrap_value_and_set(key, value, target_type_hint)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 634, in _wrap_value_and_set
    self._format_and_raise(key=key, value=val, cause=e)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 873, in format_and_raise
    _raise(ex, cause)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/_utils.py", line 771, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/basecontainer.py", line 626, in _wrap_value_and_set
    wrapped = _maybe_wrap(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 1070, in _maybe_wrap
    return _node_wrap(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/omegaconf.py", line 1043, in _node_wrap
    node = StringNode(value=value, key=key, parent=parent, is_optional=is_optional)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/nodes.py", line 174, in __init__
    super().__init__(
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/nodes.py", line 29, in __init__
    self._set_value(value)  # lgtm [py/init-calls-subclass]
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/nodes.py", line 46, in _set_value
    self._val = self.validate_and_convert(value)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/nodes.py", line 76, in validate_and_convert
    return self._validate_and_convert_impl(value)
  File "/Users/jieru/miniconda3/envs/hydra39/lib/python3.9/site-packages/omegaconf/nodes.py", line 194, in _validate_and_convert_impl
    raise ValidationError("Cannot convert '$VALUE_TYPE' to string: '$VALUE'")
omegaconf.errors.ValidationError: Cannot convert 'list' to string: '['word']'
    full_key: test_dict.hello
    reference_type=Optional[Dict[Any, str]]
    object_type=dict

This seems to be a bug(?)/workaround(?) in OC 2.1 and has since been correctly supported in OC 2.2. thought i'd file the issue here anyway since this breaks existing use cases, we should probably at least document it somewhere if not fix it.

Jasha10 commented 2 years ago

In OC 2.1, nested containers do not have full runtime type safety. This means dicts-of-lists were not type-checked deeply. OC 2.2's nested container typing was implemented by PR #890.

To migrate from OC 2.1 to 2.2, the following change to the type hint is probably the easiest:

@dataclass
class Config:
    # test_dict: Optional[Dict[Any, str]] = None
    test_dict: Optional[Dict[Any, Any]] = None  # handles both str and list-of-str

Using Optional[Dict[Any, list[str]]] might also be possible (if the user always passes a dict-of-lists-of-strings).