search

CLMBRs / ultk

A library for research in unnatural language semantics
GNU General Public License v3.0
9 stars 4 forks source link

`FrozenDict` breaks `yaml.load` #46

Closed haberchr closed 1 month ago

haberchr commented 1 month ago

The addition of FrozenDicts to Meaning breaks the yaml.load() function (TypeError: FrozenDict is immutable ):

File ~/Documents/UWLing/altk/src/ultk/util/io.py:46, in read_grammatical_expressions(filename, grammar, re_parse, universe, return_by_meaning)
     [43](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:43)     raise ValueError("Must provide grammar and universe if re-parsing expressions.")
     [45](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:45) with open(filename, "r") as f:
---> [46](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:46)     expression_list = load(f, Loader=Loader)
     [48](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:48) if re_parse:
     [49](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:49)     final_exprs = [
     [50](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:50)         (
     [51](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:51)             grammar.parse(expr_dict["term_expression"])
   (...)
     [55](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:55)         for expr_dict in expression_list
     [56](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/io.py:56)     ]

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/__init__.py:81, in load(stream, Loader)
     [79](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/__init__.py:79) loader = Loader(stream)
     [80](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/__init__.py:80) try:
---> [81](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/__init__.py:81)     return loader.get_single_data()
     [82](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/__init__.py:82) finally:
     [83](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/__init__.py:83)     loader.dispose()

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:51, in BaseConstructor.get_single_data(self)
     [49](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:49) node = self.get_single_node()
     [50](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:50) if node is not None:
---> [51](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:51)     return self.construct_document(node)
     [52](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:52) return None

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:60, in BaseConstructor.construct_document(self, node)
     [58](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:58)     self.state_generators = []
     [59](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:59)     for generator in state_generators:
---> [60](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:60)         for dummy in generator:
     [61](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:61)             pass
     [62](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:62) self.constructed_objects = {}

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:620, in FullConstructor.construct_python_object(self, suffix, node)
    [618](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:618) yield instance
    [619](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:619) deep = hasattr(instance, '__setstate__')
--> [620](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:620) state = self.construct_mapping(node, deep=deep)
    [621](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:621) self.set_python_instance_state(instance, state)

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:218, in SafeConstructor.construct_mapping(self, node, deep)
    [216](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:216) if isinstance(node, MappingNode):
    [217](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:217)     self.flatten_mapping(node)
--> [218](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:218) return super().construct_mapping(node, deep=deep)

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:143, in BaseConstructor.construct_mapping(self, node, deep)
    [140](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:140)     if not isinstance(key, collections.abc.Hashable):
    [141](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:141)         raise ConstructorError("while constructing a mapping", node.start_mark,
    [142](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:142)                 "found unhashable key", key_node.start_mark)
--> [143](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:143)     value = self.construct_object(value_node, deep=deep)
    [144](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:144)     mapping[key] = value
    [145](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:145) return mapping

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:102, in BaseConstructor.construct_object(self, node, deep)
    [100](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:100)     data = constructor(self, node)
    [101](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:101) else:
--> [102](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:102)     data = constructor(self, tag_suffix, node)
    [103](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:103) if isinstance(data, types.GeneratorType):
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:104)     generator = data

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:659, in FullConstructor.construct_python_object_new(self, suffix, node)
    [658](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:658) def construct_python_object_new(self, suffix, node):
--> [659](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:659)     return self.construct_python_object_apply(suffix, node, newobj=True)

File ~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:655, in FullConstructor.construct_python_object_apply(self, suffix, node, newobj)
    [653](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:653) if dictitems:
    [654](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:654)     for key in dictitems:
--> [655](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:655)         instance[key] = dictitems[key]
    [656](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/anaconda3/envs/altk/lib/python3.12/site-packages/yaml/constructor.py:656) return instance

File ~/Documents/UWLing/altk/src/ultk/util/frozendict.py:16, in FrozenDict.__setitem__(self, key, value)
     [15](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/frozendict.py:15) def __setitem__(self, key, value):
---> [16](https://file+.vscode-resource.vscode-cdn.net/Users/user/Documents/altk/src/examples/~/Documents/UWLing/altk/src/ultk/util/frozendict.py:16)     raise TypeError("FrozenDict is immutable")

TypeError: FrozenDict is immutable
haberchr commented 1 month ago

The issue is caused by the loader function interpreting the dictitems as mutable I believe: https://github.com/yaml/pyyaml/blob/main/lib/yaml/constructor.py#L623

Could try to patch this by adding a custom YAML constructor for FrozenDict: https://matthewpburruss.com/post/yaml/

While we're at it, @shanest, do you remember why the class tags in the YAML files had to be expanded to contain so much text? I remember they didn't used to: !!python/object/new:ultk.util.frozendict.FrozenDict instead of !FrozenDict

Relevant StackOverflow post on the tag types: https://stackoverflow.com/questions/9664113/what-does-a-single-exclamation-mark-do-in-yaml

shanest commented 1 month ago

I ran into this issue awhile ago and did implement custom load/save for the class to solve the issue; see https://github.com/CLMBRs/ultk/blob/main/src/ultk/util/frozendict.py . Are you working in a branch that might be behind main on this front? If so, try merging main back into yours; if not, can you say more about what exactly is causing it?

shanest commented 1 month ago

(It's possible that you might also need to re-save the expressions you're trying to load, not positive about that)

haberchr commented 1 month ago

Thanks for the suggestion to save the expressions! I am now able to load expressions again and see the !frozendict tag correctly reflected in the saved expressions yaml.