Open leogama opened 2 years ago
The type(p11) is type(p21) = True
is most certainly a bug, unless namedtuple does something strange that I am unaware of at this time. The current semantics are that, upon unpickling, all main classes and functions are reconstructed by default. This awkwardness related to object equality is the reason that the byref
setting exists.
~Perhaps some special care needs to be taken to handle byref
during session loading. Didn't look into the details.~
Yeah, I have to agree. The namedtuple
behavior is not what should be expected. In terms of settings, byref
(and most other settings) is intended to be active during the dump
, with only ignore
being active during the load
. The ignore
settings provides somewhat related behavior:
If *ignore=False* then objects whose class is defined in the module
*__main__* are updated to reference the existing class in *__main__*,
otherwise they are left to refer to the reconstructed type, which may
be different.
So, there's potential in using the settings to control the desired load
behavior, if alternate load
behavior is desirable.
The
type(p11) is type(p21) = True
is most certainly a bug, unless namedtuple does something strange that I am unaware of at this time.
Perhaps some special care needs to be taken to handle
byref
during session loading. Didn't look into the details.
byref
in session saving doesn't apply to objects only present in the module being saved (and not present in other modules of sys.modules
). By the way, I just realized that maybe the module __main__
should be excluded from the modmap
when saving other modules...
I guess that named tuple is fine then. What should have happened is that the check to see if the named tuple was accessible should have been done in save_type
and not here. Now namedtuple pickling is a little inconsistent, but changing it would break compatibility. Maybe we should put the check in both places, so older pickles still load but new ones are consistent?
Nope. It still seems that byref
chooses which types in the main module should be pickled and which ones should be referenced as a global. @mmckerns, should this be extended to functions too?
@leogama: A while ago, I posed the same question, and I think the answer is that it is probably a useful thing to do. See: #90, #102, and #128. We should think about changing namedtuple
pickling to be consistent, as long as the old pickled nametuples can get loaded. dill
has only been able to handle a nametuple
for a short time, and it's a behavioral change, but not a breaking change.
This is not strictly a bug, but may not be the expected behavior. There are 3 possibilities when unpickling a type object (here named "
atype
"):atype.__name__
in the moduleatype.__module__
*;atype
.atype
.(*) or the module can't be found, or the type object has no
__module__
attribute, likeenum
s andnamedtuple
s can be.The first and second cases should be straightforward to deal with by just creating or overwriting the class, respectively. In the third case, I think it's open for debate which solutions make more sense in different situations. The issue is that dill currently doesn't have a single orientation in this regard, with mixed outcomes.
The example below is based on the last commit from #450 and includes an
Enum
, anamedtuple
—and I don't know ifdataclass
is another special case—, a custom class that doesn't check the objects' classes when testing for equality and a custom class that does it._classdemo.py
The code below shows what happens when comparing "identical" objects created before and after the state of the module containing its classes is saved and restored.
Output:
Therefore, as of today dill always prefer the existing
namedtuple
type definition, even if the one being unpickled is different. On the other hand, it always prefer the unpickled type object in all other cases, overwriting the existing one even if they are identical (in the sense that they would generate identical pickle streams), which may or may not affect operations between instances of different versions of the same class.I would prefer not to think about this kind of stuff, but I'm unable to concentrate in anything else before I write it down and share with you...