Cannot pickle (or jsonify) magma structures

[steveri]

We wanted serialize and save an object that contained magma structures, e.g. bits. Because magma structures are complicated, json serialization did not work at all, as expected. So we tried pickle, and got the following error

_pickle.PicklingError: Can't pickle In(Bit): attribute lookup Bit[In] on mypackage.bit failed

Here is the simple example that generates the error

import magma as m
import pickle
pickle.dumps(m.In(m.Bit))

And here is the complete error trace

Traceback (most recent call last):
  File "simple-example.py", line 7, in <module>
    pickle.dumps(bit)
_pickle.PicklingError: Can't pickle In(Bit): attribute lookup Bit[In] on magma.bit failed

Can anyone help with this?

[steveri]

More information: I drilled down and maybe narrowed the problem to the ['_abc_impl'] component of the bit: Example code:

import magma as m
import pickle

bit = m.In(m.Bit)
d = bit.__dict__
print(d)
pickle.dumps(d['_abc_impl'])

Example result of running code:

{
  'orig_name': 'Bit',
  '_info_': (Bit, <Direction.In: 0>),
  '__module__': 'magma.bit',
  '__doc__': None,
  '__abstractmethods__': frozenset(),
  '_abc_impl': <_abc_data object at 0x7f24490023f0>
}
Traceback (most recent call last):
  File "simple-example.py", line 7, in <module>
    pickle.dumps(d['_abc_impl'])
TypeError: cannot pickle '_abc_data' object

The error cannot pickle '_abc_data' object is a known long-standing problem with pickle, but they claim to have solved it: https://github.com/uqfoundation/dill/issues/332

I tried their example (below) and did not get the error, so I guess they think it's fixed. But ours is still broken.

import dill
from abc import ABCMeta

class AbstractClass(metaclass=ABCMeta):
  pass

class ConcreteClass(AbstractClass):
  pass

dill.dumps(ConcreteClass)

The next step (if I were doing this) would be to find what's different between our abstract class and their example, and either fix our code or update their bug report. But I'm moving on to something else, so I'm hoping someone else might take up the cause...