Pickle/dill incompatibility: "CompiledDemSampler"

[MABejan]

I have been using pickle (and also tried dill) to save and load Clifford gates coming from the Stim library. The code is running fine on my laptop but not on a remote computer, where the following error occurs.

File ".../minimal_example.py", line 6, in <module>
    cliff1_list = dill.load(fl)
  File ".../my_python/lib/python3.9/site-packages/dill/_dill.py", line 272, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File ".../my_python/lib/python3.9/site-packages/dill/_dill.py", line 419, in load
    obj = StockUnpickler.load(self)
  File ".../my_python/lib/python3.9/site-packages/dill/_dill.py", line 409, in find_class
    return StockUnpickler.find_class(self, module, name)
ImportError: generic_type: type "CompiledDemSampler" is already registered!

A minimal example for producing the issue is

import dill, stim

the_file_name1 = "Stim_code/stim_1Cliff_tableaus24.pickle"
with open(the_file_name1, "rb") as fl:
    cliff1_list = dill.load(fl)

where the file was generated by

import stim, time, dill

tic=time.time()
I = stim.Tableau.from_named_gate("I")
Z0 = stim.Tableau.from_named_gate("Z")
X0 = stim.Tableau.from_named_gate("X")
Y0 = stim.Tableau.from_named_gate("Y")
H0 = stim.Tableau.from_named_gate("H")
S0 = stim.Tableau.from_named_gate("S")

V0 = H0*S0*H0*S0
W0 = H0*S0

h_list0 = [I, H0]
v_list0 = [I, V0, W0]
p_list0 = [I, X0, Y0, Z0]

all_1Cliff = []

nn=0
for h0 in h_list0:    
    for v0 in v_list0:
        for p0 in p_list0:
            all_1Cliff.append(h0*v0*p0)
            nn+=1

print(nn)

toc = time.time()
print("Generation time taken: {:4f} seconds".format(toc-tic))

tic = time.time()

tic = time.time()
final_filename = "Stim_code/stim_1Cliff_tableaus{}.pickle".format(nn)
f = open(final_filename, 'wb')
dill.dump(all_1Cliff,f,  protocol=dill.HIGHEST_PROTOCOL)            
f.close()

Please let me know if there are any details which could help.

[Strilanc]

Are you generating the files on one computer, and loading them on the other?

The issue is probably to do with how stim is split into 3 modules with the same type names, and it dynamically picks based on presence of AVX/SSE/etc. If you try to load two it collides, and probably it's trying to load the one from the other machine. Even if the load worked, you'd probably get segfaults soon after.

Hmm....

Oh, also, are you aware of stim.Tableau.iter_all(num_qubits=1, unsigned=False)? Looks lik ethat's what you're doing.

As a workaround, you can store repr(tableau) and load it with eval(the_repr, {'stim': stim}). Also i think just normal pickle should work?

[MABejan]

Sorry for the immense delay in answering back.

Indeed the problem was that I was generating the files on one computer and using them on the other. Generating the files on the same computer that I run the code on solves the problem, as you were suggesting. Thank you!

Yes, I was aware that there already existed all the n-qubit Clifford gates in stim, but I needed them in a particular order, which would have probably taken a bit longer to work out if converted from the stim default. Thank you for the pointer though.