Can't save integer connection table properties that come from globals

[philipstarkey]

Original report (archived issue) by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

In Python 3.7 with latest numpy (observed on numpy 1.16.3), one cannot save a connection table attribute or unit calibration parameter that is an integer, and that has been through a HDF5 file as a global. JSON serialisation chokes on the integer, saying it can't serialise it. Here is a minimal breaking example:

import json
import h5py
with h5py.File('test.h5', 'w') as f:
    f.attrs['x'] = 5
    json.dumps(dict(f.attrs))

Traceback (most recent call last):
  File "260.py", line 5, in <module>
    print(json.dumps(dict(f.attrs)))
  File "/usr/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type int64 is not JSON serializable

Of course JSON can serialise normal Python integers, but having been through the HDF5 file, the integer became a np.int32. So an even more minimal breaking example might be:

import json
import numpy as np
json.dumps([np.int32(5)])

And it doesn't matter if it is a np.int32 or np.int64, both break .

This works fine in Python 2 with the same numpy version and works if you convert the integer to a float instead. Looks like a regression in either Python or numpy, I'm not sure which. But I'll see if I can figure out which to report a bug to. We could work around it in labscript suite code, but should not bother if it is to be imminently fixed upstream.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

The bug report for Python is here:

https://bugs.python.org/issue24313

The cause is that numpy integers do not inherit from Python integers on Python 3, because Python integers no longer have a fixed size. In Python 2 there were ints and long ints, and the matching numpy dtypes inherited from them. But no such thing is possible in Python 3. Some cooperation between numpy and Python could make it work, but the bug report is old and there is a decent workaround, so I suspect no fix is coming any time soon.

The workaround is this:

def default(o):
    if isinstance(o, np.integer): return int(o)
    raise TypeError

import json
import numpy as np
json.dumps([np.int64(0)], default=default)

I'll make the change to labscript_utils.properties.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Fixed by PR #82

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• changed state from "new" to "resolved"

Merged in cbillington/labscript_utils/bugfix (pull request #82)

Fix issue #24: can't save integer connection table properties that are globals

→ \<\<cset ec867a32c1564e493b36cc97f358d8c94637293a>>

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Fix issue #24: can't save integer connection table properties that are globals

Applied workaround from Python bug report: https://bugs.python.org/issue24313

→ \<\<cset ac59cf45ee457d2cda8e787367134e59ba4276dc>>

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Merged in cbillington/labscript_utils/bugfix (pull request #82)

Fix issue #24: can't save integer connection table properties that are globals

→ \<\<cset ec867a32c1564e493b36cc97f358d8c94637293a>>