Globals can no longer be tuples of strings

[philipstarkey]

Original report (archived issue) by Shaun Johnstone (Bitbucket: shjohnst, GitHub: shjohnst).

---

In the past our group has used a global containing a tuple of strings to pass a list of strings through to lyse for analysis (we would list the aliases of results that we wanted to plot for the given experiment - this could be implemented better via analysis globals, but that's a whole different issue). We use a tuple since it isn't automatically iterated over by runmanager.

It seems as though somewhere along the way (most likely during Python 3 compatibility updates) the ability to save this datatype to the globals hdf5 file has been lost. When trying to compile an experiment with a global foo = ('aaa','bbb') I now get the following error:

#!python

Global foo cannot be saved as an hdf5 attribute. Globals can only have relatively simple datatypes, with no nested structures. Original error was:
TypeError: No conversion path for dtype: dtype('<U3')
Compilation aborted.

This also happens if the global is a list and the expansion is removed, to pass through the global as a list.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

Since @chrisjbillington has a thesis deadline approaching I'm happy to look into this instead if you/he likes

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Relevant discussion here:

https://github.com/h5py/h5py/issues/441

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

This can be fixed in Python 2 by telling eval() to not inherit the future imports from the surrounding scope (pull request #20).

For Python 3, strings will be unicode regardless, so that doesn't fix the problem. pull request submitted to h5py for this case. If they release a version with the fix then we can depend on that version to consider runmanager fully Python 3 compatible. Even if they don't accept my pull request I expect this will be supported in the future as more science applications move to Python 3. Either h5py will support it in some way or numpy will introduce a UTF8 encoded string array datatype, which would make it much easier.

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

So I've been playing around with this a bit today and hadn't yet come up with a workable solution. Patching h5py seems like a good one though (assuming they accept it).

However, I've also noticed that globals set to straight strings under Python 3 are stored as ndarray, which is equivalent and indistinguishable from a tuple containing a single string as far as I can tell. Furthermore, under Python 3, tuples with mixed data types (e.g. a string and an int) do not have the datatype preserved when read out by labscript (but they do on Python 2). Does your h5py patch fix those situations?

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

I can't reproduce the problem with single strings being numpy arrays, they come out in labscript code as single strings for me.

Mixed data seems to be coerced to a single datatype on both versions of Python. This is pretty useless, I've never noticed it before. Can you confirm you have a version of Python that does not do this? If you're sure it's the case, maybe it's a different version of h5py?

...I'm running out of reasons why we should use HDF5 datatypes at all instead of rolling our own serialisation as in properties.py. I want to work with kosher HDF, but this is just too much of a pain in the neck.

properties.set_device_properties() even tries to use native types before falling back to JSON serialisation. It should probably also check that the result is actually equal to what it put in, because otherwise mixed datatypes in a tuple/list might be getting coerced to be the same datatype!

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• changed state from "new" to "resolved"

Merged in cbillington/runmanager/bugfix (pull request #20)

Fix issue #56 for Python 2

Approved-by: Shaun Johnstone shaun.johnstone@monash.edu

→ \<\<cset aa1197dbd75bb3c423e48bbd7aa8576d481d6c24>>

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Fix issue #56 for Python 2. Python 3 fix will have to wait for a pull request submitted to h5py

→ \<\<cset 1b6ea60b5ad6225c8cede96caecfd8693adb1117>>

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Merged in cbillington/runmanager/bugfix (pull request #20)

Fix issue #56 for Python 2

Approved-by: Shaun Johnstone shaun.johnstone@monash.edu

→ \<\<cset aa1197dbd75bb3c423e48bbd7aa8576d481d6c24>>

[philipstarkey]

Original comment by Philip Starkey (Bitbucket: pstarkey, GitHub: philipstarkey).

---

Hmm, I think I was mistaken on both counts.

I got confused because my hack was printing b'2' on Python 3 and 2 on Python 2.7, but the 2 is still a string, just didn't get printed with quote marks. Possible I had accidentally done 'asd',#abunchofstuff for the other issue, which is equivalent to a tuple, not a single string. Single strings do seem to work fine on Python 3.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• changed state from "resolved" to "open"

Reopening because this is only solved on Python 2, and is still an issue on Python 3

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

This was long-since fixed in labscript_utils with a monkeypatch, and has since been fixed in h5py.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

• changed state from "open" to "resolved"