precisely define and implement supported dataset types

[ghost]

• [ ] decide which data types are supported for set_dataset()
• [ ] make supported types work
• [ ] unit test

from artiq import *
import numpy as np
import time

class DefaultStd(EnvExperiment):
    """demo"""
    def build(self):
        pass

    def run(self):

        tests = [
            True,
            [True, False],
            int(1234),
            [1, 2, 3]*3,
            float(1234),
            [1.1, 1.2, 1.3]*3,
            complex(1, 1),
            [complex(1,1), complex(1,2), complex(1,3)]*3,
            "asdf",
            ["asdf", "lkj"]*3,
            np.array(np.random.random(10), dtype=np.float32),
            np.array(np.random.random(10), dtype=np.float64),
            np.array(np.random.random(10), dtype=np.int32),
            np.array(np.random.random(10), dtype=np.int64),
            np.array(np.random.random(10), dtype=np.complex64),
            np.array(np.random.rand(10, 10), dtype=np.float32),
            np.array(np.random.rand(10, 10, 10), dtype=np.float32)
        ]

        for t in tests:
            self.set_dataset("test", t,
                              persist=False, save=False, broadcast=False)
            self.set_dataset("test", t,
                              persist=False, save=False, broadcast=True)
            self.set_dataset("test", t,
                              persist=False, save=True, broadcast=False)
            self.set_dataset("test", t,
                              persist=True, save=False, broadcast=False)
            self.set_dataset("test", t,
                              persist=True, save=True, broadcast=True)

[sbourdeauducq]

Let's start with item 1. What types do you need in the HDF5 output? What types do you need for broadcast?

[ghost]

Data type/class considerations involve where the data resides and where it can move.

core device <--> master <--> HDF5

core device: I gather that @kernel methods on the core device only support native python data classes right now: int, float, bool, str. That's fine.

master: Full python so any data type is possible. Fine internally, but for transport to/from core/hdf5 need to test for restricted set of data types.

HDF5: Requires mapping between master-supported types and HDF5 types.

Desired data types

• Data classes for setattr_argument() are defined in artiq.language.environment. Looks like the low-level types come down to: python bool, str, int and float, and ARTIQ-specific artiq.language.environment.FreeValue. Any of these could reasonably be piped too set_dataset().
• ARTIQ officially supports 32-bit and 64-bit. So need to assume that each data type can be of either length.
• It's helpful to use numpy on the master. I think we can safely skip uncommon types like uint8. The numpy types I think we should support are bool, int and float. And support for 1- and 2-dimensional arrays of bool, int and float; useful for representing images and 2D scans. Higher dimensional arrays and complex numbers seem like edge cases.

mapping: master -> HDF5

• Presently it looks like only python int and float are supported. master.worker_db.py
• FreeValue could be picked()
• Best if numpy is handled automatically. e.g without using np.ndarray.tolist().

mapping: HDF5 -> master

• Needed for for later "experiment browse and playback" features.
• raise TypeError() if not supported
• FreeValue could be unpickled()

mapping: core device - > master

• Peter's compiler already does this.

mapping: master -> core device

• Best to automatically use np.ndarray.tolist() when possible.
• 2d arrays to python lists of lists
• raise TypeError() if not supported

[sbourdeauducq]

Ok, some clarification is needed here. There are three places where type-related problems may appear in datasets:

1. broadcasting datasets on the network (and master-worker communication). The objects need to be serialized there, and we use PYON (a superset of JSON) for that. PYON can be arbitrarily extended, but it cannot support all Python types. (Python pickles are slow, inefficient, hard to make secure and not interoperable)
2. mutating broadcasted datasets (e.g. doing append() on a result after it has been set, for real-time plotting). The sync_struct system forwards list and dict operations; it could be extended to look more like the pc_rpc system and forward arbitrary method calls. sync_struct uses PYON for serialization.
3. writing the persistent datasets to disk. This also uses PYON so no new problem is introduced here.
4. writing the HDF5 archives. The HDF5 format has a limited structure compared to Python; we can map common types (strings, scalars, arrays...) to HDF5 types. For other types, we can either fail or PYON-serialize them and store as HDF5 string.

So the actual questions are:

• what types should PYON support
• should sync_struct support all PYON types
• what Python types should map to HDF5 native types and how

[ghost]

Regarding 1, in Sept we discussed one option for giving users greater flexibility in specifying plots in the GUI. One option was sending matplotlib code from the Master to the GUI. If RPC is to be used for this it would need to support sending code-type objects. The alternate approach was putting custom plotting code in the local storage of the GUI computer. I prefer the former approach as it stays true to the emerging ARTIQ tenant that "the Master PC defines state" and has all user data, code, etc.

Regarding 2, in Neutral atom and Penning trap experiments one real-time observable is atom/ion images. So we'd want to broadcast and real-time plot 2-dimensional datasets for images. Device -> Master -> GUI

Regarding 4, in support of experiment browsing and replay there needs to be some way of mapping HDF5 back to the python world. Not sure if HDF5 permits inclusion of a per-record comment. If so this could be used to not the original python type in the HDF5 file when it comes time to map HDF5 -> Python.

Questions

"What Python types should map to HDF5 native types and how?" I think the following would cover most needs. Includes up to 2d arrays. Plus pickle arbitrary stuff. Does anybody need UTF-8 support?

    tests = [
        bool(True),
        [True, False],
        int(1234),
        [1, 2, 3]*3,
        float(1234),
        [1.1, 1.2, 1.3]*3,
        "asdf",  
        ["asdf", "lkj"]*3,
        np.array(np.random.random(10), dtype=np.float32),
        np.array(np.random.random(10), dtype=np.float64),
        np.array(np.random.random(10), dtype=np.int32),
        np.array(np.random.random(10), dtype=np.int64),
        np.array(np.random.rand(10, 10), dtype=np.int32),
        np.array(np.random.rand(10, 10), dtype=np.int64),
        np.array(np.random.rand(10, 10), dtype=np.float32),
        np.array(np.random.rand(10, 10), dtype=np.float64)
    ]

[jordens]

IMHO the current type support in PYON is sufficient.

sync_struct's ability should remain very limited. The only thing beyond list and dict ops, that I would like to see is the ability to update elements in an ndarray. If -- beyond single element updates -- slices and fancy indexing are difficult, scratch those.

PYON-HDF5: the mapability should be limited to the "natural" cases: homogeneous n-dim arrays of the following: intersection of the numpy and hdf5 numeric types, strings.

[sbourdeauducq]

List of strings are slightly problematic, as all list elements need to have the same type and the maximum length of the string is part of the type in HDF5 (similar to C). That can be solved by examining all elements in the list and allocating the largest size, but this can be inefficient and do you need lists of strings in the first place?

For n-dimensional arrays, should we support lists of lists, or only numpy arrays?

[jordens]

Sure. ndarrays of strings are also fixed element length. I would just enforce/expect that datasets to be saved in hdf5 are ndarrays. Trying to come up with yet another mapping of a certain subset of python objects to hdf5 seems pointless to me. Especially since I suspect that they are all converted to numpy stuff anyway.

[sbourdeauducq]

What about converting lists of scalars to numpy arrays implicitly? That would cover the frequent (I believe) case where points are added one by one to a plot, and the data is saved later.

Or should we enforce that a numpy array be created from the start? The advantage of the latter approach is that the rule is simple, but it transmits at least twice the data to the clients (once for initialization, and then each element is mutated) and the data has to be (typically zero-) padded while it is generated.

[jordens]

I would prefer to see ndarrays used widely (I have tried to make that point a few times). There are a few things i don't like about lists as pedestrian replacements for ndarrays: the elements might need type checking/coercion, the math and analysis would be done on ndarrays anway, with arrays you get useful "missing-data" handling, you can assume that the "results" array has the same shape as the full "scan points" array even if the scan was not completed...

Zero-padding is rather unphysical. There is nothing special about zero. You would usually initialize it "empty" with all points masked as missing data.

Also, appending to a list as a way of communicating incremental results does not really sound like the natural thing to do. Would you not rather do something like broadcasting "(x,y)" pairs and then track those where needed (on the GUI and on the worker)?

And ndarrays as the only data format also seems likely since that is what would come out of a hdf5 on restore/browse. Distinguishing ndarrays from list on restore/browse would be absurd.

And I don't see the need to unittest h5py's ability to do what it is designed to do (store ndarrays).

Ceterum censeo (off-topic here) that h5py might not be the right level in the stack for us. In a few months we will find h5py as the lowest abstraction layer below pytables and pandas. People will generally want to use pandas for wrestling these data out of mere convencience.

[sbourdeauducq]

Masked arrays are nice, but they cannot take a round trip through HDF5:

>>> y = np.ma.masked_array([0, 1, 2], [True, False, False])
>>> y
masked_array(data = [-- 1 2],
             mask = [ True False False],
       fill_value = 999999)
>>> f = h5py.File("test.h5", "w")
>>> f.create_dataset("test", data=y)
>>> f.close()
>>> f = h5py.File("test.h5", "r")
>>> f.get("test").value
array([0, 1, 2])

So it seems an additional storage convention on top of HDF5 would be needed for those.

[sbourdeauducq]

Pandas deals with the problem by using NaN and silently converting ints to floats upon insertion of NaN:

>>> df = pandas.DataFrame([10, 20, 30])
>>> df[0]
0    10
1    20
2    30
Name: 0, dtype: int64
>>> masked = df[df > 0]
>>> masked[0]
0    10
1    20
2    30
Name: 0, dtype: int64
>>> masked = df[df > 10]
>>> masked[0]
0   NaN
1    20
2    30
Name: 0, dtype: float64

[sbourdeauducq]

So you can get funny problems:

>>> df = pandas.DataFrame([2**60+1, 20, 30])
>>> df[0][0]
1152921504606846977
>>> int(df[df > 20][0][0])
1152921504606846976
>>> int(df[df > 10][0][0])
1152921504606846977

[sbourdeauducq]

http://pandas.pydata.org/pandas-docs/stable/gotchas.html#nan-integer-na-values-and-na-type-promotions

[jordens]

ACK. There is some stuff in HDF5 about default values but that doesn't look very useful. The NaN/missing data/invalid data/masked thing has been going on for years. I remember a lot of discussion and proposals at the numpy level. Don't know what happened there.

[sbourdeauducq]

What about going numpy-only and supporting missing values for floating point arrays only, using NaN? It seems to me that in a lot of cases, missing values come from measurements that return either floating point numbers, or integers small enough to be represented exactly by a double (e.g. photon counts).

If pandas is used to generate or analyze the data, this should match its behavior too.

[jordens]

Sounds good.

[sbourdeauducq]

How do we deal with booleans? They look good in the GUI (when broadcasted), but they need to be converted to integers before they can be saved to HDF5.

I think it should be OK to convert them implicitly to int8 before saving. When the experiment analysis is replayed, it will get an int instead of a bool, but Python should grok that in most practical cases. Or we can add HDF5 metadata to indicate that the value is boolean, and the replayer will convert to bool.

[jordens]

Converting to int8 sounds good to me.

[ghost]

Confirmed that the following runs without error.

from artiq import *
import numpy as np
import time

class TestHDF5(EnvExperiment):
    """Test HDF5"""
    def build(self):
        pass

    def run(self):

        tests = [
            True,
            [True, False],
            int(1234),
            [1, 2, 3]*3,
            float(1234),
            [1.1, 1.2, 1.3]*3,
            # complex(1, 1),
            # [complex(1,1), complex(1,2), complex(1,3)]*3,
            "asdf",
            ["asdf", "lkj"]*3,
            np.array(np.random.random(10), dtype=np.float32),
            np.array(np.random.random(10), dtype=np.float64),
            np.array(np.random.random(10), dtype=np.int32),
            np.array(np.random.random(10), dtype=np.int64),
            np.array(np.random.random(10), dtype=np.complex64),
            np.array(np.random.rand(10, 10), dtype=np.float32),
            np.array(np.random.rand(10, 10, 10), dtype=np.float32)
        ]

        for t in tests:
            self.set_dataset("test", t,
                              persist=False, save=False, broadcast=False)
            self.set_dataset("test", t,
                              persist=False, save=False, broadcast=True)
            self.set_dataset("test", t,
                              persist=False, save=True, broadcast=False)
            self.set_dataset("test", t,
                              persist=True, save=False, broadcast=False)
            self.set_dataset("test", t,
                              persist=True, save=True, broadcast=True)

[ghost]

The log output is distressingly long in the event of using an unsupported type. A try block in pyon could have caught this and terminated processing of the experiment is a less verbose fashion. Ideally there would be 1-2 lines of log output.

2015-10-28 12:29:05,907 INFO:worker(26):print:KeyError: <class 'complex'> artiq.master.worker.WorkerError: Worker ended

Here's what I see instead.

artiq.master.worker.WorkerError: Worker ended while attempting to receive data
2015-10-28 12:29:05,842 ERROR:worker(26):root:Worker terminating with exception
2015-10-28 12:29:05,845 INFO:worker(26):print:Traceback (most recent call last):
2015-10-28 12:29:05,848 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 198, in main
2015-10-28 12:29:05,850 INFO:worker(26):print:    exp_inst.run()
2015-10-28 12:29:05,853 INFO:worker(26):print:  File "/home/jwbritto/gitlab/nistpenning/artiq/repository/diagnostic/test_hdf5.py", line 36, in run
2015-10-28 12:29:05,855 INFO:worker(26):print:    persist=False, save=False, broadcast=True)
2015-10-28 12:29:05,858 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/language/environment.py", line 222, in set_dataset
2015-10-28 12:29:05,860 INFO:worker(26):print:    return self.__dataset_mgr.set(key, value, broadcast, persist, save)
2015-10-28 12:29:05,862 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_db.py", line 184, in set
2015-10-28 12:29:05,864 INFO:worker(26):print:    self.broadcast[key] = (persist, value)
2015-10-28 12:29:05,866 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/sync_struct.py", line 179, in __setitem__
2015-10-28 12:29:05,869 INFO:worker(26):print:    "value": value})
2015-10-28 12:29:05,870 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 35, in parent_action
2015-10-28 12:29:05,871 INFO:worker(26):print:    put_object(request)
2015-10-28 12:29:05,873 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 19, in put_object
2015-10-28 12:29:05,874 INFO:worker(26):print:    ds = pyon.encode(obj)
2015-10-28 12:29:05,876 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 143, in encode
2015-10-28 12:29:05,878 INFO:worker(26):print:    return _Encoder(pretty).encode(x)
2015-10-28 12:29:05,879 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:29:05,881 INFO:worker(26):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:29:05,882 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in encode_dict
2015-10-28 12:29:05,883 INFO:worker(26):print:    for k, v in x.items()])
2015-10-28 12:29:05,885 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in <listcomp>
2015-10-28 12:29:05,886 INFO:worker(26):print:    for k, v in x.items()])
2015-10-28 12:29:05,887 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:29:05,889 INFO:worker(26):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:29:05,890 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in encode_dict
2015-10-28 12:29:05,891 INFO:worker(26):print:    for k, v in x.items()])
2015-10-28 12:29:05,893 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in <listcomp>
2015-10-28 12:29:05,894 INFO:worker(26):print:    for k, v in x.items()])
2015-10-28 12:29:05,895 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:29:05,897 INFO:worker(26):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:29:05,898 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 85, in encode_tuple
2015-10-28 12:29:05,899 INFO:worker(26):print:    r += ", ".join([self.encode(item) for item in x])
2015-10-28 12:29:05,901 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 85, in <listcomp>
2015-10-28 12:29:05,902 INFO:worker(26):print:    r += ", ".join([self.encode(item) for item in x])
2015-10-28 12:29:05,903 INFO:worker(26):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:29:05,905 INFO:worker(26):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:29:05,907 INFO:worker(26):print:KeyError: <class 'complex'>
2015-10-28 12:29:05,953 ERROR:master:artiq.master.scheduler:got worker exception in run stage, deleting RID 26
Traceback (most recent call last):
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/scheduler.py", line 279, in _do
    completed = await run.run()
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/scheduler.py", line 32, in worker_method
    return await m(*args, **kwargs)
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 220, in run
    completed = await self._worker_action({"action": "run"})
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 196, in _worker_action
    completed = await self._handle_worker_requests()
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 153, in _handle_worker_requests
    obj = await self._recv(self.watchdog_time())
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 141, in _recv
    raise WorkerError("Worker ended while attempting to receive data")
artiq.master.worker.WorkerError: Worker ended while attempting to receive data
2015-10-28 12:30:39,019 ERROR:worker(27):root:Worker terminating with exception
2015-10-28 12:30:39,021 INFO:worker(27):print:Traceback (most recent call last):
2015-10-28 12:30:39,022 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 198, in main
2015-10-28 12:30:39,023 INFO:worker(27):print:    exp_inst.run()
2015-10-28 12:30:39,025 INFO:worker(27):print:  File "/home/jwbritto/gitlab/nistpenning/artiq/repository/diagnostic/test_hdf5.py", line 36, in run
2015-10-28 12:30:39,027 INFO:worker(27):print:    persist=False, save=False, broadcast=True)
2015-10-28 12:30:39,028 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/language/environment.py", line 222, in set_dataset
2015-10-28 12:30:39,030 INFO:worker(27):print:    return self.__dataset_mgr.set(key, value, broadcast, persist, save)
2015-10-28 12:30:39,031 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_db.py", line 184, in set
2015-10-28 12:30:39,033 INFO:worker(27):print:    self.broadcast[key] = (persist, value)
2015-10-28 12:30:39,034 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/sync_struct.py", line 179, in __setitem__
2015-10-28 12:30:39,036 INFO:worker(27):print:    "value": value})
2015-10-28 12:30:39,038 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 35, in parent_action
2015-10-28 12:30:39,039 INFO:worker(27):print:    put_object(request)
2015-10-28 12:30:39,041 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker_impl.py", line 19, in put_object
2015-10-28 12:30:39,042 INFO:worker(27):print:    ds = pyon.encode(obj)
2015-10-28 12:30:39,043 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 143, in encode
2015-10-28 12:30:39,045 INFO:worker(27):print:    return _Encoder(pretty).encode(x)
2015-10-28 12:30:39,047 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:30:39,049 INFO:worker(27):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:30:39,050 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in encode_dict
2015-10-28 12:30:39,052 INFO:worker(27):print:    for k, v in x.items()])
2015-10-28 12:30:39,053 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in <listcomp>
2015-10-28 12:30:39,054 INFO:worker(27):print:    for k, v in x.items()])
2015-10-28 12:30:39,056 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:30:39,057 INFO:worker(27):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:30:39,058 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in encode_dict
2015-10-28 12:30:39,060 INFO:worker(27):print:    for k, v in x.items()])
2015-10-28 12:30:39,061 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 99, in <listcomp>
2015-10-28 12:30:39,062 INFO:worker(27):print:    for k, v in x.items()])
2015-10-28 12:30:39,064 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:30:39,065 INFO:worker(27):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:30:39,066 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 85, in encode_tuple
2015-10-28 12:30:39,068 INFO:worker(27):print:    r += ", ".join([self.encode(item) for item in x])
2015-10-28 12:30:39,069 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 85, in <listcomp>
2015-10-28 12:30:39,070 INFO:worker(27):print:    r += ", ".join([self.encode(item) for item in x])
2015-10-28 12:30:39,072 INFO:worker(27):print:  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/protocols/pyon.py", line 135, in encode
2015-10-28 12:30:39,073 INFO:worker(27):print:    return getattr(self, "encode_" + _encode_map[type(x)])(x)
2015-10-28 12:30:39,074 INFO:worker(27):print:KeyError: <class 'complex'>
2015-10-28 12:30:39,118 ERROR:master:artiq.master.scheduler:got worker exception in run stage, deleting RID 27
Traceback (most recent call last):
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/scheduler.py", line 279, in _do
    completed = await run.run()
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/scheduler.py", line 32, in worker_method
    return await m(*args, **kwargs)
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 220, in run
    completed = await self._worker_action({"action": "run"})
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 196, in _worker_action
    completed = await self._handle_worker_requests()
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 153, in _handle_worker_requests
    obj = await self._recv(self.watchdog_time())
  File "/home/jwbritto/anaconda3/envs/py35/lib/python3.5/site-packages/artiq/master/worker.py", line 141, in _recv
    raise WorkerError("Worker ended while attempting to receive data")
artiq.master.worker.WorkerError: Worker ended while attempting to receive data

[ghost]

https://github.com/m-labs/artiq/commit/40b4129c65645c37a88cf0d6be96117150a0f870 does not include a unit test or update to ARTIQ documentation. Regarding former, we care very much about data integrity. A nice unit test: Generate random data for each of the supported types, call set_dataset(), read back from HDF5 and then compare.

[sbourdeauducq]

From the commit: "If the dataset is saved, it must be a scalar (bool, int, float or NumPy scalar) or a NumPy array." What is unclear about this?

[sbourdeauducq]

The unittest will be updated with readback as part of experiment replay, there is no point in doing it now. If you are worried, you can look at the output of the test manually using h5dump.

Your trace would already be shorter if a) you were running the latest version b) it didn't include two experiments.

[ghost]

From the commit: "If the dataset is saved, it must be a scalar (bool, int, float or NumPy scalar) or a NumPy array."

Commit comments are not user documentation.

[sbourdeauducq]

This is in the docstring for set_dataset.

[ghost]

OK https://github.com/m-labs/artiq/blob/40b4129c65645c37a88cf0d6be96117150a0f870/artiq/language/environment.py#L212

I was looking here.

m-labs.hk/artiq/manual/core_language_reference.html?highlight=set_dataset

On Thu, Oct 29, 2015 at 10:11 AM, Sébastien Bourdeauducq < notifications@github.com> wrote:

This is in the docstring for set_dataset.

— Reply to this email directly or view it on GitHub https://github.com/m-labs/artiq/issues/145#issuecomment-152229909.