search

dask-contrib / dask-histogram

Histograms with task scheduling.
https://dask-histogram.readthedocs.io
BSD 3-Clause "New" or "Revised" License
23 stars 4 forks source link

Computing histogram with dask distributed cluster #6

Closed mpoehlmann closed 3 years ago

mpoehlmann commented 3 years ago

When running one of examples with dask distributed, I get an error. If I comment out the client line, it works.

from dask.distributed import Client
import dask_histogram as dh
import dask_histogram.boost as dhb
import dask.array as da
client = Client()
x = da.random.standard_normal(size=(100_000_000, 2), chunks=(10_000_000, 2))
h = dhb.Histogram(dh.axis.Regular(10, -3, 3), dh.axis.Regular(10, -3, 3), storage=dh.storage.Double())
h.fill(x)
h.compute()

I am using Python 3.8.6 installed with conda, boost-histogram version 1.2.1, dask version 2021.09.1, and dask-histogram version 0.2.1.dev1+g4907a18. Thank you for your work on this project!

distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.core - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.worker - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.core - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f14d26bed60>>, <Task finished name='Task-5' coro=<Worker.handle_scheduler() done, defined at /dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py:1045> exception=AttributeError("can't set attribute")>)
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.worker - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7fa9b10ddd60>>, <Task finished name='Task-5' coro=<Worker.handle_scheduler() done, defined at /dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py:1045> exception=AttributeError("can't set attribute")>)
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.core - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.worker - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f2d1b8fed60>>, <Task finished name='Task-5' coro=<Worker.handle_scheduler() done, defined at /dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py:1045> exception=AttributeError("can't set attribute")>)
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.protocol.core - CRITICAL - Failed to deserialize
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.core - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
distributed.worker - ERROR - can't set attribute
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f14d26bed60>>, <Task finished name='Task-269' coro=<Worker.handle_scheduler() done, defined at /dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py:1045> exception=AttributeError("can't set attribute")>)
Traceback (most recent call last):
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/worker.py", line 1047, in handle_scheduler
    await self.handle_stream(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/core.py", line 557, in handle_stream
    msgs = await comm.read()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/tcp.py", line 226, in read
    msg = await from_frames(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 111, in loads
    return msgpack.loads(
  File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/core.py", line 103, in _decode_default
    return merge_and_deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 475, in merge_and_deserialize
    return deserialize(header, merged_frames, deserializers=deserializers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 391, in deserialize
    deserialize(
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 407, in deserialize
    return loads(header, frames)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/serialize.py", line 86, in pickle_loads
    return pickle.loads(x, buffers=new)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 73, in loads
    return pickle.loads(x, buffers=buffers)
  File "/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/bag/core.py", line 401, in __setstate__
    self.dask, self.key = state
AttributeError: can't set attribute
---------------------------------------------------------------------------
KilledWorker                              Traceback (most recent call last)
<ipython-input-3-ff431c4d1e1e> in <module>
----> 1 h.compute()

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask_histogram/boost.py in compute(self)
    401             result_view = self.view(flow=True) + self._staged.compute().view(flow=True)
    402         else:
--> 403             result_view = self._staged.compute().view(flow=True)
    404         self[...] = result_view
    405         self._staged = None

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/base.py in compute(self, **kwargs)
    286         dask.base.compute
    287         """
--> 288         (result,) = compute(self, traverse=False, **kwargs)
    289         return result
    290 

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/dask/base.py in compute(*args, **kwargs)
    568         postcomputes.append(x.__dask_postcompute__())
    569 
--> 570     results = schedule(dsk, keys, **kwargs)
    571     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    572 

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/client.py in get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2687                     should_rejoin = False
   2688             try:
-> 2689                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2690             finally:
   2691                 for f in futures.values():

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   1964             else:
   1965                 local_worker = None
-> 1966             return self.sync(
   1967                 self._gather,
   1968                 futures,

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    858             return future
    859         else:
--> 860             return sync(
    861                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    862             )

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    324     if error[0]:
    325         typ, exc, tb = error[0]
--> 326         raise exc.with_traceback(tb)
    327     else:
    328         return result[0]

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/utils.py in f()
    307             if callback_timeout is not None:
    308                 future = asyncio.wait_for(future, callback_timeout)
--> 309             result[0] = yield future
    310         except Exception:
    311             error[0] = sys.exc_info()

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/tornado/gen.py in run(self)
    760 
    761                     try:
--> 762                         value = future.result()
    763                     except Exception:
    764                         exc_info = sys.exc_info()

/dark/dmpoehlm/software/miniconda/envs/lm2/lib/python3.8/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1829                             exc = CancelledError(key)
   1830                         else:
-> 1831                             raise exception.with_traceback(traceback)
   1832                         raise exc
   1833                     if errors == "skip":

KilledWorker: ("('standard_normal-hist-on-block-0efa75cec56b59492ad97060b3c80d95', 7)", <WorkerState 'tcp://169.237.42.20:46463', name: 2, memory: 0, processing: 10>)
douglasdavis commented 3 years ago

Thanks for raising the issue! Looks like there is a serialization issue with dask_histogram.boost.Histogram. I'll start looking into it today. In the meantime the collection interface is working with the distributed scheduler:

h = dh.factory(
    x,
    axes=(dh.axis.Regular(10, -3, 3), dh.axis.Regular(10, -3, 3)),
    storage=dh.storage.Double(),
)
h.compute()
douglasdavis commented 3 years ago

Just merged #7 -- it fixes the snippet that caused the error. Feel free to re-open if the issue persists. Thanks!