search

scipp / sciline

Build scientific pipelines for your data
https://scipp.github.io/sciline/
BSD 3-Clause "New" or "Revised" License
10 stars 2 forks source link

Sciline trips up on dask.distributed #163

Closed MridulS closed 4 months ago

MridulS commented 7 months ago

I tried running sciline with dask.distributed and it seems to trip up a code path in dask where it expects the keys in the task graph to be one of these types bytes, int, float, str and not the exotic ones we have.

I tried using the following code, maybe I did something wrong here?

from dask import distributed
from sciline.scheduler import DaskScheduler

client = distributed.Client()
scheduler = DaskScheduler(client.get)

pipeline.compute(Result, scheduler=client)

This results in a type error coming from https://github.com/dask/dask/blob/aa5a333fccd280b9535233865282019f7e3117ea/dask/core.py#L240

TypeError: Unexpected key type <class 'typing.NewType'> (key=__main__.Result)

I'm just suprised why don't we end up with these errors when we use dask.threaded

SimonHeybrock commented 6 months ago

Does it fail only when using NewType, or also with regular types as keys?

jokasimr commented 5 months ago

Yes it's a bit surprising we don't end up with the errors when using dask.threaded. According to the specification "A key is a str, bytes, int, float, or tuple thereof".