search

dask / dask

Parallel computing with task scheduling
https://dask.org
BSD 3-Clause "New" or "Revised" License
12.43k stars 1.7k forks source link

Improve blockwise_meta exception handling #4952

Open pentschev opened 5 years ago

pentschev commented 5 years ago

Current blockwise_meta/blockwise implementation defaults to dtype construction of Dask.Array when an non-trivial exception is raised when computing _meta. This is suboptimal for two reasons:

  1. It's difficult to trace that the defaulting happened within blockwise for somewhat complex user code;
  2. Exceptions that come from the array compute task (with NumPy, CuPy, etc.) are not raised to the user during graph construction.

Ideally, we would handle all possible exceptions, with all remaining ones assumed to be user's mistake, such as passing casting to cupy.einsum (which isn't supported), as discussed in https://github.com/dask/dask/pull/4914.

Given that _meta now exists, Dask has become a universal scheduler, which probably means we can't handle every single possible exception, but we should at least allow developers to know about such cases, as well as letting users know they possibly made a mistake. We should therefore let most exceptions pass back to the user in blockwise.

This is loosely related to #4951, where we wish to allow users/developers be alerted of conditions that Dask can't handle and should probably be better handled by the library which Dask is using as a backend through __array_function__.

jakirkham commented 5 years ago

Just checking in, is this resolved by the xref'd PRs above @pentschev or is there still more to do here?

pentschev commented 5 years ago

I'd say that #4954 improved this dramatically, but I don't consider it resolved, as per my comment https://github.com/dask/dask/pull/4954#issuecomment-502717153. In particular https://github.com/dask/dask/pull/4954/files#diff-3598e11c3b3a16881fecff61abdf1f4cR118 would still prevent other exceptions from a backend library's code to be raised. We don't yet have a general solution for that, so I would prefer to leave this open for now.