search

scikit-hep / awkward-0.x

Manipulate arrays of complex data structures as easily as Numpy.
BSD 3-Clause "New" or "Revised" License
215 stars 39 forks source link

Allow ufunc on VirtualArray to use numpy 'out' #223

Closed nsmith- closed 4 years ago

nsmith- commented 4 years ago

If all inputs are plain numpy or only wrapped by VirtualArray, then numpy should not have any trouble doing in-place ops. If any other awkward types are passed in, they will fail in their respective ufunc callbacks. So we are really only losing the ability to throw early before materializing in all cases.

jpivarski commented 4 years ago

I didn't want to open this up because in-place operations can't be done in general. Allowing it sometimes means the user has to understand their layout (not just their high-level types), and this is becoming more hidden, not less.

If the output of guaranteed to be flat NumPy and it's a ufunc, I think that means that the inputs must all be flat NumPy (possibly virtual), too. Maybe the constrains could be clearer if we had a different interface?

Maybe this should only be done through Dask?

nsmith- commented 4 years ago

If the output of guaranteed to be flat NumPy and it's a ufunc, I think that means that the inputs must all be flat NumPy (possibly virtual), too. Maybe the constrains could be clearer if we had a different interface?

This is probably true, although I don't know how there could be a different interface--this is already a callback here. I didn't test that specifically because if its not the case, then after materializing, the subsequent call with materialized arrays will throw anyway. I'm basically only after one class of exception:

x = np.full(5, 1.1)
a = np.ones(5)
b = ak.VirtualArray(lambda: x, (), type=ak.type.ArrayType(5, np.double))
a *= b

where all the inputs are numpy or virtual-wrapped numpy and the output is numpy.

jpivarski commented 4 years ago

I don't think I'll be allowing this in Awkward 1 (structure of the layout is too hidden to make it easily explainable when you can do it and when you can't). But I can approve this in Awkward 0 if you want the temporary ability to do this.

nsmith- commented 4 years ago

After some discussion we decided not best to make special exception for VirtualArray, especially pending the awkward1 situation where virtual-ness is more concealed under the AwkwardArray type.