search

SciTools / biggus

:no_entry: [DEPRECATED] Virtual large arrays and lazy evaluation.
http://biggus.readthedocs.io/
GNU Lesser General Public License v3.0
54 stars 27 forks source link

Setting fill_value interfere with iris._merge #94

Closed ocefpaf closed 9 years ago

ocefpaf commented 10 years ago

Assigning a fill_value property in Array.fill_value will create a masked array default fill_value even when the original data is not masked.

That prevents cubes merging in iris.

The failure happens when slicing cubes and trying to merge them again. If one of the slices happens to be masked and the other is not the merging will fail. That occurs because the merging will be between the original fill_value, from the original cube metadata, and the default fill_value assigned by biggus lazy_data().

Don't know if the best approach would be for iris to preserve the original fill_value when slicing data that is masked (even when there is nothing to fill -- that would be an iris fix), or if biggus could avoid attaching this property to non-masked objects.

pelson commented 10 years ago

or if biggus could avoid attaching this property to non-masked objects.

I've been a little uncomfortable with biggus doing this too. It is certainly a break from numpy to make the Array object fundamentally have a fill_value and it doesn't sit well in some circumstances IMHO.

ocefpaf commented 10 years ago

@pelson Thanks for looking into this. Is there an easy solution? What would be the impact of adding this attribute only if the original array is a masked array? Currently I have to go through all my cubes and modify the fill_value to be the same as the one assigned by biggus so I can merge them.

pelson commented 9 years ago

@ocefpaf - it is hard to be certain, but I think #105 addressed this problem. Do you have a simple case with which we can reproduce the problem?

ocefpaf commented 9 years ago

Yes I do . I will test this later today (I am away from my laptop now) and close the issue if it works.

ocefpaf commented 9 years ago

Yes, this is fixed! Thanks a lot @pelson

PS: I do not have a simple case, but I just removed my workaround for this issue and everything worked out fine in my notebooks.