inaos / iron-array

2 stars 0 forks source link

Optimize expressions with views #581

Closed martaiborra closed 2 years ago

martaiborra commented 2 years ago

Currently, the expression evaluation defaults to use ITERCHUNK method whenever there is a view inside an expression (https://github.com/inaos/iron-array/blob/f71e29cbac9aa6ba8be2fc494dc631e62b6d868d/src/iarray_expression.c#L611). In order to get the same performance than iterblosc we need our views to be more flexible and accept this method as well.

martaiborra commented 2 years ago

We should also be able to work in parallel with views, which means we should be able to remove the following line: https://github.com/inaos/iron-array/blob/f71e29cbac9aa6ba8be2fc494dc631e62b6d868d/src/iarray_views.c#L573

martaiborra commented 2 years ago

If we would like to support the ITERBLOSC eval method in slices, we would have to change how views are built. The schunk size would have to be the size from the view and not from the original viewed schunk. Furthermore, the chunk to be retrieved from the original container will have to be computed each time in the post filter, and in the worst scenario, we would have to read from 2 chunks when dim = 1, 4 chunks when dim = 2, 8 chunks when dim = 3.... So instead, what we can do is support ITERBLOSC in views only when the view's shape is the same than the viewed container's shape.

martaiborra commented 2 years ago

We did a benchmark to compare the old way of evaluating expressions with the new way, evaluating type views, and as expected, the evaluation is faster with the changes from PR #598 than with the old version.

Old version: newplot

New version: newplot-2