knizhnik / imcs

In-Memory Columnar Store extension for PostgreSQL
Apache License 2.0
201 stars 33 forks source link

Window function continues executing even after last element of input series #61

Closed pavitrakumar78 closed 3 years ago

pavitrakumar78 commented 3 years ago

Hi,

I'm working a lot with window functions and wanted to know if this behavior is expected because it's very different from what happens with vanilla PG12.

The issue: Assume there's a stock whose prices range from 2020-07-30 to 2020-09-24. If I use cs_window_avg with period 14 on prices using the get method like this: sample_table_get('AA.US', '2020-07-30', '2020-09-04'), I expect only 27 rows, but instead I get 27+13 rows (40 rows total). Meaning, the window function continues even after the last row whereas, in normal PG12, window functions only output until the last element of the input series.

To reproduce this problem, please execute the commands in this pastebin: https://pastebin.com/60WQqSbC

I understand I can again use shifts but I already use a lot of shifts to compensate for IMCS not supporting NULL values and adding shifts again for this would be very hard to do and moreover, since I don't always know what the end date will be, having the moving average work only till the end of the given input series will help make queries a lot simpler.

If this fix can't be supported/added in the imcs-master branch, then any help with what functions to patch in imcs files would be much appreciated so that I can clone and path in my copy.

Edit: I was messing with the func.c file and added this small correction (before the loop over tile_size) to the _next function right under the line #define IMCS_WINDOW_AGG_DEF(TYPE, AGG_TYPE, MNEM, NEXT, INIT) (which I presume is the iterator for moving the window?):

    if(tile_size>ctx->interval){                                        
        tile_size -= (ctx->interval-1);                                 
    } 

This seems to fix the issue of having extra elements - but is this fix safe? do I have to create a similar patch elsewhere?

pavitrakumar78 commented 3 years ago

Thanks for the fix! The window functions only till the last element now!