uwdata / arquero

Query processing and transformation of array-backed data tables.
https://idl.uw.edu/arquero/
BSD 3-Clause "New" or "Revised" License
1.33k stars 64 forks source link

Allow nth_value to take a column as the 'nth' argument #375

Open ae3e opened 1 month ago

ae3e commented 1 month ago

I would like to use the nth_value function with a nth parameter coming from an another column (see the example below). But it fails. I don't really know if it's the expected behavior or if there's something wrong?

aq
  .table({
    value: [69, 108, 178, 207, 253, 268, 312, 281, 221, 142, 72, 52],
    index: [3, 2, 3, 5, 6, 8, 2, 2, 7, 5, 4, 5]
  })
  .derive({
    nth: (vv) => aq.op.nth_value(vv.value, vv.index)
  })

gives an error RuntimeError: data is not defined

nth always has to be a constant?

jheer commented 1 month ago

Thanks for posting. Yes, currently nth has to be an externally provided value, such as a constant or parameter. I will mark this as a feature request to enable column values as arguments as well.

ae3e commented 1 month ago

Thanks for your answer and this great lib,

In the meantime, I finally tried to add my first custom function. Seems to work but I still have to do dig a little more...

aq.addWindowFunction(
  "my_nth_value",
  {
    create: () => ({
      init: () => {}, // No initialization needed
      value: (window, columns) => {
        let value1 = window.value(window.index, columns)[1];
        value1 = +value1;
        if (!(value1 >= 0))
          throw Error("nth_value nth must be equal or greater than zero.");

        const i = window.i0 + value1;
        return i < window.i1 ? window.value(i, columns)[0] : null;
      }
    }),
    param: [2] // 2 field inputs, 0 extra parameters
  },
  { override: true }
);

and a simple use case

aq
  .table({
    value: [69, 108, 178, 207, 253, 268, 312, 281, 221, 142, 72, 52],
    index: [3, 2, 3, 5, 6, 8, 2, 2, 7, 5, 4, 5]
  })
  .derive({
    nth: (vv) => aq.op.my_nth_value([vv.value, vv.index])
  })