brimdata / zed

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.38k stars 67 forks source link

uniq causes lateral subquery to hang #5294

Open philrz opened 1 week ago

philrz commented 1 week ago

tl;dr

This gives the one expected line of output, but then hangs and does not exit.

$ echo '3 1 4 1 5 9 2 6 5 3 5' | zq -z 'array:=collect(this) | over array => (sort | uniq | array:=collect(this))' -
{array:[1,2,3,4,5,6,9]}
[...never exits...]

Details

Repro is with Zed commit 2357e17. I bumped into it while responding to a community Slack thread.

If I rebuild the pipeline gradually, I can see where it starts to hang.

This exits ok:

$ zq -version
Version: v1.17.0-74-g2357e178

$ echo '3 1 4 1 5 9 2 6 5 3 5' | zq -z 'array:=collect(this) | over array => (sort)' -
1
1
2
3
3
4
5
5
5
6
9

But introducing uniq causes the hang.

$ echo '3 1 4 1 5 9 2 6 5 3 5' | zq -z 'array:=collect(this) | over array => (sort | uniq)' -
1
2
3
4
5
6
9
[...never exits...]