brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.38k stars 64 forks source link

Hang caused by applying "merge this" post-fork #5285

Open philrz opened 4 weeks ago

philrz commented 4 weeks ago

tl;dr

This hangs for me:

$ for n in $(seq 1 701); do  echo "{ts:$n}"; done | zq -z 'fork (=>pass =>pass) | merge this' -
{ts:1}
{ts:1}
{ts:2}
{ts:2}
...
{ts:699}
{ts:700}

Details

Repro is with Zed commit 491dadf.

The repro shown above runs fine if the upper end of the seq is 700, but once I change it to 701, it hangs. I originally found this in the Zed lake context, where the magic crossover was when the upper end of the seq went from 54656 (works ok) to 54657 (hangs).

I also noticed it doesn't happen if I instead do merge ts, which indeed is probably what a user would have intended here. But #3678 leads me to believe merge this should be acceptable.