Debugging by comparing batch and streaming’s results of each levels's sub-plan.

[lmatz]

For a sql query, we first translate it into a logical plan. Each time, we get rid of the highest operator and then convert this sub plan into both batch and streaming plan.

In the end, we would have a series of sub plans whose batch and streaming results must be the same.

Finally, we do binary search on all the comparisons. So we can determine the location of the bugs in a systematic and automatic approach.

Is this tool worth implementing? Opinions are welcome!

[lmatz]

It sounds like it should be an independent tool. batch and streaming should be generalized as two different execution engines. logical plan should also be an independent representation so that all optimizers of different execution engines can be integrated.

[st1page]

when a logical plan node convert to streaming plan node, we can store the logical sub plan tree in the stream plan node, and we can insert a MViewNode on the top of any stream node, and we can compare the query on the MV and the batch query of the logcial plan in that stream node

[lmatz]

when a logical plan node convert to streaming plan node, we can store the logical sub plan tree in the stream plan node, and we can insert a MViewNode on the top of any stream node, and we can compare the query on the MV and the batch query of the logcial plan in that stream node

Agree. We may take extra care for those cases, i.e. two-phase aggregation, that several physical operators together may correspond to a single logical operator?

[liurenjie1024]

It doesn't seem reasonable to me. Logical plan of materialized view maintaince is in fact incremental algebra, not relational algebra, so same query in mv maintaince maybe different in batch and stream.

[lmatz]

These types of bugs are rare in reality, so they are not worth it.

If later, when more fuzz testing tools are introduced and it is necessary, we reopen it.

[kwannoel]

fyi something like this is being implemented in #10558, tracked by https://github.com/risingwavelabs/risingwave/issues/10427.

It doesn't seem reasonable to me. Logical plan of materialized view maintaince is in fact incremental algebra, not relational algebra, so same query in mv maintaince maybe different in batch and stream.

Didn't quite understand it, maybe you can share some examples?

AFAIK main issue is with ordering, but I bypass that by sorting the output rows.

I think in general as long as most queries should match, we can use this approach to catch some logic bugs.

This is still experimental, but it was easy to implement, so I think it is worth a shot, just to try and see how effective it is in catching bugs.

If turns out most "bugs" it catches are not actually bugs, we can scrap it if there's no easy way to adapt it.

[st1page]

I think this issue is not saying simply to compare the result for the same test case but to use it as a kind of tool to debug. So I think #10558 is just related to this issue but does not implement the issue.