datafuselabs / databend

๐——๐—ฎ๐˜๐—ฎ, ๐—”๐—ป๐—ฎ๐—น๐˜†๐˜๐—ถ๐—ฐ๐˜€ & ๐—”๐—œ. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
https://docs.databend.com
Other
7.31k stars 704 forks source link

feat: select support consume stream #15541

Closed zhyass closed 2 weeks ago

zhyass commented 2 weeks ago

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Key Features

  1. Added WITH CONSUME clause support for stream.

  2. Allows execution of SELECT ... FROM stream_name WITH CONSUME ... to consume data.

  3. Stream data is consumed as part of the query execution, making it easier to manage and process stream data within SQL queries.

NOTE:

Please note that streams marked with "WITH CONSUME" are only guaranteed to be consumed if the query completes successfully.

Example Usage

mysql> create table t(a int);
Query OK, 0 rows affected (0.14 sec)

mysql> insert into t values(1),(2);
Query OK, 2 rows affected (0.14 sec)

mysql> create stream s on table t append_only=false;
Query OK, 0 rows affected (0.10 sec)

mysql> insert into t values(3);
Query OK, 1 row affected (0.12 sec)

mysql> delete from t where a=1;
Query OK, 1 row affected (0.21 sec)

mysql> select * from s;
+------+---------------+----------------------------------------+------------------+
| a    | change$action | change$row_id                          | change$is_update |
+------+---------------+----------------------------------------+------------------+
|    1 | DELETE        | 842124d0a6d94111b215bfe7973b5c2d000000 |                0 |
|    3 | INSERT        | e5009676789b4584b0e18f4fbb660ca3000000 |                0 |
+------+---------------+----------------------------------------+------------------+
2 rows in set (0.36 sec)
Read 4 rows, 437.00 B in 0.099 sec., 40.56 rows/sec., 4.33 KiB/sec.

mysql> select a from s with consume as ss where ss.change$action = 'INSERT';
+------+
| a    |
+------+
|    3 |
+------+
1 row in set (0.31 sec)
Read 4 rows, 437.00 B in 0.057 sec., 69.83 rows/sec., 7.45 KiB/sec.

mysql> select * from s;
Empty set (0.16 sec)
Read 0 rows, 0.00 B in 0.008 sec., 0 rows/sec., 0.00 B/sec.

mysql> select * from t with consume;
ERROR 1105 (HY000): StorageUnsupported. Code: 3902, Text = WITH CONSUME only support in STREAM.

Tests

Type of change


This change isโ€‚Reviewable

BohuTANG commented 2 weeks ago

@soyeric128 please update this feature to our doc, thanks.