[Feature Request]: Real-Time Computing

[fengttt]

Is there an existing issue for the same feature request?

• [X] I have checked the existing issues.

Is your feature request related to a problem?

How do we do streaming?

I propose we add the following,

CREATE SOURCE XXX -- as if create a table KAFKA OPTIONS -- or other connectors WITH REFRESH EVERY 5 MINUTES;

CREATE SINK YYY -- as if it is create a view KAFKA OPTIONS --

-- or CREATE MATERIALIZED VIEW VVV;

Describe the feature you'd like

There should be a well defined subset of sql in materialized view.

Lets limit it to select where join group by without negation.

Documentation, Adoption, Use Case, Migration Strategy

https://doc.weixin.qq.com/doc/w3_AS0AvwapAOkQUuW8k73S0aonmrMEI?scode=AJsA6gc3AA8wehp2Wh

Taskes

• [ ] #10970
• [ ] #10971
• [ ] #11043
• [ ] #11601

[nnsgmsone]

I think create view is good enough. No need for the user to care whether it materializes or not.

[dengn]

The whole 0.8.0 thing for streaming is to move trace log tables from external tables to internal tables. Please define what feature will be implemented for 1.0 and what's their behavior.

[gavinyue]

The whole 0.8.0 thing for streaming is to move trace log tables from external tables to internal tables. Please define what feature will be implemented for 1.0 and what's their behavior.

A quick summary for the plan of 1.0.

1. Create Stream on Kafka Topic Like external table, users can create table over kafka topic.
2. Create and manage the Stream Processing Pipeline, calling Flow, process events in the stream.

[domingozhang]

Can I conclude as following: Create external stream with kafka

CREATE SOURCE | SINK CONNECTOR [IF NOT EXISTS] connector_name CONNECTOR_TYPE WITH( property_name = expression [, ...]);

Create stream

CREATE [OR REPLACE] [EXTERNAL] STREAM [IF NOT EXISTS] stream_name 
  ( { column_name data_type [KEY | HEADERS | HEADER(key)] } [, ...] )
  WITH ( property_name = expression [, ...] );

CREATE STREAM AS SELECT COLUMNS FROM another-stream

User needs a new privilege STREAM to create or drop a stream or a connector

I want to confirm whether all key words will be implemented in 1.0, such as KEY, HEADERS, property name and so on. thx @gavinyue

[gavinyue]

Can I conclude as following: Create external stream with kafka

CREATE SOURCE | SINK CONNECTOR [IF NOT EXISTS] connector_name CONNECTOR_TYPE WITH( property_name = expression [, ...]);

Create stream

CREATE [OR REPLACE] [EXTERNAL] STREAM [IF NOT EXISTS] stream_name 
  ( { column_name data_type [KEY | HEADERS | HEADER(key)] } [, ...] )
  WITH ( property_name = expression [, ...] );

CREATE STREAM AS SELECT COLUMNS FROM another-stream

User needs a new privilege STREAM to create or drop a stream or a connector

I want to confirm whether all key words will be implemented in 1.0, such as KEY, HEADERS, property name and so on. thx @gavinyue

Yes. For create, the above doc reflects with current design and planned in 1.0