search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.1k stars 587 forks source link

split connector implementations into crates #16841

Open xxchan opened 6 months ago

xxchan commented 6 months ago

BTW, I think it's time to split connector implementations off and design a clear interface for it, as there are now community developers willing to develop new connectors. šŸ¤”

_Originally posted by @wangrunji0408 in https://github.com/risingwavelabs/risingwave/pull/16759#discussion_r1604355825_

related work https://github.com/risingwavelabs/risingwave/pull/12981

related discussion:

  • But default features is to make developers happy, which can be hard to fix.

The UDF module is touched by few people, so it makes sense to require manual enabling on the features.

However, imagine we're going to add conditional compiles for the connector module, which is actively developed and maintained by a considerable amount of developers. We still need the effort to inform them how to enable these features, especially the IDE part, which is not better than the approach above.

The point is the most of developers do not pay much attention to the compile time. If something comes naturally, it's good. If it requires extra effort, they might give up. Thus in my opinion, it's fine to treat it as a hidden treasure needed to be discovered by some manual efforts. šŸ˜„

_Originally posted by @BugenZhao in https://github.com/risingwavelabs/risingwave/pull/16759#discussion_r1606553437_

The point is the most of developers do not pay much attention to the compile time. If something comes naturally, it's good. If it requires extra effort, they might give up.

I 100% agree. However, my opinion from this fact is that it's better to offer "free" fast compile time for developers. If they are not good at it, we'd better make the default faster. Saving every one's time does good to the whole world!

_Originally posted by @xxchan in https://github.com/risingwavelabs/risingwave/pull/16759#discussion_r1606787396_

how to enable these features, especially the IDE part

I'm thinking whether it's possible to offer IDE experience without config for feature flags. e.g., cfg_or_panic(madsim).

One idea is that: we put each connector impl in separated crate, which is an optional dependency. If so, maybe the IDE can work without config.

_Originally posted by @xxchan in https://github.com/risingwavelabs/risingwave/pull/16759#discussion_r1606791754_

xxchan commented 5 months ago

From the diff in this PR, it seems the boundary is quite clear and it should be easy to split them out. https://github.com/risingwavelabs/risingwave/pull/17154/

The question is just which plan do we choose: feature flags? separated crates?