risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.06k stars 581 forks source link

feat: add rw_rate_limit system catalog #19466

Closed xxchan closed 1 day ago

xxchan commented 2 days ago

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

It's not easy to know what jobs are rate limited, so add this.

I want to avoid iterating over all fragments, so I introduced a new fragment flag for FsFetch. This means rw_rate_limit won't show fs fragments created before this PR, and I think it's acceptable.

Checklist

Documentation

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

xxchan commented 2 days ago

This stack of pull requests is managed by Graphite. Learn more about stacking.

xxchan commented 2 days ago

Oops. Seems this system catalog is not isolated between databases 🤔

image

Some catalogs support this, but some don't: e.g,

yezizp2012 commented 2 days ago

Oops. Seems this system catalog is not isolated between databases 🤔

image

Some catalogs support this, but some don't: e.g,

  • rw_tables is isolated
  • rw_fragments is not

System catalogs about metadata of the streaming job are all not isolated between databases. They are used solely for debugging.

xxchan commented 2 days ago

Yes, I was also a little hesitant to add isolation which might hurt debugging. 🤪

xxchan commented 2 days ago

Merge activity