search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://www.risingwave.com/slack
Apache License 2.0
6.76k stars 557 forks source link

Locking between DML and DDL #10289

Open chenzl25 opened 1 year ago

chenzl25 commented 1 year ago

Is your feature request related to a problem? Please describe.

As we continue to develop and refine our database operations, we need to consider how we handle locks during DML and DDL (including scaling) operations.

Currently, it's unclear how our system handles locks, especially when DDL and DML operations are executed concurrently on the same objects. Potential issues could arise from conflicts between these operations, which could lead to data inconsistency or loss.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

fuyufjh commented 1 year ago

May I ask your current ideas or plan?

I think DDLs should be mutual-exclusive with each other; while DMLs are not, where conflicts can be resolved regarding their order of arriving at MaterializeExecutor.

chenzl25 commented 1 year ago

May I ask your current ideas or plan?

I think DDLs should be mutual-exclusive with each other; while DMLs are not, where conflicts can be resolved regarding their order of arriving at MaterializeExecutor.

Currently, our DDLs could interrupt other concurrent DMLs which could cause data loss and inconsistency. My idea is to use a Tokio::RwLock to ensure DDLs don't affect the concurrent DMLs.

liurenjie1024 commented 1 year ago

I think RwLock may not be enough. We should disable concurrent ddl/dml with lock in meta.

github-actions[bot] commented 2 months ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄