risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.06k stars 580 forks source link

bug: compactor can not pull tasks from meta #18257

Open wcy-fdu opened 2 months ago

wcy-fdu commented 2 months ago

Describe the bug

Summarize

compactor can not pull tasks from meta, which cause compactor write stall and barrier stuck. Recover after restarting the cluster.

image

Log: https://grafana.prod.risingwave.cloud/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=P557A01600E289668&var-namespace=rwc-g1hm7vadpqepspe0hkqlq5jc08-test-private-link&var-pod=risingwave-compactor-default-8454565bcb-rft6x&var-search=fail&from=1724334653000&to=1724345453000

Metrics: https://grafana.prod.risingwave.cloud/d/EpkBw5W4k/risingwave-dev-dashboard?from=1724334653000&orgId=1&to=1724345453000&var-component=All&var-datasource=edtasmghzt3wgf&var-instance=risingwave&var-namespace=rwc-g1hm7vadpqepspe0hkqlq5jc08-test-private-link&var-pod=All&var-table=All

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

Li0k commented 2 months ago

https://github.com/risingwavelabs/risingwave/pull/17979

Adjusted the base level compaction strategy, we haven't reproduced this bug yet, if it happens again I'll look into it further.

github-actions[bot] commented 3 weeks ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄

wcy-fdu commented 3 weeks ago

Did we observed similar issue recently? If not, I'd like to close this issue for a while. cc @Li0k