search

cadence-workflow / cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
https://cadenceworkflow.io
MIT License
8.33k stars 801 forks source link

Fix some persistence APIs to support sharded/partitioned SQL database and NoSQL database #4064

Open longquanzheng opened 3 years ago

longquanzheng commented 3 years ago

Is your feature request related to a problem? Please describe.

Initially we wanted to design our SQL layer to support sharded/partitioned SQL DB. Hence all query are intentially using shardID and avoid doing cross shard query. Even though we haven't found a real use case yet.

However, in PR https://github.com/uber/cadence/pull/4059 seems like we have to do cross shard query. Otherwise it will introduce a lot more complexity to the code.

We will this keep issue open, until we find some real use case of sharded SQL database in the future.

Proposed Solution A clear and concise description of what you want to happen. N/A

Additional context Add any other context or screenshots about the feature request here. N/A

meiliang86 commented 3 years ago

The current SQL interface/implementation works with sharded SQL (Uber iss going to use it), and we need to keep it that way. We need to fix #4059 so it does not rely on cross shard operations (query might be fine, but definitely no transactions)

Shaddoll commented 3 years ago

We found an issue with mysql schema. Current mysql implementation doesn't support sharding, we have to fix it so that we won't have cross shard transaction in sharded mysql.