search

uber / cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
https://cadenceworkflow.io
MIT License
7.96k stars 772 forks source link

Introducing MAPQ: Multi-tenant, Auto-partitioned, Persistent Queue #6132

Closed taylanisikdemir closed 6 days ago

taylanisikdemir commented 2 weeks ago

What changed? See the README

How did you test it? Unit tests are added to cover basic initialization, policy based tree construction.

codecov[bot] commented 2 weeks ago

Codecov Report

Attention: Patch coverage is 81.19403% with 63 lines in your changes missing coverage. Please review.

Project coverage is 72.36%. Comparing base (83ebf7a) to head (18c1e1c).

:exclamation: Current head 18c1e1c differs from pull request most recent head 2643b58

Please upload reports for the commit 2643b58 to get more accurate results.

Additional details and impacted files | [Files](https://app.codecov.io/gh/uber/cadence/pull/6132?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber) | Coverage Δ | | |---|---|---| | [common/mapq/dispatcher/dispatcher.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Fdispatcher%2Fdispatcher.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvZGlzcGF0Y2hlci9kaXNwYXRjaGVyLmdv) | `90.00% <90.00%> (ø)` | | | [common/mapq/mapq.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Fmapq.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvbWFwcS5nbw==) | `93.75% <93.75%> (ø)` | | | [common/mapq/types/item.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Ftypes%2Fitem.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvdHlwZXMvaXRlbS5nbw==) | `91.66% <91.66%> (ø)` | | | [common/mapq/types/policy\_collection.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Ftypes%2Fpolicy_collection.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvdHlwZXMvcG9saWN5X2NvbGxlY3Rpb24uZ28=) | `94.91% <94.91%> (ø)` | | | [common/mapq/client\_impl.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Fclient_impl.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvY2xpZW50X2ltcGwuZ28=) | `72.72% <72.72%> (ø)` | | | [common/mapq/types/policy.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Ftypes%2Fpolicy.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvdHlwZXMvcG9saWN5Lmdv) | `61.29% <61.29%> (ø)` | | | [common/mapq/tree/queue\_tree.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Ftree%2Fqueue_tree.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvdHJlZS9xdWV1ZV90cmVlLmdv) | `74.64% <74.64%> (ø)` | | | [common/mapq/tree/queue\_tree\_node.go](https://app.codecov.io/gh/uber/cadence/pull/6132?src=pr&el=tree&filepath=common%2Fmapq%2Ftree%2Fqueue_tree_node.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-Y29tbW9uL21hcHEvdHJlZS9xdWV1ZV90cmVlX25vZGUuZ28=) | `76.31% <76.31%> (ø)` | | ... and [23 files with indirect coverage changes](https://app.codecov.io/gh/uber/cadence/pull/6132/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/uber/cadence/pull/6132?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/uber/cadence/pull/6132?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber). Last update [83ebf7a...2643b58](https://app.codecov.io/gh/uber/cadence/pull/6132?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber).
coveralls commented 2 weeks ago

Pull Request Test Coverage Report for Build 01900ef3-4f9c-48cc-b38e-39c97c85ad36

Details

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/dispatcher/dispatcher.go 24 29 82.76%
common/mapq/client_impl.go 18 30 60.0%
common/mapq/types/policy.go 29 44 65.91%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
<!-- Total: 370 447 82.77% -->
Files with Coverage Reduction New Missed Lines %
service/matching/tasklist/task_list_manager.go 2 77.05%
service/history/replication/task_processor.go 2 82.76%
service/history/task/transfer_active_task_executor.go 2 72.77%
common/persistence/visibility_single_manager.go 2 99.36%
service/matching/tasklist/matcher.go 2 89.35%
service/matching/tasklist/task_reader.go 2 77.72%
common/persistence/historyManager.go 2 66.67%
common/persistence/nosql/nosql_task_store.go 3 85.52%
service/history/task/fetcher.go 4 86.08%
common/asyncworkflow/queue/consumer/default_consumer.go 6 93.82%
<!-- Total: 754 -->
Totals Coverage Status
Change from base Build 01900d42-a47b-496e-814a-80e3cc64dff7: 0.03%
Covered Lines: 106902
Relevant Lines: 149540
💛 - Coveralls
coveralls commented 2 weeks ago

Pull Request Test Coverage Report for Build 019012ba-cdeb-477a-b161-0bed5e192bb7

Details

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/dispatcher/dispatcher.go 24 29 82.76%
common/mapq/client_impl.go 18 30 60.0%
common/mapq/types/policy.go 29 44 65.91%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
<!-- Total: 370 447 82.77% -->
Files with Coverage Reduction New Missed Lines %
common/task/weighted_round_robin_task_scheduler.go 2 89.05%
service/matching/tasklist/task_list_manager.go 2 77.05%
common/task/fifo_task_scheduler.go 2 87.63%
tools/cli/admin_db_decode_thrift.go 2 70.51%
service/history/replication/task_processor.go 2 82.76%
service/history/task/transfer_active_task_executor.go 2 72.77%
common/persistence/visibility_single_manager.go 2 99.36%
common/persistence/historyManager.go 2 66.67%
service/history/task/task.go 3 84.81%
common/persistence/statsComputer.go 3 98.21%
<!-- Total: 772 -->
Totals Coverage Status
Change from base Build 01900d42-a47b-496e-814a-80e3cc64dff7: 0.02%
Covered Lines: 106888
Relevant Lines: 149540
💛 - Coveralls
coveralls commented 2 weeks ago

Pull Request Test Coverage Report for Build 0190141d-2c12-4d43-be38-0107cc2cf921

Details

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/dispatcher/dispatcher.go 25 27 92.59%
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/client_impl.go 24 30 80.0%
common/mapq/types/policy.go 29 41 70.73%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
<!-- Total: 377 442 85.29% -->
Files with Coverage Reduction New Missed Lines %
service/history/shard/context.go 2 78.36%
service/matching/tasklist/db.go 2 73.23%
common/peerprovider/ringpopprovider/config.go 2 81.58%
service/matching/tasklist/task_list_manager.go 2 77.05%
service/history/replication/task_processor.go 2 82.76%
common/persistence/visibility_single_manager.go 2 99.36%
service/matching/tasklist/matcher.go 2 89.35%
common/log/tag/tags.go 3 50.46%
common/persistence/nosql/nosql_task_store.go 3 85.52%
common/archiver/filestore/historyArchiver.go 4 80.95%
<!-- Total: 772 -->
Totals Coverage Status
Change from base Build 01900d42-a47b-496e-814a-80e3cc64dff7: 0.02%
Covered Lines: 106875
Relevant Lines: 149535
💛 - Coveralls
coveralls commented 6 days ago

Pull Request Test Coverage Report for Build 01904b76-7f40-4850-b643-9077d580a9ce

Details

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/mapq/dispatcher/dispatcher.go 25 27 92.59%
common/mapq/mapq.go 41 43 95.35%
common/mapq/types/item.go 33 36 91.67%
common/mapq/types/policy_collection.go 69 72 95.83%
common/mapq/client_impl.go 24 30 80.0%
common/mapq/types/policy.go 29 41 70.73%
common/mapq/tree/queue_tree.go 71 89 79.78%
common/mapq/tree/queue_tree_node.go 85 104 81.73%
<!-- Total: 377 442 85.29% -->
Files with Coverage Reduction New Missed Lines %
common/task/weighted_round_robin_task_scheduler.go 2 89.05%
common/task/fifo_task_scheduler.go 2 84.54%
service/matching/tasklist/matcher.go 2 90.91%
service/matching/tasklist/task_reader.go 2 75.33%
service/history/handler/handler.go 3 96.2%
service/history/queue/timer_queue_processor_base.go 3 77.87%
common/persistence/statsComputer.go 3 98.21%
common/archiver/filestore/historyArchiver.go 4 80.95%
service/history/task/transfer_standby_task_executor.go 6 87.35%
service/frontend/api/handler.go 6 75.62%
<!-- Total: 52 -->
Totals Coverage Status
Change from base Build 01903cd7-c1ac-49f3-a7a4-fe9da6c16ce7: 0.01%
Covered Lines: 107092
Relevant Lines: 149680
💛 - Coveralls
taylanisikdemir commented 6 days ago

Merging to unblock splitting of development. Feel free to add comments. Everything in this Pr is subject to change in follow up iterations.