Remove internal retry from AddTask

[Shaddoll]

What changed?

• Remove retry logic from AddTask
• Increase the wait time for sync match
• Add wait time for local sync match
• Add wait time for writing task to database if sync match fails

Why?

• The retry logic is not optimal for AddTask function. It spends a short time on sync match without waiting for a poller and then another short time waiting for the tasks to be written to database, and it retries if these 2 steps fails. But if the retry succeeds on sync match, it means time is wasted on waiting for tasks to be written to database. Thus, we should allocate some time for sync match first and then some time for writing tasks to database.
• 200 milliseconds is too short for sync match if the task needs to be forwarded for a remote sync match
• add wait time for local sync match to increase hit rate because this is the cheapest way to match a task

How did you test it? unit tests, some tests are adjust to run faster because AddTask takes longer due to extra wait time for local sync match

Potential risks AddActivityTask/AddDecisionTask availability may drop, and task matching latency can increase. But we can rollback the change if that happens.

Release notes

Documentation Changes

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 65.21739% with 8 lines in your changes missing coverage. Please review.

Project coverage is 72.87%. Comparing base (f3350d0) to head (8590c9c). Report is 2 commits behind head on master.

Additional details and impacted files

| [Files](https://app.codecov.io/gh/uber/cadence/pull/6195?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber) | Coverage Δ | | |---|---|---| | [service/matching/tasklist/matcher.go](https://app.codecov.io/gh/uber/cadence/pull/6195?src=pr&el=tree&filepath=service%2Fmatching%2Ftasklist%2Fmatcher.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-c2VydmljZS9tYXRjaGluZy90YXNrbGlzdC9tYXRjaGVyLmdv) | `82.53% <100.00%> (+0.15%)` | :arrow_up: | | [service/matching/tasklist/task\_reader.go](https://app.codecov.io/gh/uber/cadence/pull/6195?src=pr&el=tree&filepath=service%2Fmatching%2Ftasklist%2Ftask_reader.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-c2VydmljZS9tYXRjaGluZy90YXNrbGlzdC90YXNrX3JlYWRlci5nbw==) | `73.88% <0.00%> (ø)` | | | [service/matching/tasklist/task\_writer.go](https://app.codecov.io/gh/uber/cadence/pull/6195?src=pr&el=tree&filepath=service%2Fmatching%2Ftasklist%2Ftask_writer.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-c2VydmljZS9tYXRjaGluZy90YXNrbGlzdC90YXNrX3dyaXRlci5nbw==) | `77.44% <33.33%> (ø)` | | | [service/matching/tasklist/task\_list\_manager.go](https://app.codecov.io/gh/uber/cadence/pull/6195?src=pr&el=tree&filepath=service%2Fmatching%2Ftasklist%2Ftask_list_manager.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber#diff-c2VydmljZS9tYXRjaGluZy90YXNrbGlzdC90YXNrX2xpc3RfbWFuYWdlci5nbw==) | `64.62% <68.75%> (+0.55%)` | :arrow_up: | ... and [6 files with indirect coverage changes](https://app.codecov.io/gh/uber/cadence/pull/6195/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/uber/cadence/pull/6195?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/uber/cadence/pull/6195?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber). Last update [f3350d0...8590c9c](https://app.codecov.io/gh/uber/cadence/pull/6195?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=uber).