Add log and metrics to workflow termination events

[fimanishi]

What changed? Added logs to workflow termination events, containing the reason for the termination, tagged with the domainName, workflowID, terminationReason, and runID. Added metrics to workflow termination events, using a counter per domain WorkflowTerminateCounterPerDomain under the HistoryTerminateWorkflowExecutionScope scope.

Why? Improve workflow termination visibility, allowing Cadence and clients to easily find terminated workflows. This is particularly important to provide better information for workflows terminated during failovers.

How did you test it? Unit tests.

Potential risks The risks are associated with the changes in functions parameters being passed. Need to ensure that the parameters are correct and that they do not contain nil values.

Release notes

Documentation Changes

[coveralls]

Pull Request Test Coverage Report for Build 01904dd0-8aa3-4316-a934-ce3147335fd0

Details

• 63 of 66 (95.45%) changed or added relevant lines in 12 files are covered.
• 39 unchanged lines in 11 files lost coverage.
• Overall coverage decreased (-0.04%) to 71.512%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
service/history/task/transfer_active_task_executor.go	0	3	0.0%
<!--	Total:	63	66	95.45%	-->

Files with Coverage Reduction	New Missed Lines	%
common/task/weighted_round_robin_task_scheduler.go	2	88.06%
common/peerprovider/ringpopprovider/config.go	2	81.58%
common/task/fifo_task_scheduler.go	2	83.51%
common/util.go	2	91.84%
service/history/queue/transfer_queue_processor.go	2	56.49%
common/persistence/nosql/nosql_task_store.go	3	85.52%
service/history/task/transfer_active_task_executor.go	3	72.63%
common/persistence/statsComputer.go	3	98.21%
service/history/task/transfer_standby_task_executor.go	6	86.94%
service/history/task/fetcher.go	6	85.05%
<!--	Total:	39		-->

Totals
Change from base Build 01904bcc-3b56-4d05-8b48-d9bec7a9b7ea:	-0.04%
Covered Lines:	106766
Relevant Lines:	149299

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 019050ae-cea2-4903-9de4-b1001335191c

Details

• 65 of 68 (95.59%) changed or added relevant lines in 12 files are covered.
• 54 unchanged lines in 14 files lost coverage.
• Overall coverage decreased (-0.03%) to 71.509%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
service/history/task/transfer_active_task_executor.go	0	3	0.0%
<!--	Total:	65	68	95.59%	-->

Files with Coverage Reduction	New Missed Lines	%
service/history/task/transfer_active_task_executor.go	1	72.76%
common/task/weighted_round_robin_task_scheduler.go	2	89.05%
service/matching/tasklist/db.go	2	73.23%
common/task/fifo_task_scheduler.go	2	85.57%
common/util.go	2	91.84%
service/history/task/task.go	3	84.81%
common/log/tag/tags.go	3	50.46%
common/persistence/nosql/nosql_task_store.go	3	85.52%
service/history/task/timer_standby_task_executor.go	3	85.63%
service/history/task/fetcher.go	3	86.6%
<!--	Total:	54		-->

Totals
Change from base Build 0190508e-3f7e-4c0d-b546-4eaa2a2045f1:	-0.03%
Covered Lines:	106763
Relevant Lines:	149301

---

💛 - Coveralls

[coveralls]

Pull Request Test Coverage Report for Build 019060fb-a2f5-46ac-a87c-29c4b9ea3cef

Details

• 64 of 67 (95.52%) changed or added relevant lines in 13 files are covered.
• 2406 unchanged lines in 53 files lost coverage.
• Overall coverage decreased (-0.02%) to 71.507%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
service/history/task/transfer_active_task_executor.go	0	3	0.0%
<!--	Total:	64	67	95.52%	-->

Files with Coverage Reduction	New Missed Lines	%
common/task/weighted_round_robin_task_scheduler.go	1	89.05%
common/persistence/sql/sqlplugin/postgres/task.go	2	73.4%
service/matching/tasklist/db.go	2	73.23%
common/dynamicconfig/constants.go	2	99.05%
service/matching/tasklist/task_list_manager.go	2	76.65%
common/dynamicconfig/config.go	2	98.92%
common/util.go	2	91.84%
common/constants.go	2	0.0%
service/history/task/task.go	3	84.81%
common/log/tag/tags.go	3	50.73%
<!--	Total:	2406		-->

Totals
Change from base Build 019060e3-e556-4061-8a61-c080801be36f:	-0.02%
Covered Lines:	106760
Relevant Lines:	149300

---

💛 - Coveralls