Concurrent map writes of search attributes due to potential racing condition

more repro:

/usr/bin/make -f /Users/qlong/indeed/iwf/iwf-server/Makefile -C /Users/qlong/indeed/iwf/iwf-server integTests
go test -v ./integ
=== RUN   TestAnyCommandCloseWorkflowTemporal
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /api/v1/workflowState/start --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateStart-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflowState/decide --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateDecide-fm (3 handlers)
2022/12/29 09:22:27 INFO  No logger configured for temporal client. Created default one.
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /                         --> github.com/indeedeng/iwf/service/api.(*handler).index-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/start    --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStart-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/signal   --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSignal-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/dataobjects/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetDataObjects-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/searchattributes/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetSearchAttributes-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/get      --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGet-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/getWithWait --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetWithWait-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/search   --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSearch-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/reset    --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowReset-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/stop     --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStop-fm (3 handlers)
2022/12/29 09:22:27 INFO  Started Worker Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@
2022/12/29 09:22:27 received API request {any_command_close1672334547 any_command_close 10 http://localhost:9714 S1 <nil> <nil> <nil>}
2022/12/29 09:22:27 Started workflow WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8
[GIN] 2022/12/29 - 09:22:27 | 200 |    8.913833ms |             ::1 | POST     "/api/v1/workflow/start"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> test-signal-name2 0x140003aa410}
[GIN] 2022/12/29 - 09:22:27 | 200 |    7.184583ms |             ::1 | POST     "/api/v1/workflow/signal"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> <nil> <nil>}
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 11 ActivityType StateStart
2022/12/29 09:22:27 INFO  StateStartActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 11 ActivityType StateStart Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x1400042c920 [] []}}
2022/12/29 09:22:27 received state start request,  {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x140003ab1d0 [] []}
[GIN] 2022/12/29 - 09:22:27 | 200 |         155µs |             ::1 | POST     "/api/v1/workflowState/start"
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 17 ActivityType StateDecide
2022/12/29 09:22:27 INFO  StateDecideActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 17 ActivityType StateDecide Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x140008d07a0 [] [] [] 0x1400041db80}}
2022/12/29 09:22:27 received state decide request,  {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S1-1} any_command_close S1 0x1400042d3d0 [] [] [] 0x140004f53b0}
[GIN] 2022/12/29 - 09:22:27 | 200 |      90.708µs |             ::1 | POST     "/api/v1/workflowState/decide"
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 25 ActivityType StateStart
2022/12/29 09:22:27 INFO  StateStartActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 25 ActivityType StateStart Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] []}}
2022/12/29 09:22:27 received state start request,  {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] []}
[GIN] 2022/12/29 - 09:22:27 | 200 |      43.208µs |             ::1 | POST     "/api/v1/workflowState/start"
2022/12/29 09:22:27 DEBUG ExecuteActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 Attempt 1 ActivityID 31 ActivityType StateDecide
2022/12/29 09:22:27 INFO  StateDecideActivity Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@ ActivityID 31 ActivityType StateDecide Attempt 1 WorkflowType Interpreter WorkflowID any_command_close1672334547 RunID 3c9b39a7-4779-4ddd-839e-bf0477debca8 input {http://localhost:9714 {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] [] [] 0x1400041df40}}
2022/12/29 09:22:27 received state decide request,  {{any_command_close1672334547 3c9b39a7-4779-4ddd-839e-bf0477debca8 1672334547 S2-1} any_command_close S2 <nil> [] [] [] 0x14000136f50}
[GIN] 2022/12/29 - 09:22:27 | 200 |      69.208µs |             ::1 | POST     "/api/v1/workflowState/decide"
[GIN] 2022/12/29 - 09:22:27 | 200 |  103.351375ms |             ::1 | POST     "/api/v1/workflow/getWithWait"
2022/12/29 09:22:27 INFO  Stopped Worker Namespace default TaskQueue Interpreter_DEFAULT WorkerID 6745@IT-USA-VX8131@
--- PASS: TestAnyCommandCloseWorkflowTemporal (0.21s)
=== RUN   TestAnyCommandCloseWorkflowCadence
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /api/v1/workflowState/start --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateStart-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflowState/decide --> github.com/indeedeng/iwf/integ/workflow/common.WorkflowHandler.ApiV1WorkflowStateDecide-fm (3 handlers)
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /                         --> github.com/indeedeng/iwf/service/api.(*handler).index-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/start    --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStart-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/signal   --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSignal-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/dataobjects/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetDataObjects-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/searchattributes/get --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetSearchAttributes-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/get      --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGet-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/getWithWait --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetWithWait-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/search   --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowSearch-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/reset    --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowReset-fm (3 handlers)
[GIN-debug] POST   /api/v1/workflow/stop     --> github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowStop-fm (3 handlers)
{"level":"info","ts":"2022-12-29T09:22:27.363-0800","caller":"internal/internal_worker.go:223","msg":"No logger configured for cadence worker. Created default one."}
{"level":"info","ts":"2022-12-29T09:22:27.363-0800","caller":"internal/internal_worker.go:227","msg":"No metrics scope configured for cadence worker. Use NoopScope as default."}
{"level":"info","ts":"2022-12-29T09:22:27.434-0800","caller":"internal/internal_worker.go:834","msg":"Started Workflow Worker","Domain":"default","TaskList":"Interpreter_DEFAULT","WorkerID":"6745@IT-USA-VX8131@Interpreter_DEFAULT@2f8d29d1-e39a-433e-a7a2-f00411651f29"}
{"level":"info","ts":"2022-12-29T09:22:27.448-0800","caller":"internal/internal_worker.go:859","msg":"Started Activity Worker","Domain":"default","TaskList":"Interpreter_DEFAULT","WorkerID":"6745@IT-USA-VX8131@Interpreter_DEFAULT@2f8d29d1-e39a-433e-a7a2-f00411651f29"}
2022/12/29 09:22:27 received API request {any_command_close1672334547 any_command_close 10 http://localhost:9714 S1 <nil> <nil> <nil>}
2022/12/29 09:22:27 Started workflow WorkflowID any_command_close1672334547 RunID eaf1acf2-9109-4d32-9e28-db1a9c40737b
[GIN] 2022/12/29 - 09:22:27 | 200 |   21.851791ms |             ::1 | POST     "/api/v1/workflow/start"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> test-signal-name2 0x140007d4f90}
[GIN] 2022/12/29 - 09:22:27 | 200 |   20.215042ms |             ::1 | POST     "/api/v1/workflow/signal"
2022/12/29 09:22:27 received API request {any_command_close1672334547 <nil> <nil> <nil>}
fatal error: concurrent map writes

goroutine 268 [running]:
runtime.throw({0x1050c9ae5?, 0x14000b8a768?})
    /usr/local/go-dists/1.18.3/go/src/runtime/panic.go:992 +0x50 fp=0x1400049ec90 sp=0x1400049ec60 pc=0x10424cbb0
runtime.mapassign_faststr(0x10542fc00?, 0x14000ce06b0?, {0x1050c7518, 0x14})
    /usr/local/go-dists/1.18.3/go/src/runtime/map_faststr.go:212 +0x3bc fp=0x1400049ed00 sp=0x1400049ec90 pc=0x10422829c
go.uber.org/cadence/internal.mergeSearchAttributes(...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:354
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).updateWorkflowInfoWithSearchAttributes(...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:339
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).UpsertSearchAttributes(0x1400087d560, 0x1400049ee58?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:334 +0x258 fp=0x1400049edf0 sp=0x1400049ed00 pc=0x104d72998
go.uber.org/cadence/internal.(*workflowEnvironmentImpl).GetVersion(0x1400087d560, {0x1050b0fc9, 0x6}, 0x1055f75a0?, 0x1)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:623 +0x148 fp=0x1400049efa0 sp=0x1400049edf0 pc=0x104d74a38
go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).GetVersion(0x1400049f058?, {0x1053c9b00?, 0x10563eea8?}, {0x1050b0fc9?, 0x1400049f038?}, 0x104e86684?, 0x105490940?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1614 +0x40 fp=0x1400049efe0 sp=0x1400049efa0 pc=0x104db58a0
go.uber.org/cadence/internal.GetVersion({0x105656c48, 0x14000e17ae0}, {0x1050b0fc9, 0x6}, 0x78?, 0x105554a80?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:1610 +0x90 fp=0x1400049f040 sp=0x1400049efe0 pc=0x104db5800
go.uber.org/cadence/workflow.GetVersion(...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/workflow/workflow.go:344
github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GetVersion(0x1400049f0d8?, {0x1056416c8?, 0x14000e21a90?}, {0x1050b0fc9, 0x6}, 0x0?, 0x1400049f108?)
    /Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflowProvider.go:161 +0x70 fp=0x1400049f090 sp=0x1400049f040 pc=0x104e866a0
github.com/indeedeng/iwf/service/interpreter.(*globalVersioner).isAfterVersionOfUsingGlobalVersioning(...)
    /Users/qlong/indeed/iwf/iwf-server/service/interpreter/globalVersionProvider.go:21
github.com/indeedeng/iwf/service/interpreter.InterpreterImpl({0x1056416c8?, 0x14000e21a90}, {0x10565dbe8, 0x106133940}, {{0x140003bceb8, 0x11}, {0x140003bced0, 0x15}, {0x14000ce0688, 0x2}, ...})
    /Users/qlong/indeed/iwf/iwf-server/service/interpreter/workflowImpl.go:12 +0xc8 fp=0x1400049f3a0 sp=0x1400049f090 pc=0x104e80638
github.com/indeedeng/iwf/service/interpreter/cadence.Interpreter({0x105656c48?, 0x14000e17ae0}, {{0x140003bceb8, 0x11}, {0x140003bced0, 0x15}, {0x14000ce0688, 0x2}, {0x0, 0x0}, ...})
    /Users/qlong/indeed/iwf/iwf-server/service/interpreter/cadence/workflow.go:10 +0xb0 fp=0x1400049f450 sp=0x1400049f3a0 pc=0x104e85790
runtime.call256(0x140001ebcb0, 0x10563b0b8, 0x14000cd9100, 0x78, 0x78, 0x88, 0x1400049fa80)
    /usr/local/go-dists/1.18.3/go/src/runtime/asm_arm64.s:515 +0x84 fp=0x1400049f560 sp=0x1400049f450 pc=0x10427ea34
runtime.reflectcall(0x105554a80?, 0x14000cd9000?, 0x4?, 0x50c5041?, 0x1?, 0x12?, 0x105554a80?)
    <autogenerated>:1 +0x34 fp=0x1400049f5a0 sp=0x1400049f560 pc=0x104282394
reflect.Value.call({0x1054210e0?, 0x10563b0b8?, 0x1400049fc88?}, {0x1050a7a2d, 0x4}, {0x14000bc9680, 0x2, 0x104d686dc?})
    /usr/local/go-dists/1.18.3/go/src/reflect/value.go:556 +0x5e4 fp=0x1400049fc10 sp=0x1400049f5a0 pc=0x1042e15c4
reflect.Value.Call({0x1054210e0?, 0x10563b0b8?, 0x1400049fcd8?}, {0x14000bc9680, 0x2, 0x2})
    /usr/local/go-dists/1.18.3/go/src/reflect/value.go:339 +0x98 fp=0x1400049fc90 sp=0x1400049fc10 pc=0x1042e0dd8
go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0x14000bc92c0?, {0x105656c48?, 0x14000e17ae0?}, {0x14000e21a50?, 0x14000d2e120?}, {0x14000e21a70?, 0x1?, 0x14000003c28?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/workflow.go:681 +0x200 fp=0x1400049fdd0 sp=0x1400049fc90 pc=0x104db0330
go.uber.org/cadence/internal.(*workflowExecutor).Execute(0x14000ca89e0, {0x105656c48, 0x14000e17ae0}, {0x14000d2e120, 0x85, 0x90})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker.go:700 +0x3b8 fp=0x1400049feb0 sp=0x1400049fdd0 pc=0x104d96168
go.uber.org/cadence/internal.(*syncWorkflowDefinition).Execute.func1({0x105656e40, 0x14000bc9590})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:467 +0xe4 fp=0x1400049ff20 sp=0x1400049feb0 pc=0x104d9ce04
go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0x14000cc1060?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:884 +0xc4 fp=0x1400049ffb0 sp=0x1400049ff20 pc=0x104d9f214
go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func2()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:885 +0x34 fp=0x1400049ffd0 sp=0x1400049ffb0 pc=0x104d9f114
runtime.goexit()
    /usr/local/go-dists/1.18.3/go/src/runtime/asm_arm64.s:1263 +0x4 fp=0x1400049ffd0 sp=0x1400049ffd0 pc=0x104280884
created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:875 +0x314

goroutine 1 [chan receive]:
testing.(*T).Run(0x14000582820, {0x1050e1465?, 0x9b7f7eca3f5f?}, 0x10563b028)
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1487 +0x314
testing.runTests.func1(0x0?)
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1839 +0x74
testing.tRunner(0x14000582820, 0x140004fbcb8)
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1439 +0x110
testing.runTests(0x14000147f40?, {0x1060ed340, 0x10, 0x10}, {0xfa00000000000000?, 0x104222190?, 0x106102da0?})
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1837 +0x3f0
testing.(*M).Run(0x14000147f40)
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1719 +0x500
main.main()
    _testmain.go:77 +0x1d0

goroutine 59 [select]:
net/http.(*persistConn).roundTrip(0x140007dd680, 0x1400095b340)
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:2620 +0x810
net/http.(*Transport).roundTrip(0x1060ece20, 0x14000257c00)
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:594 +0x6bc
net/http.(*Transport).RoundTrip(0x14000c82b00?, 0x105644c48?)
    /usr/local/go-dists/1.18.3/go/src/net/http/roundtrip.go:17 +0x20
net/http.send(0x14000257c00, {0x105644c48, 0x1060ece20}, {0x1044e4d10?, 0x8?, 0x0?})
    /usr/local/go-dists/1.18.3/go/src/net/http/client.go:252 +0x4fc
net/http.(*Client).send(0x1061023e0, 0x14000257c00, {0x140005b1428?, 0x104221d20?, 0x0?})
    /usr/local/go-dists/1.18.3/go/src/net/http/client.go:176 +0x9c
net/http.(*Client).do(0x1061023e0, 0x14000257c00)
    /usr/local/go-dists/1.18.3/go/src/net/http/client.go:725 +0x6ec
net/http.(*Client).Do(...)
    /usr/local/go-dists/1.18.3/go/src/net/http/client.go:593
github.com/indeedeng/iwf/gen/iwfidl.(*APIClient).callAPI(0x1400016b140, 0x105655208?)
    /Users/qlong/indeed/iwf/iwf-server/gen/iwfidl/client.go:171 +0x9c
github.com/indeedeng/iwf/gen/iwfidl.(*DefaultApiService).ApiV1WorkflowGetWithWaitPostExecute(0x1400016b148, {{0x105655208, 0x14000128008}, {0x10565eca8, 0x1400016b148}, 0x14000961170})
    /Users/qlong/indeed/iwf/iwf-server/gen/iwfidl/api_default.go:459 +0x314
github.com/indeedeng/iwf/gen/iwfidl.ApiApiV1WorkflowGetWithWaitPostRequest.Execute(...)
    /Users/qlong/indeed/iwf/iwf-server/gen/iwfidl/api_default.go:398
github.com/indeedeng/iwf/integ.doTestAnyCommandCloseWorkflow(0x1400040bba0, {0x1050b30c0, 0x7})
    /Users/qlong/indeed/iwf/iwf-server/integ/any_command_close_test.go:79 +0x6d0
github.com/indeedeng/iwf/integ.TestAnyCommandCloseWorkflowCadence(0x1400040bba0)
    /Users/qlong/indeed/iwf/iwf-server/integ/any_command_close_test.go:29 +0x68
testing.tRunner(0x1400040bba0, 0x10563b028)
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1439 +0x110
created by testing.(*T).Run
    /usr/local/go-dists/1.18.3/go/src/testing/testing.go:1486 +0x300

goroutine 94 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x14000b84900)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400050cd80, {0x10537e980?, 0x14000b88258}, 0x104915610?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x140005f9768?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x14000b846c0, 0x140005f9798, 0x140005f9768)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x14000b846c0, {0x10537e980?, 0x14000b88258?})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x140001eb230?}, {0x14000044f40?, 0x140007d54b0?}, {0x1053a8bc0, 0x14000b882a0}, {0x10537e980, 0x14000b88258}, 0x0?, {0x140007e51c0?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x140001eb230?}, {0x14000044f40?, 0x3c?}, {0x1053a8bc0?, 0x14000b882a0?}, {0x10537e980?, 0x14000b88258?}, {0x140007d54b0?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x140007d9560}, 0x1400050ccc0, 0x1050b5b05?, 0x14000124c00, {0x1?, 0x10426c2c4?, 0x106102da0?})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x140007d9560}, 0x1400050ccc0)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x140007d9560?}, 0x1050a82fd?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x140007d9560}, 0x1400050ccc0, {0x105658fb0, 0x140001eb110})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d9560}, 0x104759b78?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x140007d9560}, 0x1400050ccc0, {0x105658fb0?, 0x140001eb0e0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d9560}, 0x140001eb0b0?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x140007d9560}, 0x140003a8600?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x140008514a0, {0x105655240?, 0x140007d9560?}, {0x1050d348a?, 0x1010140005faee8?}, {0x10564d008?, 0x140002caa10?}, 0x10563b460, {0x140007d95c0?, 0x7?, ...})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkflowAPIYARPCCaller).GetWorkflowExecutionHistory(0x140007e6c80?, {0x105655240?, 0x140007d9560?}, 0x10426c94c?, {0x140007d95c0?, 0x104222190?, 0x140001eb020?})
    /Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_workflow.pb.yarpc.go:505 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.GetWorkflowExecutionHistory({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x140007d9560}, ...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:86 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).GetWorkflowExecutionHistory(0x140008516e0, {0x105655240, 0x140007d9560}, 0x140005fb0e8?, {0x140007d95c0, 0x4, 0x6})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:163 +0x80
go.uber.org/cadence/internal.(*workflowClient).GetWorkflowHistory.func1.1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:524 +0xcc
go.uber.org/cadence/internal/common/backoff.Retry({0x1056551d0, 0x140005d0380}, 0x140005fb2c8, {0x105643488, 0x140001948d0}, 0x10563bfc0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/backoff/retry.go:98 +0x8c
go.uber.org/cadence/internal.(*workflowClient).GetWorkflowHistory.func1({0x0, 0x0, 0x0})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:515 +0x334
go.uber.org/cadence/internal.(*historyEventIteratorImpl).HasNext(0x140007d9500)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:1071 +0x8c
go.uber.org/cadence/internal.(*workflowRunImpl).Get(0x140007d9440, {0x1056551d0, 0x140005d0380}, {0x105382dc0?, 0x14000b881f8})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow_client.go:1124 +0x74
github.com/indeedeng/iwf/service/api/cadence.(*cadenceClient).GetWorkflowResult(0x0?, {0x1056551d0, 0x140005d0380}, {0x105382dc0, 0x14000b881f8}, {0x140003420a0?, 0x0?}, {0x0?, 0x0?})
    /Users/qlong/indeed/iwf/iwf-server/service/api/cadence/client.go:177 +0x70
github.com/indeedeng/iwf/service/api.(*serviceImpl).doApiV1WorkflowGetPost(0x140004711c0, {0x1056551d0, 0x140005d0380}, {{0x140003420a0, 0x1b}, 0x0, 0x0, 0x0}, 0x1)
    /Users/qlong/indeed/iwf/iwf-server/service/api/service.go:138 +0x160
github.com/indeedeng/iwf/service/api.(*serviceImpl).ApiV1WorkflowGetWithWaitPost(0x140005fb600?, {0x1056551d0?, 0x140005d0380?}, {{0x140003420a0, 0x1b}, 0x0, 0x0, 0x0})
    /Users/qlong/indeed/iwf/iwf-server/service/api/service.go:126 +0x44
github.com/indeedeng/iwf/service/api.(*handler).doApiV1WorkflowGetPost(0x1400086a340, 0x140005dc200, 0x1)
    /Users/qlong/indeed/iwf/iwf-server/service/api/handler.go:158 +0x1f8
github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowGetWithWait(0x140005fb6a8?, 0x1042294d0?)
    /Users/qlong/indeed/iwf/iwf-server/service/api/handler.go:144 +0x24
github.com/gin-gonic/gin.(*Context).Next(...)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/context.go:173
github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0x140005dc200)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/recovery.go:101 +0x80
github.com/gin-gonic/gin.(*Context).Next(...)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/context.go:173
github.com/gin-gonic/gin.LoggerWithConfig.func1(0x140005dc200)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/logger.go:240 +0xb0
github.com/gin-gonic/gin.(*Context).Next(...)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/context.go:173
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0x1400086e340, 0x140005dc200)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/gin.go:616 +0x568
github.com/gin-gonic/gin.(*Engine).ServeHTTP(0x1400086e340, {0x105654118?, 0x140005d80e0}, 0x140005dc100)
    /Users/qlong/go/pkg/mod/github.com/gin-gonic/gin@v1.8.1/gin.go:572 +0x1d8
net/http.serverHandler.ServeHTTP({0x105649508?}, {0x105654118, 0x140005d80e0}, 0x140005dc100)
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:2916 +0x3fc
net/http.(*conn).serve(0x140003fe820, {0x105655278, 0x14000a58c60})
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:1966 +0x56c
created by net/http.(*Server).Serve
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:3071 +0x450

goroutine 147 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d800, 0x72)
    /usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x1400095e280?, 0x14000960851?, 0x0)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x1400095e280, {0x14000960851, 0x1, 0x1})
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x1400095e280, {0x14000960851?, 0x14000860fa8?, 0x104d99d2c?})
    /usr/local/go-dists/1.18.3/go/src/net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x1400018aa38, {0x14000960851?, 0x100000001?, 0x104d99ef0?})
    /usr/local/go-dists/1.18.3/go/src/net/net.go:183 +0x38
net/http.(*connReader).backgroundRead(0x14000960840)
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:672 +0x48
created by net/http.(*connReader).startBackgroundRead
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:668 +0xd0

goroutine 299 [select]:
go.uber.org/cadence/internal.(*locallyDispatchedActivityTunnel).getTask(0x140004b1728)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:193 +0x60
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).pollLocallyDispatchedActivity(0x14000874080, {0x1043160f8?, 0x140008440d8?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1122 +0x2c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x14000874080, {0x105655240?, 0x14000a62f60?}, 0x14000878418?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x14000a62f60?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c

goroutine 104 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x14000874080, {0xd0?}, 0x14000799278)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0x14000874080)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1113 +0x8c
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852540)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852540)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 337 [select]:
go.uber.org/cadence/internal.(*locallyDispatchedActivityTunnel).getTask(0x140004b1728)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:193 +0x60
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).pollLocallyDispatchedActivity(0x14000874080, {0x1043160f8?, 0x14000844018?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1122 +0x2c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x14000874080, {0x105655240?, 0x1400060e360?}, 0x14000878238?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x1400060e360?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c

goroutine 119 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x14000a49c20)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x140009772c0, {0x10537e980?, 0x140007999e0}, 0x104915610?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x140004a8488?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x14000a499e0, 0x140004a84b8, 0x140004a8488)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x14000a499e0, {0x10537e980?, 0x140007999e0?})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x14000a58ff0?}, {0x1400047cc40?, 0x140008d1a40?}, {0x1053a8bc0, 0x14000799a28}, {0x10537e980, 0x140007999e0}, 0x0?, {0x14000a5a540?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x14000a58ff0?}, {0x1400047cc40?, 0x32?}, {0x1053a8bc0?, 0x14000799a28?}, {0x10537e980?, 0x140007999e0?}, {0x140008d1a40?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x14000899a40}, 0x14000977200, 0x1050b5b05?, 0x140001cc968, {0x1?, 0x18?, 0x106102da0?})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x14000899a40}, 0x14000977200)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x14000899a40?}, 0x1050a82fd?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x14000899a40}, 0x14000977200, {0x105658fb0, 0x14000a58ed0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x14000899a40}, 0x104759b78?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x14000899a40}, 0x14000977200, {0x105658fb0?, 0x14000a58ea0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x14000899a40}, 0x14000a58e70?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x14000899a40}, 0x12d552708?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x14000899a40?}, {0x1050c5f4c?, 0x14000903c08?}, {0x10564dde8?, 0x14000a54d20?}, 0x10563b3d0, {0x14000899b00?, 0x14000a58d80?, ...})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0x1400009b900?, {0x105655240?, 0x14000899a40?}, 0x10426c94c?, {0x14000899b00?, 0x104222190?, 0x14000a58d50?})
    /Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x14000899a40}, ...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:121 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0x14000851b60, {0x105655240, 0x14000899a40}, 0x4?, {0x14000899b00, 0x4, 0x6})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:212 +0x80
go.uber.org/cadence/internal.(*activityTaskPoller).poll(0x1400080de80, {0x105655240, 0x14000899a40})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:991 +0x348
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x1400080de80, {0x105655240?, 0x14000899a40?}, 0x14000055e00?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x14000899a40?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c

goroutine 102 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x140007e6370, 0x1)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/controlbuf.go:408 +0x15c
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x140007d8840)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/controlbuf.go:535 +0x78
google.golang.org/grpc/internal/transport.newHTTP2Client.func3()
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:419 +0x68
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:417 +0x16e0

goroutine 105 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x14000874080, {0xe0?}, 0x14000bfdde8)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0x14000874080)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1113 +0x8c
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852540)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852540)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 96 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x140009639e0)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400079f500, {0x10537e980?, 0x140005a9680}, 0x104915610?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x140009164e8?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x140009637a0, 0x14000916518, 0x140009164e8)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x140009637a0, {0x10537e980?, 0x140005a9680?})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x14000961b90?}, {0x14000c00440?, 0x14000c80710?}, {0x1053a8bc0, 0x140005a96c8}, {0x10537e980, 0x140005a9680}, 0x0?, {0x1400095b780?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x14000961b90?}, {0x14000c00440?, 0x32?}, {0x1053a8bc0?, 0x140005a96c8?}, {0x10537e980?, 0x140005a9680?}, {0x14000c80710?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x1400095ccc0}, 0x1400079f440, 0x0?, 0x1400018aaf0, {0x14000864348?, 0x18?, 0x106102da0?})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x1400095ccc0}, 0x1400079f440)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x1400095ccc0?}, 0x1050a82fd?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x1400095ccc0}, 0x1400079f440, {0x105658fb0, 0x14000961aa0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095ccc0}, 0x104759b78?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x1400095ccc0}, 0x1400079f440, {0x105658fb0?, 0x14000961a70})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095ccc0}, 0x14000961a40?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x1400095ccc0}, 0x0?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x1400095ccc0?}, {0x1050c5f5f?, 0x14000682d58?}, {0x10564de48?, 0x1400095cde0?}, 0x10563b3d8, {0x1400095cd80?, 0x13?, ...})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForDecisionTask(0x1400018fce0?, {0x105655240?, 0x1400095ccc0?}, 0x10426c94c?, {0x1400095cd80?, 0x104222190?, 0x140009619b0?})
    /Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:361 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForDecisionTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x1400095ccc0}, ...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:126 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForDecisionTask(0x14000851b60, {0x105655240, 0x1400095ccc0}, 0x0?, {0x1400095cd80, 0x4, 0x6})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:219 +0x80
go.uber.org/cadence/internal.(*workflowTaskPoller).poll(0x140008209c0, {0x105655240, 0x1400095ccc0})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:767 +0x20c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c

goroutine 101 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d710, 0x72)
    /usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x14000a56580?, 0x140007e8000?, 0x0)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x14000a56580, {0x140007e8000, 0x8000, 0x8000})
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x14000a56580, {0x140007e8000?, 0x1400008ebf8?, 0x1046b74c8?})
    /usr/local/go-dists/1.18.3/go/src/net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x14000124780, {0x140007e8000?, 0x1400008ec58?, 0x10426c2c4?})
    /usr/local/go-dists/1.18.3/go/src/net/net.go:183 +0x38
bufio.(*Reader).Read(0x140007d87e0, {0x140007ba200, 0x9, 0x14000bc8900?})
    /usr/local/go-dists/1.18.3/go/src/bufio/bufio.go:236 +0x1e4
io.ReadAtLeast({0x105641168, 0x140007d87e0}, {0x140007ba200, 0x9, 0x9}, 0x9)
    /usr/local/go-dists/1.18.3/go/src/io/io.go:331 +0xa4
io.ReadFull(...)
    /usr/local/go-dists/1.18.3/go/src/io/io.go:350
golang.org/x/net/http2.readFrameHeader({0x140007ba200?, 0x9?, 0x1400008ed58?}, {0x105641168?, 0x140007d87e0?})
    /Users/qlong/go/pkg/mod/golang.org/x/net@v0.1.0/http2/frame.go:237 +0x5c
golang.org/x/net/http2.(*Framer).ReadFrame(0x140007ba1c0)
    /Users/qlong/go/pkg/mod/golang.org/x/net@v0.1.0/http2/frame.go:498 +0x84
google.golang.org/grpc/internal/transport.(*http2Client).reader(0x14000947860)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:1516 +0x2e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/http2_client.go:369 +0x1204

goroutine 163 [chan receive]:
go.uber.org/cadence/internal.(*coroutineState).call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:839
go.uber.org/cadence/internal.(*dispatcherImpl).ExecuteUntilAllBlocked(0x140007e4f00)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:923 +0x1fc
go.uber.org/cadence/internal.executeDispatcher({0x105656c48, 0x14000e17ae0}, {0x105656cf0, 0x140007e4f00})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:540 +0x7c
go.uber.org/cadence/internal.(*syncWorkflowDefinition).OnDecisionTaskStarted(0x1400087d560?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_workflow.go:514 +0x34
go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0x14000e0ae70, 0x14000d32180, 0xd8?, 0x1)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_event_handlers.go:820 +0x7a0
go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0x1400050ce40, 0x14000bc8ea0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_handlers.go:895 +0x15ec
go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0x14000816b00, 0x14000bc8ea0, 0x14000bc8ed0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_handlers.go:787 +0x320
go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask(0x140008209c0, 0x140009618c0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:341 +0x260
go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask(0x140008209c0, {0x10538a0c0?, 0x140009618c0?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:314 +0x74
go.uber.org/cadence/internal.(*baseWorker).processTask(0x14000852620, {0x105389b80?, 0x14000c805a0?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:324 +0xe0
created by go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:253 +0x98

goroutine 112 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d260, 0x72)
    /usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x140001ec080?, 0x140007ab000?, 0x0)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0x140001ec080, {0x140007ab000, 0x1000, 0x1000})
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:167 +0x1e4
net.(*netFD).Read(0x140001ec080, {0x140007ab000?, 0x14000596c58?, 0x10421a668?})
    /usr/local/go-dists/1.18.3/go/src/net/fd_posix.go:55 +0x2c
net.(*conn).Read(0x14000124bb8, {0x140007ab000?, 0x0?, 0x0?})
    /usr/local/go-dists/1.18.3/go/src/net/net.go:183 +0x38
net/http.(*persistConn).Read(0x140007dd680, {0x140007ab000?, 0x10421a710?, 0x14000956960?})
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:1929 +0x54
bufio.(*Reader).fill(0x140007d8fc0)
    /usr/local/go-dists/1.18.3/go/src/bufio/bufio.go:106 +0x100
bufio.(*Reader).Peek(0x140007d8fc0, 0x1)
    /usr/local/go-dists/1.18.3/go/src/bufio/bufio.go:144 +0x64
net/http.(*persistConn).readLoop(0x140007dd680)
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:2093 +0x148
created by net/http.(*Transport).dialConn
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:1750 +0x11f4

goroutine 113 [select]:
net/http.(*persistConn).writeLoop(0x140007dd680)
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:2392 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go-dists/1.18.3/go/src/net/http/transport.go:1751 +0x1240

goroutine 92 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x14000962120)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400079f080, {0x10537e980?, 0x140005a9020}, 0x104915610?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x1400076e4e8?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x1400090be60, 0x1400076e518, 0x1400076e4e8)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x1400090be60, {0x10537e980?, 0x140005a9020?})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x14000960210?}, {0x14000122740?, 0x1400042db50?}, {0x1053a8bc0, 0x140005a9068}, {0x10537e980, 0x140005a9020}, 0x0?, {0x1400095ab40?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x14000960210?}, {0x14000122740?, 0x32?}, {0x1053a8bc0?, 0x140005a9068?}, {0x10537e980?, 0x140005a9020?}, {0x1400042db50?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x1400095c3c0}, 0x1400079efc0, 0x1050b5b05?, 0x1400018aa08, {0x1?, 0x18?, 0x106102da0?})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x1400095c3c0}, 0x1400079efc0)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x1400095c3c0?}, 0x1050a82fd?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x1400095c3c0}, 0x1400079efc0, {0x105658fb0, 0x140009600f0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095c3c0}, 0x104759b78?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x1400095c3c0}, 0x1400079efc0, {0x105658fb0?, 0x140009600c0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x1400095c3c0}, 0x14000960090?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x1400095c3c0}, 0x1?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x1400095c3c0?}, {0x1050c5f5f?, 0x1400008bc68?}, {0x10564de48?, 0x1400095c4e0?}, 0x10563b3d8, {0x1400095c480?, 0x32?, ...})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForDecisionTask(0x1400018f860?, {0x105655240?, 0x1400095c3c0?}, 0x10426c94c?, {0x1400095c480?, 0x104222190?, 0x14000919fb0?})
    /Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:361 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForDecisionTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x1400095c3c0}, ...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:126 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForDecisionTask(0x14000851b60, {0x105655240, 0x1400095c3c0}, 0x1042b5fbc?, {0x1400095c480, 0x4, 0x6})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:219 +0x80
go.uber.org/cadence/internal.(*workflowTaskPoller).poll(0x140008209c0, {0x105655240, 0x1400095c3c0})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:767 +0x20c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c

goroutine 106 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852540)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130

goroutine 60 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d350, 0x72)
    /usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x140000b3d80?, 0x105512080?, 0x0)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0x140000b3d80)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x140000b3d80)
    /usr/local/go-dists/1.18.3/go/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x140001c9ae8)
    /usr/local/go-dists/1.18.3/go/src/net/tcpsock_posix.go:139 +0x2c
net.(*TCPListener).Accept(0x140001c9ae8)
    /usr/local/go-dists/1.18.3/go/src/net/tcpsock.go:288 +0x30
net/http.(*Server).Serve(0x140008521c0, {0x105653f08, 0x140001c9ae8})
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:3039 +0x318
net/http.(*Server).ListenAndServe(0x140008521c0)
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:2968 +0x88
github.com/indeedeng/iwf/integ.startWorkflowWorker.func1()
    /Users/qlong/indeed/iwf/iwf-server/integ/util.go:42 +0x24
created by github.com/indeedeng/iwf/integ.startWorkflowWorker
    /Users/qlong/indeed/iwf/iwf-server/integ/util.go:41 +0x20c

goroutine 132 [IO wait]:
internal/poll.runtime_pollWait(0x12d53d530, 0x72)
    /usr/local/go-dists/1.18.3/go/src/runtime/netpoll.go:302 +0xa4
internal/poll.(*pollDesc).wait(0x14000a56600?, 0x4?, 0x0)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:83 +0x2c
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0x14000a56600)
    /usr/local/go-dists/1.18.3/go/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x14000a56600)
    /usr/local/go-dists/1.18.3/go/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000799950)
    /usr/local/go-dists/1.18.3/go/src/net/tcpsock_posix.go:139 +0x2c
net.(*TCPListener).Accept(0x14000799950)
    /usr/local/go-dists/1.18.3/go/src/net/tcpsock.go:288 +0x30
net/http.(*Server).Serve(0x14000852380, {0x105653f08, 0x14000799950})
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:3039 +0x318
net/http.(*Server).ListenAndServe(0x14000852380)
    /usr/local/go-dists/1.18.3/go/src/net/http/server.go:2968 +0x88
github.com/indeedeng/iwf/integ.startIwfService.func3()
    /Users/qlong/indeed/iwf/iwf-server/integ/util.go:84 +0x24
created by github.com/indeedeng/iwf/integ.startIwfService
    /Users/qlong/indeed/iwf/iwf-server/integ/util.go:83 +0x4ec

goroutine 65 [select]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0x140003cd600)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/balancer_conn_wrappers.go:112 +0x68
created by google.golang.org/grpc.newCCBalancerWrapper
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/balancer_conn_wrappers.go:73 +0x230

goroutine 103 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0x140007dd440)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:324 +0x70
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/internal/transport/transport.go:339
google.golang.org/grpc.(*csAttempt).recvMsg(0x1400050c9c0, {0x10537e980?, 0x140001c9bc0}, 0x104915610?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:1011 +0x7c
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x14000a52488?)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:865 +0x2c
google.golang.org/grpc.(*clientStream).withRetry(0x140007dd200, 0x14000a524b8, 0x14000a52488)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:724 +0x1c8
google.golang.org/grpc.(*clientStream).RecvMsg(0x140007dd200, {0x10537e980?, 0x140001c9bc0?})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/stream.go:864 +0xbc
google.golang.org/grpc.invoke({0x105655278?, 0x140007df6b0?}, {0x14000044b40?, 0x140007d4b00?}, {0x1053a8bc0, 0x140001c9c08}, {0x10537e980, 0x140001c9bc0}, 0x0?, {0x140007e45c0?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:73 +0xb8
google.golang.org/grpc.(*ClientConn).Invoke(0x14000866000?, {0x105655278?, 0x140007df6b0?}, {0x14000044b40?, 0x32?}, {0x1053a8bc0?, 0x140001c9c08?}, {0x10537e980?, 0x140001c9bc0?}, {0x140007d4b00?, ...})
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/call.go:37 +0x1a8
go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0x14000850ea0, {0x105655240, 0x140007d8960}, 0x1400050c900, 0x0?, 0x14000124988, {0x14000865868?, 0x18?, 0x106102da0?})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x368
go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0x14000850ea0, {0x105655240, 0x140007d8960}, 0x1400050c900)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0xc0
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240?, 0x140007d8960?}, 0x1050a82fd?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
go.uber.org/yarpc/internal/observability.(*Middleware).Call(0x14000844a80, {0x105655240, 0x140007d8960}, 0x1400050c900, {0x105658fb0, 0x140007df5c0})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0xf8
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x0, 0x0}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d8960}, 0x104759b78?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x105580d00?, {0x105655240, 0x140007d8960}, 0x1400050c900, {0x105658fb0?, 0x140007df590})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xbc
go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0x14000470e50, 0x1, 0x1}, {0x105658530, 0x14000850ea0}}, {0x105655240, 0x140007d8960}, 0x140007df560?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0x100
go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x105658eb0?, 0x14000851020?}, {0x12d78cb18?, 0x14000851020?}}, {0x105655240, 0x140007d8960}, 0x14000907a18?)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x90
go.uber.org/yarpc/encoding/protobuf.(*client).Call(0x14000851500, {0x105655240?, 0x140007d8960?}, {0x1050c5f4c?, 0x14000682d58?}, {0x10564dde8?, 0x140007e6500?}, 0x10563b3d0, {0x140007d8a20?, 0x140007df470?, ...})
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0xb8
github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0x1400016afa0?, {0x105655240?, 0x140007d8960?}, 0x10426c94c?, {0x140007d8a20?, 0x104222190?, 0x140007df440?})
    /Users/qlong/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220505064515-7bb6b0808383/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x64
go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1056581f0, 0x1400086a260}, {0x10565d0c0, 0x1400086a280}, {0x10565d038, 0x1400086a2a0}, {0x10565ae50, 0x1400086a2c0}}, {0x105655240, 0x140007d8960}, ...)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/compatibility/adapter.go:121 +0x88
go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0x14000851b60, {0x105655240, 0x140007d8960}, 0x0?, {0x140007d8a20, 0x4, 0x6})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/common/metrics/service_wrapper.go:212 +0x80
go.uber.org/cadence/internal.(*activityTaskPoller).poll(0x1400080de80, {0x105655240, 0x140007d8960})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:991 +0x348
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0x1400080de80, {0x105655240?, 0x140007d8960?}, 0x14000055e00?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1019 +0x3c
go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetricsFunc.func1({0x105655240?, 0x140007d8960?})
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1013 +0x3c
go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:256 +0x50
created by go.uber.org/cadence/internal.(*basePoller).doPoll
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:255 +0x18c

goroutine 131 [select]:
google.golang.org/grpc.(*ClientConn).WaitForStateChange(0x14000866000, {0x1056551d0, 0x140003cd700}, 0x2)
    /Users/qlong/go/pkg/mod/google.golang.org/grpc@v1.50.1/clientconn.go:515 +0x98
go.uber.org/yarpc/transport/grpc.(*grpcPeer).monitorConnectionStatus(0x140003cd740)
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/peer.go:82 +0x84
created by go.uber.org/yarpc/transport/grpc.(*Transport).newPeer
    /Users/qlong/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/peer.go:69 +0x52c

goroutine 133 [select]:
go.uber.org/cadence/internal.(*localActivityTunnel).getTask(0x0?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:167 +0x5c
go.uber.org/cadence/internal.(*localActivityTaskPoller).PollTask(0x14000471340?)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:540 +0x24
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852700)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852700)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 134 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852700)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130

goroutine 135 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x140008209c0, {0xa0?}, 0x1400042d9b0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0x140008209c0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:298 +0x70
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852620)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852620)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 136 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x140008209c0, {0xa0?}, 0x14000c805b0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0x140008209c0)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:298 +0x70
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852620)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852620)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 137 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852620)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130

goroutine 138 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x1400080de80, {0xf0?}, 0x14000799968)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0x1400080de80)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1042 +0x88
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852460)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852460)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 139 [select]:
go.uber.org/cadence/internal.(*basePoller).doPoll(0x1400080de80, {0xb0?}, 0x140001c9b78)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:261 +0x1c4
go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0x1400080de80)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_task_pollers.go:1042 +0x88
go.uber.org/cadence/internal.(*baseWorker).pollTask(0x14000852460)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:263 +0x90
go.uber.org/cadence/internal.(*baseWorker).runPoller(0x14000852460)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:227 +0x8c
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:190 +0x70

goroutine 140 [select]:
go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0x14000852460)
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:241 +0xdc
created by go.uber.org/cadence/internal.(*baseWorker).Start
    /Users/qlong/go/pkg/mod/go.uber.org/cadence@v0.19.0/internal/internal_worker_base.go:194 +0x130
FAIL    github.com/indeedeng/iwf/integ  1.891s
FAIL
make: *** [integTests] Error 1

Process finished with exit code 2

More repro again:

2024-05-20T19:36:41.8763137Z fatal error: concurrent map writes
2024-05-20T19:36:41.8794825Z 
2024-05-20T19:36:41.8795349Z goroutine 15556 [running]:
2024-05-20T19:36:41.8798377Z github.com/indeedeng/iwf/service/interpreter.(*ContinueAsNewer).allThreadsDrained(0xc001156240, {0x1a9bb20, 0xc000ff3ac0})
2024-05-20T19:36:41.8800225Z    /home/runner/work/iwf/iwf/service/interpreter/continueAsNewer.go:200 +0x779
2024-05-20T19:36:41.8802201Z github.com/indeedeng/iwf/service/interpreter.(*ContinueAsNewer).DrainThreads.func1()
2024-05-20T19:36:41.8803536Z    /home/runner/work/iwf/iwf/service/interpreter/continueAsNewer.go:160 +0x29
2024-05-20T19:36:41.8805022Z go.uber.org/cadence/internal.Await({0x1ab2890, 0xc000d7aba0}, 0xc0005b3aa0)
2024-05-20T19:36:41.8806708Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:597 +0xbb
2024-05-20T19:36:41.8807918Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.8809130Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.8810576Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x0?, {0x1a9bb20?, 0xc000ff3ac0?}, 0x0?)
2024-05-20T19:36:41.8812134Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.8813561Z github.com/indeedeng/iwf/service/interpreter.(*ContinueAsNewer).DrainThreads(0xc001156240, {0x1a9bb20?, 0xc000ff3ac0})
2024-05-20T19:36:41.8815570Z    /home/runner/work/iwf/iwf/service/interpreter/continueAsNewer.go:159 +0xc4
2024-05-20T19:36:41.8817629Z github.com/indeedeng/iwf/service/interpreter.InterpreterImpl({0x1a9bb20?, 0xc000ff3ac0}, {0x1ac6300, 0xc000ff3ad0}, {{0xc001132950, 0xb}, {0xc001424768, 0x15}, 0xc000ff3970, {0x0, ...}, ...})
2024-05-20T19:36:41.8819580Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:258 +0x1f68
2024-05-20T19:36:41.8821469Z github.com/indeedeng/iwf/service/interpreter/cadence.Interpreter({0x1ab2890?, 0xc000d7aba0}, {{0xc001132950, 0xb}, {0xc001424768, 0x15}, 0xc000ff3970, {0x0, 0x0, 0x0}, ...})
2024-05-20T19:36:41.8823356Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflow.go:10 +0x17d
2024-05-20T19:36:41.8824661Z reflect.Value.call({0x15f8fa0?, 0x19028a0?, 0x1834e00?}, {0x1838b1a, 0x4}, {0xc000a74e70, 0x2, 0x18?})
2024-05-20T19:36:41.8825998Z    /opt/hostedtoolcache/go/1.21.10/x64/src/reflect/value.go:596 +0xce7
2024-05-20T19:36:41.8827135Z reflect.Value.Call({0x15f8fa0?, 0x19028a0?, 0x108f808?}, {0xc000a74e70?, 0x1834e00?, 0x1a99670?})
2024-05-20T19:36:41.8828274Z    /opt/hostedtoolcache/go/1.21.10/x64/src/reflect/value.go:380 +0xb9
2024-05-20T19:36:41.8830097Z go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc00101cd50, {0x1ab2890?, 0xc000d7aba0?}, {0x15f8fa0?, 0x15d3d00?}, {0xc000ff3aa0?, 0x1, 0xc000a2be60?})
2024-05-20T19:36:41.8832484Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:688 +0x44a
2024-05-20T19:36:41.8834409Z go.uber.org/cadence/internal.(*workflowExecutor).Execute(0xc0004d0d00, {0x1ab2890, 0xc000d7aba0}, {0xc0009d4580, 0x2a2, 0x2c0})
2024-05-20T19:36:41.8836412Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker.go:667 +0x392
2024-05-20T19:36:41.8838033Z go.uber.org/cadence/internal.(*syncWorkflowDefinition).Execute.func1({0x1ab0868, 0xc00101d020})
2024-05-20T19:36:41.8839821Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:467 +0xcc
2024-05-20T19:36:41.8841376Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc001817630?)
2024-05-20T19:36:41.8843101Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.8844718Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 15555
2024-05-20T19:36:41.8846509Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.8847551Z 
2024-05-20T19:36:41.8847717Z goroutine 1 [chan receive]:
2024-05-20T19:36:41.8848359Z testing.(*T).Run(0xc0004aa820, {0x1885022?, 0x5873bc?}, 0x1901b70)
2024-05-20T19:36:41.8849310Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:1649 +0x3c8
2024-05-20T19:36:41.8850258Z testing.runTests.func1(0xc00058a800?)
2024-05-20T19:36:41.8851038Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:2054 +0x3e
2024-05-20T19:36:41.8852010Z testing.tRunner(0xc0004aa820, 0xc000907950)
2024-05-20T19:36:41.8852810Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:1595 +0xff
2024-05-20T19:36:41.8853978Z testing.runTests(0xc0005d3cc0?, {0x26d0e20, 0x55, 0x55}, {0xc0006c40c0?, 0xc0000f8408?, 0x26e0980?})
2024-05-20T19:36:41.8855152Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:2052 +0x445
2024-05-20T19:36:41.8856079Z testing.(*M).Run(0xc0005d3cc0)
2024-05-20T19:36:41.8856800Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:1925 +0x636
2024-05-20T19:36:41.8857672Z github.com/indeedeng/iwf/integ.TestMain(0x1838020?)
2024-05-20T19:36:41.8858433Z    /home/runner/work/iwf/iwf/integ/main_test.go:110 +0x152
2024-05-20T19:36:41.8859079Z main.main()
2024-05-20T19:36:41.8859435Z    _testmain.go:251 +0x1e6
2024-05-20T19:36:41.8859754Z 
2024-05-20T19:36:41.8859914Z goroutine 15237 [select]:
2024-05-20T19:36:41.8860762Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce580, {0xe0?, 0x53?}, 0xc000f71cb0)
2024-05-20T19:36:41.8862839Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.8864543Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0xc000fce580)
2024-05-20T19:36:41.8866398Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1142 +0xb5
2024-05-20T19:36:41.8867904Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc000)
2024-05-20T19:36:41.8869607Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.8871938Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc000)
2024-05-20T19:36:41.8873577Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.8875126Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.8876923Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.8878011Z 
2024-05-20T19:36:41.8878191Z goroutine 15573 [chan receive]:
2024-05-20T19:36:41.8879136Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000796640, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.8881194Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.8882532Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.8883970Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.8885381Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000a75500}, 0xc0007968c0)
2024-05-20T19:36:41.8886924Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.8888049Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.8889390Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.8891396Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x10?, {0x1a9bb20?, 0xc000ff3d60?}, 0x7fd73c2b05c0?)
2024-05-20T19:36:41.8893208Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.8894729Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func6({0x1a9bb20?, 0xc000ff3d60})
2024-05-20T19:36:41.8896408Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:169 +0x124
2024-05-20T19:36:41.8898016Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000a75500})
2024-05-20T19:36:41.8899831Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.8901254Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc0015ac870?)
2024-05-20T19:36:41.8903064Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.8905153Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 15556
2024-05-20T19:36:41.8907131Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.8908206Z 
2024-05-20T19:36:41.8908396Z goroutine 11474 [chan receive]:
2024-05-20T19:36:41.8909747Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc00060d940, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.8911636Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.8913038Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.8914820Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.8916286Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000ae5b60}, 0xc00060dac0)
2024-05-20T19:36:41.8917895Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.8918936Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.8920278Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.8922119Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x10?, {0x1a9bb20?, 0xc001190d00?}, 0x7fd73c137158?)
2024-05-20T19:36:41.8923626Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.8925101Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func6({0x1a9bb20?, 0xc001190d00})
2024-05-20T19:36:41.8926163Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:169 +0x124
2024-05-20T19:36:41.8927455Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000ae5b60})
2024-05-20T19:36:41.8929138Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.8930494Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc0017af380?)
2024-05-20T19:36:41.8932411Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.8933541Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 11468
2024-05-20T19:36:41.8934910Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.8936020Z 
2024-05-20T19:36:41.8936201Z goroutine 15531 [IO wait]:
2024-05-20T19:36:41.8936759Z internal/poll.runtime_pollWait(0x7fd78727e310, 0x72)
2024-05-20T19:36:41.8937635Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/netpoll.go:343 +0x85
2024-05-20T19:36:41.8938591Z internal/poll.(*pollDesc).wait(0xc001001580?, 0xc001158f11?, 0x0)
2024-05-20T19:36:41.8939627Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2024-05-20T19:36:41.8940528Z internal/poll.(*pollDesc).waitRead(...)
2024-05-20T19:36:41.8941366Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-20T19:36:41.8942360Z internal/poll.(*FD).Read(0xc001001580, {0xc001158f11, 0x1, 0x1})
2024-05-20T19:36:41.8943310Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_unix.go:164 +0x27a
2024-05-20T19:36:41.8944361Z net.(*netFD).Read(0xc001001580, {0xc001158f11?, 0xc000942f40?, 0x470930?})
2024-05-20T19:36:41.8945318Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/fd_posix.go:55 +0x25
2024-05-20T19:36:41.8946411Z net.(*conn).Read(0xc001584388, {0xc001158f11?, 0x37?, 0xc000942ee8?})
2024-05-20T19:36:41.8947304Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/net.go:179 +0x45
2024-05-20T19:36:41.8948098Z net/http.(*connReader).backgroundRead(0xc001158f00)
2024-05-20T19:36:41.8948954Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:683 +0x37
2024-05-20T19:36:41.8949992Z created by net/http.(*connReader).startBackgroundRead in goroutine 15288
2024-05-20T19:36:41.8951019Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:679 +0xba
2024-05-20T19:36:41.8951637Z 
2024-05-20T19:36:41.8951794Z goroutine 15217 [select]:
2024-05-20T19:36:41.8952646Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce580, {0xe0?, 0x53?}, 0xc00139eb40)
2024-05-20T19:36:41.8954490Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.8956152Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0xc000fce580)
2024-05-20T19:36:41.8958221Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1142 +0xb5
2024-05-20T19:36:41.8959679Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc000)
2024-05-20T19:36:41.8961293Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.8962717Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc000)
2024-05-20T19:36:41.8964322Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.8965841Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.8967520Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.8968551Z 
2024-05-20T19:36:41.8968740Z goroutine 12 [IO wait, 7 minutes]:
2024-05-20T19:36:41.8969409Z internal/poll.runtime_pollWait(0x7fd78727edb8, 0x72)
2024-05-20T19:36:41.8970299Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/netpoll.go:343 +0x85
2024-05-20T19:36:41.8971245Z internal/poll.(*pollDesc).wait(0xc0006ba000?, 0xc0006c6000?, 0x0)
2024-05-20T19:36:41.8972292Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2024-05-20T19:36:41.8973209Z internal/poll.(*pollDesc).waitRead(...)
2024-05-20T19:36:41.8974046Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-20T19:36:41.8975444Z internal/poll.(*FD).Read(0xc0006ba000, {0xc0006c6000, 0x8000, 0x8000})
2024-05-20T19:36:41.8976491Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_unix.go:164 +0x27a
2024-05-20T19:36:41.8977505Z net.(*netFD).Read(0xc0006ba000, {0xc0006c6000?, 0x1040100000000?, 0x0?})
2024-05-20T19:36:41.8978430Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/fd_posix.go:55 +0x25
2024-05-20T19:36:41.8979374Z net.(*conn).Read(0xc0004cc010, {0xc0006c6000?, 0xc0006b1d68?, 0xbd3aff?})
2024-05-20T19:36:41.8980276Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/net.go:179 +0x45
2024-05-20T19:36:41.8981116Z bufio.(*Reader).Read(0xc0006c4000, {0xc0006de040, 0x9, 0x30?})
2024-05-20T19:36:41.8981992Z    /opt/hostedtoolcache/go/1.21.10/x64/src/bufio/bufio.go:244 +0x197
2024-05-20T19:36:41.8982960Z io.ReadAtLeast({0x1a9ae40, 0xc0006c4000}, {0xc0006de040, 0x9, 0x9}, 0x9)
2024-05-20T19:36:41.8983863Z    /opt/hostedtoolcache/go/1.21.10/x64/src/io/io.go:335 +0x90
2024-05-20T19:36:41.8984532Z io.ReadFull(...)
2024-05-20T19:36:41.8984927Z    /opt/hostedtoolcache/go/1.21.10/x64/src/io/io.go:354
2024-05-20T19:36:41.8985708Z golang.org/x/net/http2.readFrameHeader({0xc0006de040, 0x9, 0xc0006b4450?}, {0x1a9ae40?, 0xc0006c4000?})
2024-05-20T19:36:41.8986611Z    /home/runner/go/pkg/mod/golang.org/x/net@v0.21.0/http2/frame.go:237 +0x65
2024-05-20T19:36:41.8987316Z golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006de000)
2024-05-20T19:36:41.8988169Z    /home/runner/go/pkg/mod/golang.org/x/net@v0.21.0/http2/frame.go:498 +0x85
2024-05-20T19:36:41.8989101Z google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00054ed80, 0xc000794000?)
2024-05-20T19:36:41.8990656Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/http2_client.go:1595 +0x245
2024-05-20T19:36:41.8992129Z created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 25
2024-05-20T19:36:41.8993587Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/http2_client.go:397 +0x1e9e
2024-05-20T19:36:41.8994518Z 
2024-05-20T19:36:41.8994699Z goroutine 23 [select, 7 minutes]:
2024-05-20T19:36:41.8995762Z google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000440300, {0x1ab04e8, 0xc0004d6140})
2024-05-20T19:36:41.8997422Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/grpcsync/callback_serializer.go:83 +0x112
2024-05-20T19:36:41.8998970Z created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 20
2024-05-20T19:36:41.9000541Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/grpcsync/callback_serializer.go:55 +0x129
2024-05-20T19:36:41.9001727Z 
2024-05-20T19:36:41.9001934Z goroutine 24 [select, 7 minutes]:
2024-05-20T19:36:41.9002980Z google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000440360, {0x1ab04e8, 0xc0004d6190})
2024-05-20T19:36:41.9004967Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/grpcsync/callback_serializer.go:83 +0x112
2024-05-20T19:36:41.9006548Z created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 20
2024-05-20T19:36:41.9008138Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/grpcsync/callback_serializer.go:55 +0x129
2024-05-20T19:36:41.9009117Z 
2024-05-20T19:36:41.9009299Z goroutine 26 [select, 7 minutes]:
2024-05-20T19:36:41.9010307Z google.golang.org/grpc.(*ClientConn).WaitForStateChange(0xc000018000, {0x1ab04e8, 0xc0004d6370}, 0x2)
2024-05-20T19:36:41.9011676Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/clientconn.go:684 +0xb1
2024-05-20T19:36:41.9012943Z go.uber.org/yarpc/transport/grpc.(*grpcPeer).monitorConnectionStatus(0xc0003d8300)
2024-05-20T19:36:41.9014219Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/peer.go:82 +0x85
2024-05-20T19:36:41.9015637Z created by go.uber.org/yarpc/transport/grpc.(*Transport).newPeer in goroutine 20
2024-05-20T19:36:41.9016918Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/peer.go:69 +0x578
2024-05-20T19:36:41.9017834Z 
2024-05-20T19:36:41.9018048Z goroutine 6551 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9019079Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc0000f0480, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9020979Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9022355Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9023875Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9025341Z go.uber.org/cadence/internal.Await({0x1ab2890, 0xc000a56480}, 0xc0000f07c0)
2024-05-20T19:36:41.9026938Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9028125Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9029519Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9031269Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc000a91110?, {0x1a9bb20?, 0xc000a91100?}, 0xc001784f40?)
2024-05-20T19:36:41.9033076Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9035216Z github.com/indeedeng/iwf/service/interpreter.InterpreterImpl({0x1a9bb20?, 0xc000a91100}, {0x1ac6300, 0xc000a91110}, {{0xc0002e5648, 0x6}, {0xc0018a52d8, 0x15}, 0xc000a910d0, {0x0, ...}, ...})
2024-05-20T19:36:41.9037315Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:248 +0x1efb
2024-05-20T19:36:41.9038717Z github.com/indeedeng/iwf/service/interpreter/cadence.Interpreter({0x1ab2890?, 0xc000a56480}, {{0xc0002e5648, 0x6}, {0xc0018a52d8, 0x15}, 0xc000a910d0, {0x0, 0x0, 0x0}, ...})
2024-05-20T19:36:41.9040144Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflow.go:10 +0x17d
2024-05-20T19:36:41.9041216Z reflect.Value.call({0x15f8fa0?, 0x19028a0?, 0x1834e00?}, {0x1838b1a, 0x4}, {0xc001176ae0, 0x2, 0x18?})
2024-05-20T19:36:41.9042432Z    /opt/hostedtoolcache/go/1.21.10/x64/src/reflect/value.go:596 +0xce7
2024-05-20T19:36:41.9043618Z reflect.Value.Call({0x15f8fa0?, 0x19028a0?, 0x108f808?}, {0xc001176ae0?, 0x1834e00?, 0x1a99670?})
2024-05-20T19:36:41.9044788Z    /opt/hostedtoolcache/go/1.21.10/x64/src/reflect/value.go:380 +0xb9
2024-05-20T19:36:41.9046619Z go.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow(0xc0011766f0, {0x1ab2890?, 0xc000a56480?}, {0x15f8fa0?, 0x15d3d00?}, {0xc000a910e0?, 0x1, 0xc000bbae60?})
2024-05-20T19:36:41.9049294Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:688 +0x44a
2024-05-20T19:36:41.9051105Z go.uber.org/cadence/internal.(*workflowExecutor).Execute(0xc000441900, {0x1ab2890, 0xc000a56480}, {0xc0007384d0, 0xa4, 0xb0})
2024-05-20T19:36:41.9053130Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker.go:667 +0x392
2024-05-20T19:36:41.9054347Z go.uber.org/cadence/internal.(*syncWorkflowDefinition).Execute.func1({0x1ab0868, 0xc0011769f0})
2024-05-20T19:36:41.9055917Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:467 +0xcc
2024-05-20T19:36:41.9057221Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000c08c50?)
2024-05-20T19:36:41.9058620Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9059913Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6550
2024-05-20T19:36:41.9061374Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9062077Z 
2024-05-20T19:36:41.9062248Z goroutine 15471 [select]:
2024-05-20T19:36:41.9062641Z net/http.(*persistConn).writeLoop(0xc000cadc20)
2024-05-20T19:36:41.9063605Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:2421 +0xe5
2024-05-20T19:36:41.9064302Z created by net/http.(*Transport).dialConn in goroutine 15516
2024-05-20T19:36:41.9065364Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:1777 +0x16f1
2024-05-20T19:36:41.9065950Z 
2024-05-20T19:36:41.9066133Z goroutine 15134 [select]:
2024-05-20T19:36:41.9066535Z net/http.(*persistConn).roundTrip(0xc00161cb40, 0xc00069f980)
2024-05-20T19:36:41.9067363Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:2652 +0x979
2024-05-20T19:36:41.9068166Z net/http.(*Transport).roundTrip(0x26c9ae0, 0xc0011c5400)
2024-05-20T19:36:41.9068832Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:604 +0x79a
2024-05-20T19:36:41.9069622Z net/http.(*Transport).RoundTrip(0x4279bc?, 0x1a9b880?)
2024-05-20T19:36:41.9070372Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/roundtrip.go:17 +0x13
2024-05-20T19:36:41.9071132Z net/http.send(0xc0011c5400, {0x1a9b880, 0x26c9ae0}, {0x412a01?, 0xc0011c5400?, 0x0?})
2024-05-20T19:36:41.9072084Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/client.go:260 +0x606
2024-05-20T19:36:41.9072895Z net/http.(*Client).send(0x26dfd40, 0xc0011c5400, {0xf8?, 0x1814560?, 0x0?})
2024-05-20T19:36:41.9073594Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/client.go:181 +0x98
2024-05-20T19:36:41.9074363Z net/http.(*Client).do(0x26dfd40, 0xc0011c5400)
2024-05-20T19:36:41.9075387Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/client.go:724 +0x912
2024-05-20T19:36:41.9076192Z net/http.(*Client).Do(...)
2024-05-20T19:36:41.9076853Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/client.go:590
2024-05-20T19:36:41.9077967Z github.com/indeedeng/iwf/gen/iwfidl.(*APIClient).callAPI(0xc000be05e0, 0xc0006e8de8?)
2024-05-20T19:36:41.9079069Z    /home/runner/work/iwf/iwf/gen/iwfidl/client.go:248 +0x107
2024-05-20T19:36:41.9080456Z github.com/indeedeng/iwf/gen/iwfidl.(*DefaultApiService).ApiV1WorkflowGetWithWaitPostExecute(0xc000be05e8, {{0x1ab0328, 0x2712020}, {0x1aca7c0, 0xc000be05e8}, 0xc0006b5560})
2024-05-20T19:36:41.9081815Z    /home/runner/work/iwf/iwf/gen/iwfidl/api_default.go:660 +0x45f
2024-05-20T19:36:41.9082747Z github.com/indeedeng/iwf/gen/iwfidl.ApiApiV1WorkflowGetWithWaitPostRequest.Execute(...)
2024-05-20T19:36:41.9083677Z    /home/runner/work/iwf/iwf/gen/iwfidl/api_default.go:598
2024-05-20T19:36:41.9084643Z github.com/indeedeng/iwf/integ.doTestPersistenceWorkflow(0xc000ba3380?, {0x184257c, 0x7}, 0x0, 0x0?, 0xc0013b68a0)
2024-05-20T19:36:41.9085754Z    /home/runner/work/iwf/iwf/integ/persistence_test.go:334 +0x3466
2024-05-20T19:36:41.9086897Z github.com/indeedeng/iwf/integ.TestPersistenceWorkflowCadenceContinueAsNew(0xc000ba3380)
2024-05-20T19:36:41.9087958Z    /home/runner/work/iwf/iwf/integ/persistence_test.go:94 +0x5b
2024-05-20T19:36:41.9088433Z testing.tRunner(0xc000ba3380, 0x1901b70)
2024-05-20T19:36:41.9089120Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:1595 +0xff
2024-05-20T19:36:41.9089771Z created by testing.(*T).Run in goroutine 1
2024-05-20T19:36:41.9090613Z    /opt/hostedtoolcache/go/1.21.10/x64/src/testing/testing.go:1648 +0x3ad
2024-05-20T19:36:41.9091271Z 
2024-05-20T19:36:41.9091479Z goroutine 6554 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9092524Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc0000f0600, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9094403Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9095791Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9096969Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9098015Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc001177020}, 0xc001177590)
2024-05-20T19:36:41.9099279Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9100082Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9101354Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9102965Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc00082fea0?, {0x1a9bb20?, 0xc000a91350?}, 0x186e6a3?)
2024-05-20T19:36:41.9104555Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9106308Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func3({0x1a9bb20, 0xc000a91350})
2024-05-20T19:36:41.9107710Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:99 +0x172
2024-05-20T19:36:41.9109269Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc001177020})
2024-05-20T19:36:41.9110914Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9112283Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000c09df0?)
2024-05-20T19:36:41.9114065Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9115714Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6551
2024-05-20T19:36:41.9117569Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9118780Z 
2024-05-20T19:36:41.9118971Z goroutine 13 [select, 7 minutes]:
2024-05-20T19:36:41.9119833Z google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00007ec80, 0x1)
2024-05-20T19:36:41.9121272Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/controlbuf.go:418 +0x113
2024-05-20T19:36:41.9122621Z google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000452070)
2024-05-20T19:36:41.9124437Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/controlbuf.go:552 +0x86
2024-05-20T19:36:41.9125330Z google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
2024-05-20T19:36:41.9126202Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/http2_client.go:451 +0x85
2024-05-20T19:36:41.9127290Z created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 25
2024-05-20T19:36:41.9128246Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/http2_client.go:449 +0x2450
2024-05-20T19:36:41.9128902Z 
2024-05-20T19:36:41.9128998Z goroutine 15435 [select]:
2024-05-20T19:36:41.9129456Z google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0008e0240)
2024-05-20T19:36:41.9130455Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:331 +0x7c
2024-05-20T19:36:41.9131205Z google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
2024-05-20T19:36:41.9131907Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:346
2024-05-20T19:36:41.9132706Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc001441520, {0x155e160?, 0xc000e744e0}, 0xc0009224f8?)
2024-05-20T19:36:41.9133438Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1066 +0xc9
2024-05-20T19:36:41.9134060Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2712020?)
2024-05-20T19:36:41.9134655Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:917 +0x1f
2024-05-20T19:36:41.9135635Z google.golang.org/grpc.(*clientStream).withRetry(0xc0008e0000, 0xc000922528, 0xc0009224f8)
2024-05-20T19:36:41.9136344Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:768 +0x13a
2024-05-20T19:36:41.9137030Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc0008e0000, {0x155e160?, 0xc000e744e0?})
2024-05-20T19:36:41.9137709Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:916 +0x12e
2024-05-20T19:36:41.9138671Z google.golang.org/grpc.invoke({0x1ab04b0?, 0xc000e93140?}, {0xc000d7cd00?, 0x412a05?}, {0x155f420, 0xc000e74528}, {0x155e160, 0xc000e744e0}, 0x412d85?, {0xc000a03780, ...})
2024-05-20T19:36:41.9139714Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:78 +0xcb
2024-05-20T19:36:41.9140696Z google.golang.org/grpc.(*ClientConn).Invoke(0xc000814800, {0x1ab04b0, 0xc000e93140}, {0xc000d7cd00, 0x32}, {0x155f420, 0xc000e74528}, {0x155e160, 0xc000e744e0}, {0xc0013d4a90, ...})
2024-05-20T19:36:41.9141669Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:42 +0x2cb
2024-05-20T19:36:41.9142693Z go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0xc0013b6ab0, {0x1ab0558, 0xc00043af50}, 0xc0009a4540, 0x1838f92?, 0xc001776a40, {0xc18b07726b75ad41?, 0xc001018ea8?, 0x26e0980?})
2024-05-20T19:36:41.9143777Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x4e5
2024-05-20T19:36:41.9144581Z go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0xc0013b6ab0, {0x1ab0558, 0xc00043af50}, 0xc0009a4540)
2024-05-20T19:36:41.9145371Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0x126
2024-05-20T19:36:41.9146433Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558?, 0xc00043af50?}, 0x184f9bf?)
2024-05-20T19:36:41.9147527Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
2024-05-20T19:36:41.9148529Z go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc00183bc80, {0x1ab0558, 0xc00043af50}, 0xc0009a4540, {0x1ab4e00, 0xc000e93050})
2024-05-20T19:36:41.9149619Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0x162
2024-05-20T19:36:41.9150718Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc00043af50}, 0xc0013b6ab0?)
2024-05-20T19:36:41.9151801Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9153372Z go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x17706e0?, {0x1ab0558, 0xc00043af50}, 0xc0009a4540, {0x1ab4e00?, 0xc000e93020})
2024-05-20T19:36:41.9155835Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xa8
2024-05-20T19:36:41.9162861Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x1, 0x1}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc00043af50}, 0xc000814c00?)
2024-05-20T19:36:41.9165578Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9170553Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
2024-05-20T19:36:41.9172357Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
2024-05-20T19:36:41.9177522Z go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
2024-05-20T19:36:41.9178793Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
2024-05-20T19:36:41.9180659Z go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x1ab4c80?, 0xc0013b6c30?}, {0x7fd73c34b008?, 0xc0013b6c30?}}, {0x1ab0558, 0xc00043af50}, 0x412a05?)
2024-05-20T19:36:41.9182712Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x92
2024-05-20T19:36:41.9184099Z go.uber.org/yarpc/encoding/protobuf.(*client).Call(0xc0013b7260, {0x1ab0558?, 0xc00043af50?}, {0x18556fb?, 0x7?}, {0x1aa9be0?, 0xc0008739a0?}, 0x1903898, {0xc0018a61e0, 0x4, ...})
2024-05-20T19:36:41.9185180Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0x10c
2024-05-20T19:36:41.9186562Z github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0xc001562960?, {0x1ab0558?, 0xc00043af50?}, 0xc000c37cc0?, {0xc0018a61e0?, 0xc000c37c90?, 0x184f117?})
2024-05-20T19:36:41.9188072Z    /home/runner/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220713235846-fda89e95df1e/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x66
2024-05-20T19:36:41.9189794Z go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1ab3c80, 0xc0010956f0}, {0x1abd1d8, 0xc001095710}, {0x1abd260, 0xc001095730}, {0x1ab7ea0, 0xc001095750}}, {0x1ab0558, 0xc00043af50}, ...)
2024-05-20T19:36:41.9191337Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/compatibility/adapter.go:121 +0x95
2024-05-20T19:36:41.9192626Z go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0xc001586fc0, {0x1ab0558, 0xc00043af50}, 0xc000c6fd40?, {0xc0018a61e0, 0x4, 0x6})
2024-05-20T19:36:41.9194052Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/metrics/service_wrapper.go:213 +0x9d
2024-05-20T19:36:41.9195027Z go.uber.org/cadence/internal.(*activityTaskPoller).poll(0xc000fce500, {0x1ab0558, 0xc00043af50})
2024-05-20T19:36:41.9196046Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1005 +0x33e
2024-05-20T19:36:41.9197300Z go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0xc000fce500, {0x1ab0558?, 0xc00043af50?}, 0xc000c1a1d8?)
2024-05-20T19:36:41.9198457Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1048 +0x42
2024-05-20T19:36:41.9199559Z go.uber.org/cadence/internal.(*activityTaskPoller).PollTask.(*activityTaskPoller).pollWithMetricsFunc.func1({0x1ab0558?, 0xc00043af50?})
2024-05-20T19:36:41.9200885Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1042 +0x2e
2024-05-20T19:36:41.9201834Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9202704Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9203558Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15226
2024-05-20T19:36:41.9204525Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9205121Z 
2024-05-20T19:36:41.9205220Z goroutine 15226 [select]:
2024-05-20T19:36:41.9205715Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce500, {0xe0?, 0x53?}, 0xc000e744b0)
2024-05-20T19:36:41.9206705Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9207546Z go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0xc000fce500)
2024-05-20T19:36:41.9208665Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1071 +0xb5
2024-05-20T19:36:41.9209708Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fc9b00)
2024-05-20T19:36:41.9210629Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9211448Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fc9b00)
2024-05-20T19:36:41.9212337Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9213181Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9214124Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9214702Z 
2024-05-20T19:36:41.9214804Z goroutine 15238 [select]:
2024-05-20T19:36:41.9215486Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce580, {0xe0?, 0x53?}, 0xc000e743d8)
2024-05-20T19:36:41.9216502Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9217427Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0xc000fce580)
2024-05-20T19:36:41.9218572Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1142 +0xb5
2024-05-20T19:36:41.9219374Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc000)
2024-05-20T19:36:41.9220263Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9221067Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc000)
2024-05-20T19:36:41.9221946Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9222795Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9223734Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9224307Z 
2024-05-20T19:36:41.9224407Z goroutine 15188 [select]:
2024-05-20T19:36:41.9224975Z google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000691300, {0x1ab04e8, 0xc001182370})
2024-05-20T19:36:41.9225902Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/grpcsync/callback_serializer.go:83 +0x112
2024-05-20T19:36:41.9226782Z created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 15137
2024-05-20T19:36:41.9227650Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/grpcsync/callback_serializer.go:55 +0x129
2024-05-20T19:36:41.9228276Z 
2024-05-20T19:36:41.9228372Z goroutine 15470 [IO wait]:
2024-05-20T19:36:41.9228707Z internal/poll.runtime_pollWait(0x7fd78727e7e8, 0x72)
2024-05-20T19:36:41.9229217Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/netpoll.go:343 +0x85
2024-05-20T19:36:41.9229762Z internal/poll.(*pollDesc).wait(0xc0010acd00?, 0xc000b1a000?, 0x0)
2024-05-20T19:36:41.9230368Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2024-05-20T19:36:41.9230887Z internal/poll.(*pollDesc).waitRead(...)
2024-05-20T19:36:41.9231370Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-20T19:36:41.9231964Z internal/poll.(*FD).Read(0xc0010acd00, {0xc000b1a000, 0x1000, 0x1000})
2024-05-20T19:36:41.9232556Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_unix.go:164 +0x27a
2024-05-20T19:36:41.9233160Z net.(*netFD).Read(0xc0010acd00, {0xc000b1a000?, 0xc0010acd00?, 0x6fc23ac00?})
2024-05-20T19:36:41.9233718Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/fd_posix.go:55 +0x25
2024-05-20T19:36:41.9234223Z net.(*conn).Read(0xc001584848, {0xc000b1a000?, 0x0?, 0x1ab6100?})
2024-05-20T19:36:41.9234840Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/net.go:179 +0x45
2024-05-20T19:36:41.9235396Z net/http.(*persistConn).Read(0xc000cadc20, {0xc000b1a000?, 0x40a8c5?, 0x60?})
2024-05-20T19:36:41.9235998Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:1954 +0x4a
2024-05-20T19:36:41.9236464Z bufio.(*Reader).fill(0xc000d7a120)
2024-05-20T19:36:41.9236863Z    /opt/hostedtoolcache/go/1.21.10/x64/src/bufio/bufio.go:113 +0x103
2024-05-20T19:36:41.9237727Z bufio.(*Reader).Peek(0xc000d7a120, 0x1)
2024-05-20T19:36:41.9238163Z    /opt/hostedtoolcache/go/1.21.10/x64/src/bufio/bufio.go:151 +0x53
2024-05-20T19:36:41.9238622Z net/http.(*persistConn).readLoop(0xc000cadc20)
2024-05-20T19:36:41.9239120Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:2118 +0x1b9
2024-05-20T19:36:41.9239681Z created by net/http.(*Transport).dialConn in goroutine 15516
2024-05-20T19:36:41.9240237Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:1776 +0x169f
2024-05-20T19:36:41.9240612Z 
2024-05-20T19:36:41.9240708Z goroutine 14773 [sleep]:
2024-05-20T19:36:41.9240970Z time.Sleep(0x46dcc5f0)
2024-05-20T19:36:41.9241334Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/time.go:195 +0x125
2024-05-20T19:36:41.9242036Z go.uber.org/cadence/internal/common/backoff.(*ConcurrentRetrier).throttleInternal(0xc0011733c0)
2024-05-20T19:36:41.9243099Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:65 +0x8b
2024-05-20T19:36:41.9244068Z go.uber.org/cadence/internal/common/backoff.(*ConcurrentRetrier).Throttle(...)
2024-05-20T19:36:41.9244990Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:51
2024-05-20T19:36:41.9245757Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc00161de60)
2024-05-20T19:36:41.9246651Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:291 +0x21c
2024-05-20T19:36:41.9247447Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc00161de60)
2024-05-20T19:36:41.9248331Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9249173Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 14758
2024-05-20T19:36:41.9250106Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9250689Z 
2024-05-20T19:36:41.9250782Z goroutine 15465 [select]:
2024-05-20T19:36:41.9251231Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTunnel).getTask(...)
2024-05-20T19:36:41.9252156Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:193
2024-05-20T19:36:41.9253332Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).pollLocallyDispatchedActivity(0xc000fce580, {0xc000505120?, 0x26b40f0?})
2024-05-20T19:36:41.9254558Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1151 +0x8b
2024-05-20T19:36:41.9255809Z go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0xc000fce580, {0x1ab0558?, 0xc000407e30?}, 0x3?)
2024-05-20T19:36:41.9256921Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1048 +0x42
2024-05-20T19:36:41.9258100Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask.(*activityTaskPoller).pollWithMetricsFunc.func1({0x1ab0558?, 0xc000407e30?})
2024-05-20T19:36:41.9259369Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1042 +0x2e
2024-05-20T19:36:41.9260139Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9261000Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9262019Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15214
2024-05-20T19:36:41.9262987Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9263578Z 
2024-05-20T19:36:41.9263679Z goroutine 15227 [select]:
2024-05-20T19:36:41.9264104Z go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0xc000fc9b00)
2024-05-20T19:36:41.9265036Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:256 +0x11d
2024-05-20T19:36:41.9265885Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9266821Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:209 +0x138
2024-05-20T19:36:41.9267403Z 
2024-05-20T19:36:41.9267496Z goroutine 15236 [select]:
2024-05-20T19:36:41.9267989Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce580, {0xe0?, 0x53?}, 0xc000be8ac8)
2024-05-20T19:36:41.9268983Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9269942Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0xc000fce580)
2024-05-20T19:36:41.9270971Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1142 +0xb5
2024-05-20T19:36:41.9271857Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc000)
2024-05-20T19:36:41.9272749Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9273545Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc000)
2024-05-20T19:36:41.9274436Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9275283Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9276215Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9276785Z 
2024-05-20T19:36:41.9276877Z goroutine 15276 [select]:
2024-05-20T19:36:41.9277193Z net/http.(*persistConn).writeLoop(0xc000f27d40)
2024-05-20T19:36:41.9277698Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:2421 +0xe5
2024-05-20T19:36:41.9278257Z created by net/http.(*Transport).dialConn in goroutine 15257
2024-05-20T19:36:41.9278813Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:1777 +0x16f1
2024-05-20T19:36:41.9279183Z 
2024-05-20T19:36:41.9279287Z goroutine 11814 [semacquire]:
2024-05-20T19:36:41.9279582Z sync.runtime_Semacquire(0xc000944fd0?)
2024-05-20T19:36:41.9280092Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/sema.go:62 +0x25
2024-05-20T19:36:41.9280518Z sync.(*WaitGroup).Wait(0x10c5be5?)
2024-05-20T19:36:41.9280942Z    /opt/hostedtoolcache/go/1.21.10/x64/src/sync/waitgroup.go:116 +0x48
2024-05-20T19:36:41.9281506Z go.uber.org/cadence/internal/common/util.AwaitWaitGroup.func1()
2024-05-20T19:36:41.9282569Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/util/util.go:52 +0x25
2024-05-20T19:36:41.9283443Z created by go.uber.org/cadence/internal/common/util.AwaitWaitGroup in goroutine 11484
2024-05-20T19:36:41.9284417Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/util/util.go:51 +0x7d
2024-05-20T19:36:41.9284973Z 
2024-05-20T19:36:41.9285095Z goroutine 6401 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9285664Z go.temporal.io/sdk/internal.(*coroutineState).initialYield(0xc000890cd0, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9286438Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:945 +0x5f
2024-05-20T19:36:41.9287065Z go.temporal.io/sdk/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9287789Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:955
2024-05-20T19:36:41.9288654Z go.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).Await(0x1ab08a0?, {0x1ab08a0, 0xc000a8d920}, 0xc000a8df50)
2024-05-20T19:36:41.9289490Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:364 +0xa5
2024-05-20T19:36:41.9290140Z go.temporal.io/sdk/internal.Await({0x1ab08a0, 0xc000a8d920}, 0xc000a8d920?)
2024-05-20T19:36:41.9290788Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:349 +0x58
2024-05-20T19:36:41.9291298Z go.temporal.io/sdk/workflow.Await(...)
2024-05-20T19:36:41.9291840Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/workflow/deterministic_wrappers.go:84
2024-05-20T19:36:41.9292782Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).Await(0x10?, {0x1a9bb20?, 0xc0004cf450?}, 0x1865c58?)
2024-05-20T19:36:41.9293737Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:203 +0x7e
2024-05-20T19:36:41.9294591Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func2({0x1a9bb20, 0xc0004cf450})
2024-05-20T19:36:41.9295642Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:75 +0x1c4
2024-05-20T19:36:41.9296608Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).GoNamed.func1({0x1ab08a0?, 0xc000a8d920})
2024-05-20T19:36:41.9297814Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:180 +0xfd
2024-05-20T19:36:41.9298557Z go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine.func1(0xc000ef6cd0?)
2024-05-20T19:36:41.9299291Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1057 +0xcf
2024-05-20T19:36:41.9300041Z created by go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine in goroutine 6399
2024-05-20T19:36:41.9300811Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1048 +0x338
2024-05-20T19:36:41.9301263Z 
2024-05-20T19:36:41.9301364Z goroutine 15151 [select]:
2024-05-20T19:36:41.9301753Z go.uber.org/cadence/internal.(*localActivityTunnel).getTask(...)
2024-05-20T19:36:41.9302645Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:167
2024-05-20T19:36:41.9303488Z go.uber.org/cadence/internal.(*localActivityTaskPoller).PollTask(0xc000c5aae0?)
2024-05-20T19:36:41.9304450Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:540 +0x65
2024-05-20T19:36:41.9305239Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc900)
2024-05-20T19:36:41.9306120Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9306926Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc900)
2024-05-20T19:36:41.9307896Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9308743Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9309680Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9310410Z 
2024-05-20T19:36:41.9310547Z goroutine 6556 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9311118Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc0000f06c0, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9312122Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9312873Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9313674Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9314466Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc001177170}, 0xc0011775f0)
2024-05-20T19:36:41.9315496Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9316152Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9316911Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9318010Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc000a910b0?, {0x1a9bb20?, 0xc000a91390?}, 0x185a519?)
2024-05-20T19:36:41.9318994Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9319826Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func5({0x1a9bb20, 0xc000a91390})
2024-05-20T19:36:41.9320600Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:143 +0x1a9
2024-05-20T19:36:41.9321474Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc001177170})
2024-05-20T19:36:41.9322381Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9323127Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000c08c50?)
2024-05-20T19:36:41.9324094Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9324993Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6551
2024-05-20T19:36:41.9326062Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9326636Z 
2024-05-20T19:36:41.9326729Z goroutine 15543 [select]:
2024-05-20T19:36:41.9327181Z google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000d73e60)
2024-05-20T19:36:41.9327953Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:331 +0x7c
2024-05-20T19:36:41.9328673Z google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
2024-05-20T19:36:41.9329387Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:346
2024-05-20T19:36:41.9330201Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc00085c1a0, {0x155e160?, 0xc000f71ea8}, 0xc0008664f8?)
2024-05-20T19:36:41.9330924Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1066 +0xc9
2024-05-20T19:36:41.9331525Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2712020?)
2024-05-20T19:36:41.9332120Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:917 +0x1f
2024-05-20T19:36:41.9332798Z google.golang.org/grpc.(*clientStream).withRetry(0xc000d73c20, 0xc000866528, 0xc0008664f8)
2024-05-20T19:36:41.9333489Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:768 +0x13a
2024-05-20T19:36:41.9334171Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d73c20, {0x155e160?, 0xc000f71ea8?})
2024-05-20T19:36:41.9334930Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:916 +0x12e
2024-05-20T19:36:41.9336343Z google.golang.org/grpc.invoke({0x1ab04b0?, 0xc000898ae0?}, {0xc0006f3c40?, 0x412a05?}, {0x155f420, 0xc000f71ef0}, {0x155e160, 0xc000f71ea8}, 0x412d85?, {0xc000dfa000, ...})
2024-05-20T19:36:41.9337310Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:78 +0xcb
2024-05-20T19:36:41.9338315Z google.golang.org/grpc.(*ClientConn).Invoke(0xc000814800, {0x1ab04b0, 0xc000898ae0}, {0xc0006f3c40, 0x32}, {0x155f420, 0xc000f71ef0}, {0x155e160, 0xc000f71ea8}, {0xc00088ce40, ...})
2024-05-20T19:36:41.9339307Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:42 +0x2cb
2024-05-20T19:36:41.9340337Z go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0xc0013b6ab0, {0x1ab0558, 0xc0003c2460}, 0xc000d44600, 0x1838f92?, 0xc0017771e8, {0xc18b07727267fbd8?, 0xc001018ea8?, 0x26e0980?})
2024-05-20T19:36:41.9341408Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x4e5
2024-05-20T19:36:41.9342370Z go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0xc0013b6ab0, {0x1ab0558, 0xc0003c2460}, 0xc000d44600)
2024-05-20T19:36:41.9343164Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0x126
2024-05-20T19:36:41.9344217Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558?, 0xc0003c2460?}, 0x184f9bf?)
2024-05-20T19:36:41.9345302Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
2024-05-20T19:36:41.9346303Z go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc00183bc80, {0x1ab0558, 0xc0003c2460}, 0xc000d44600, {0x1ab4e00, 0xc0008989f0})
2024-05-20T19:36:41.9347294Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0x162
2024-05-20T19:36:41.9348383Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc0003c2460}, 0xc0013b6ab0?)
2024-05-20T19:36:41.9349480Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9350522Z go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x17706e0?, {0x1ab0558, 0xc0003c2460}, 0xc000d44600, {0x1ab4e00?, 0xc0008989c0})
2024-05-20T19:36:41.9351624Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xa8
2024-05-20T19:36:41.9352857Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x1, 0x1}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc0003c2460}, 0xc000070c00?)
2024-05-20T19:36:41.9353933Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9354896Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
2024-05-20T19:36:41.9355597Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
2024-05-20T19:36:41.9356316Z go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
2024-05-20T19:36:41.9356990Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
2024-05-20T19:36:41.9357996Z go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x1ab4c80?, 0xc0013b6c30?}, {0x7fd73c34b008?, 0xc0013b6c30?}}, {0x1ab0558, 0xc0003c2460}, 0x1860a14?)
2024-05-20T19:36:41.9359070Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x92
2024-05-20T19:36:41.9360176Z go.uber.org/yarpc/encoding/protobuf.(*client).Call(0xc0013b7260, {0x1ab0558?, 0xc0003c2460?}, {0x18556fb?, 0x7?}, {0x1aa9be0?, 0xc000705bd0?}, 0x1903898, {0xc00067f8c0, 0x4, ...})
2024-05-20T19:36:41.9361209Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0x10c
2024-05-20T19:36:41.9362511Z github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0xc0018b4c40?, {0x1ab0558?, 0xc0003c2460?}, 0xc000c36cc0?, {0xc00067f8c0?, 0xc000c36c90?, 0x184f117?})
2024-05-20T19:36:41.9364116Z    /home/runner/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220713235846-fda89e95df1e/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x66
2024-05-20T19:36:41.9365970Z go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1ab3c80, 0xc0010956f0}, {0x1abd1d8, 0xc001095710}, {0x1abd260, 0xc001095730}, {0x1ab7ea0, 0xc001095750}}, {0x1ab0558, 0xc0003c2460}, ...)
2024-05-20T19:36:41.9367534Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/compatibility/adapter.go:121 +0x95
2024-05-20T19:36:41.9368812Z go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0xc001586fc0, {0x1ab0558, 0xc0003c2460}, 0xc0017770d8?, {0xc00067f8c0, 0x4, 0x6})
2024-05-20T19:36:41.9370259Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/metrics/service_wrapper.go:213 +0x9d
2024-05-20T19:36:41.9371231Z go.uber.org/cadence/internal.(*activityTaskPoller).poll(0xc000fce500, {0x1ab0558, 0xc0003c2460})
2024-05-20T19:36:41.9372404Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1005 +0x33e
2024-05-20T19:36:41.9373458Z go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0xc000fce500, {0x1ab0558?, 0xc0003c2460?}, 0xc000453c70?)
2024-05-20T19:36:41.9374605Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1048 +0x42
2024-05-20T19:36:41.9375939Z go.uber.org/cadence/internal.(*activityTaskPoller).PollTask.(*activityTaskPoller).pollWithMetricsFunc.func1({0x1ab0558?, 0xc0003c2460?})
2024-05-20T19:36:41.9377169Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1042 +0x2e
2024-05-20T19:36:41.9377946Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9378803Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9379688Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15225
2024-05-20T19:36:41.9380651Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9381247Z 
2024-05-20T19:36:41.9381370Z goroutine 6402 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9382040Z go.temporal.io/sdk/internal.(*coroutineState).initialYield(0xc000890d20, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9382934Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:945 +0x5f
2024-05-20T19:36:41.9383685Z go.temporal.io/sdk/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9384303Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:955
2024-05-20T19:36:41.9385333Z go.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).Await(0x1ab08a0?, {0x1ab08a0, 0xc000a8d9e0}, 0xc000a8df80)
2024-05-20T19:36:41.9386256Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:364 +0xa5
2024-05-20T19:36:41.9387009Z go.temporal.io/sdk/internal.Await({0x1ab08a0, 0xc000a8d9e0}, 0xc000a8d9e0?)
2024-05-20T19:36:41.9387738Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:349 +0x58
2024-05-20T19:36:41.9388261Z go.temporal.io/sdk/workflow.Await(...)
2024-05-20T19:36:41.9388893Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/workflow/deterministic_wrappers.go:84
2024-05-20T19:36:41.9390043Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).Await(0xc0006beea0?, {0x1a9bb20?, 0xc0004cf470?}, 0x186e6a3?)
2024-05-20T19:36:41.9391120Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:203 +0x7e
2024-05-20T19:36:41.9392022Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func3({0x1a9bb20, 0xc0004cf470})
2024-05-20T19:36:41.9393025Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:99 +0x172
2024-05-20T19:36:41.9394053Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).GoNamed.func1({0x1ab08a0?, 0xc000a8d9e0})
2024-05-20T19:36:41.9395049Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:180 +0xfd
2024-05-20T19:36:41.9395863Z go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine.func1(0xc000ef6df0?)
2024-05-20T19:36:41.9396681Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1057 +0xcf
2024-05-20T19:36:41.9397509Z created by go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine in goroutine 6399
2024-05-20T19:36:41.9398347Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1048 +0x338
2024-05-20T19:36:41.9398884Z 
2024-05-20T19:36:41.9398983Z goroutine 14717 [sleep]:
2024-05-20T19:36:41.9399244Z time.Sleep(0x253e07cd)
2024-05-20T19:36:41.9399712Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/time.go:195 +0x125
2024-05-20T19:36:41.9400658Z go.uber.org/cadence/internal/common/backoff.(*ConcurrentRetrier).throttleInternal(0xc0011732c0)
2024-05-20T19:36:41.9401864Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:65 +0x8b
2024-05-20T19:36:41.9402894Z go.uber.org/cadence/internal/common/backoff.(*ConcurrentRetrier).Throttle(...)
2024-05-20T19:36:41.9403928Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:51
2024-05-20T19:36:41.9404802Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc00161d560)
2024-05-20T19:36:41.9405780Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:291 +0x21c
2024-05-20T19:36:41.9406720Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc00161d560)
2024-05-20T19:36:41.9407729Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9408660Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 14758
2024-05-20T19:36:41.9409692Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9410366Z 
2024-05-20T19:36:41.9410473Z goroutine 15570 [chan receive]:
2024-05-20T19:36:41.9411108Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000796540, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9412403Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9413243Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9414148Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9415013Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000a752f0}, 0xc000a758f0)
2024-05-20T19:36:41.9416127Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9416886Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9417743Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9419018Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc0008c9ea0?, {0x1a9bb20?, 0xc000ff3d00?}, 0x186e6a3?)
2024-05-20T19:36:41.9420089Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9421082Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func3({0x1a9bb20, 0xc000ff3d00})
2024-05-20T19:36:41.9421996Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:99 +0x172
2024-05-20T19:36:41.9422951Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000a752f0})
2024-05-20T19:36:41.9424117Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9424989Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000882d90?)
2024-05-20T19:36:41.9426065Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9427013Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 15556
2024-05-20T19:36:41.9428076Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9428696Z 
2024-05-20T19:36:41.9428803Z goroutine 15219 [select]:
2024-05-20T19:36:41.9429308Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce500, {0xe0?, 0x53?}, 0xc000858378)
2024-05-20T19:36:41.9430333Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9431184Z go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0xc000fce500)
2024-05-20T19:36:41.9432416Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1071 +0xb5
2024-05-20T19:36:41.9433248Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fc9b00)
2024-05-20T19:36:41.9434157Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9434972Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fc9b00)
2024-05-20T19:36:41.9435859Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9436705Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9437644Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9438227Z 
2024-05-20T19:36:41.9438335Z goroutine 15572 [chan receive]:
2024-05-20T19:36:41.9438931Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000796600, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9439972Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9440722Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9441521Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9442436Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000a75440}, 0xc000a75950)
2024-05-20T19:36:41.9443305Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9443950Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9444700Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9445802Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc00175c698?, {0x1a9bb20?, 0xc000ff3d40?}, 0x185a519?)
2024-05-20T19:36:41.9446786Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9447621Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func5({0x1a9bb20, 0xc000ff3d40})
2024-05-20T19:36:41.9448569Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:143 +0x1a9
2024-05-20T19:36:41.9449447Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000a75440})
2024-05-20T19:36:41.9450345Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9451106Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc0015ac820?)
2024-05-20T19:36:41.9452084Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9453228Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 15556
2024-05-20T19:36:41.9454242Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9454822Z 
2024-05-20T19:36:41.9454943Z goroutine 6420 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9455730Z go.temporal.io/sdk/internal.(*coroutineState).initialYield(0xc0005d7ae0, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9456534Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:945 +0x5f
2024-05-20T19:36:41.9459042Z go.temporal.io/sdk/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9459690Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:955
2024-05-20T19:36:41.9460558Z go.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).Await(0x1ab08a0?, {0x1ab08a0, 0xc000ec1290}, 0xc0005d7d60)
2024-05-20T19:36:41.9461411Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:364 +0xa5
2024-05-20T19:36:41.9462162Z go.temporal.io/sdk/internal.Await({0x1ab08a0, 0xc000ec1290}, 0xc000ec1290?)
2024-05-20T19:36:41.9462828Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:349 +0x58
2024-05-20T19:36:41.9463337Z go.temporal.io/sdk/workflow.Await(...)
2024-05-20T19:36:41.9463884Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/workflow/deterministic_wrappers.go:84
2024-05-20T19:36:41.9464834Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).Await(0x0?, {0x1a9bb20?, 0xc0000ef840?}, 0x18383c8?)
2024-05-20T19:36:41.9465819Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:203 +0x7e
2024-05-20T19:36:41.9466636Z github.com/indeedeng/iwf/service/interpreter.executeState.func2({0x1a9bb20, 0xc0000ef840})
2024-05-20T19:36:41.9467374Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:579 +0x264
2024-05-20T19:36:41.9468236Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).GoNamed.func1({0x1ab08a0?, 0xc000ec1290})
2024-05-20T19:36:41.9469153Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:180 +0xfd
2024-05-20T19:36:41.9469925Z go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine.func1(0xc00046d060?)
2024-05-20T19:36:41.9470666Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1057 +0xcf
2024-05-20T19:36:41.9471421Z created by go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine in goroutine 6406
2024-05-20T19:36:41.9472290Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1048 +0x338
2024-05-20T19:36:41.9472749Z 
2024-05-20T19:36:41.9472851Z goroutine 15205 [select]:
2024-05-20T19:36:41.9473336Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000d47980, {0x1?, 0x0?}, 0xc000c18c60)
2024-05-20T19:36:41.9474373Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9475224Z go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0xc000d47980)
2024-05-20T19:36:41.9476162Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:298 +0x65
2024-05-20T19:36:41.9477140Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc480)
2024-05-20T19:36:41.9478054Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9478965Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc480)
2024-05-20T19:36:41.9479881Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9480728Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9481842Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9482608Z 
2024-05-20T19:36:41.9482715Z goroutine 15317 [IO wait]:
2024-05-20T19:36:41.9483076Z internal/poll.runtime_pollWait(0x7fd78727ead0, 0x72)
2024-05-20T19:36:41.9483601Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/netpoll.go:343 +0x85
2024-05-20T19:36:41.9484168Z internal/poll.(*pollDesc).wait(0xc000e83e80?, 0xc000394000?, 0x0)
2024-05-20T19:36:41.9484785Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2024-05-20T19:36:41.9485316Z internal/poll.(*pollDesc).waitRead(...)
2024-05-20T19:36:41.9485801Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-20T19:36:41.9486505Z internal/poll.(*FD).Read(0xc000e83e80, {0xc000394000, 0x1000, 0x1000})
2024-05-20T19:36:41.9487112Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_unix.go:164 +0x27a
2024-05-20T19:36:41.9487701Z net.(*netFD).Read(0xc000e83e80, {0xc000394000?, 0x7b8820?, 0x26c9ae0?})
2024-05-20T19:36:41.9488246Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/fd_posix.go:55 +0x25
2024-05-20T19:36:41.9488838Z net.(*conn).Read(0xc000ba8348, {0xc000394000?, 0x0?, 0xc00040da00?})
2024-05-20T19:36:41.9489351Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/net.go:179 +0x45
2024-05-20T19:36:41.9489947Z net/http.(*persistConn).Read(0xc0005fbb00, {0xc000394000?, 0xc001593260?, 0xc000d63d38?})
2024-05-20T19:36:41.9490598Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:1954 +0x4a
2024-05-20T19:36:41.9491075Z bufio.(*Reader).fill(0xc000608ae0)
2024-05-20T19:36:41.9491482Z    /opt/hostedtoolcache/go/1.21.10/x64/src/bufio/bufio.go:113 +0x103
2024-05-20T19:36:41.9491929Z bufio.(*Reader).Peek(0xc000608ae0, 0x1)
2024-05-20T19:36:41.9492366Z    /opt/hostedtoolcache/go/1.21.10/x64/src/bufio/bufio.go:151 +0x53
2024-05-20T19:36:41.9492829Z net/http.(*persistConn).readLoop(0xc0005fbb00)
2024-05-20T19:36:41.9493328Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:2118 +0x1b9
2024-05-20T19:36:41.9493890Z created by net/http.(*Transport).dialConn in goroutine 15265
2024-05-20T19:36:41.9494451Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/transport.go:1776 +0x169f
2024-05-20T19:36:41.9494833Z 
2024-05-20T19:36:41.9494959Z goroutine 6555 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9495695Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc0000f0640, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9496898Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9497787Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9498743Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9499664Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc0011770b0}, 0xc0011775c0)
2024-05-20T19:36:41.9500571Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9501283Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9502064Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9503274Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x7fd73c35eb38?, {0x1a9bb20?, 0xc000a91370?}, 0x186e6c3?)
2024-05-20T19:36:41.9504278Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9505120Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func4({0x1a9bb20, 0xc000a91370})
2024-05-20T19:36:41.9505906Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:123 +0x10a
2024-05-20T19:36:41.9506893Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc0011770b0})
2024-05-20T19:36:41.9507905Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9508790Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000edd830?)
2024-05-20T19:36:41.9509773Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9510685Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6551
2024-05-20T19:36:41.9511696Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9512271Z 
2024-05-20T19:36:41.9512379Z goroutine 15216 [select]:
2024-05-20T19:36:41.9512873Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce580, {0xe0?, 0x53?}, 0xc001756438)
2024-05-20T19:36:41.9514093Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9515052Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask(0xc000fce580)
2024-05-20T19:36:41.9516099Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1142 +0xb5
2024-05-20T19:36:41.9517013Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc000)
2024-05-20T19:36:41.9517924Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9518734Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc000)
2024-05-20T19:36:41.9519619Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9520474Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9521426Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9522010Z 
2024-05-20T19:36:41.9522109Z goroutine 15433 [select]:
2024-05-20T19:36:41.9522576Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTunnel).getTask(...)
2024-05-20T19:36:41.9523517Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:193
2024-05-20T19:36:41.9524641Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).pollLocallyDispatchedActivity(0xc000fce580, {0xc00037c710?, 0x26b40f0?})
2024-05-20T19:36:41.9525872Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1151 +0x8b
2024-05-20T19:36:41.9526989Z go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0xc000fce580, {0x1ab0558?, 0xc00043aee0?}, 0x3?)
2024-05-20T19:36:41.9528107Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1048 +0x42
2024-05-20T19:36:41.9529310Z go.uber.org/cadence/internal.(*locallyDispatchedActivityTaskPoller).PollTask.(*activityTaskPoller).pollWithMetricsFunc.func1({0x1ab0558?, 0xc00043aee0?})
2024-05-20T19:36:41.9530716Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1042 +0x2e
2024-05-20T19:36:41.9531506Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9532374Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9533240Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15238
2024-05-20T19:36:41.9534208Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9534804Z 
2024-05-20T19:36:41.9534902Z goroutine 15444 [IO wait]:
2024-05-20T19:36:41.9535394Z internal/poll.runtime_pollWait(0x7fd78727eeb0, 0x72)
2024-05-20T19:36:41.9535908Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/netpoll.go:343 +0x85
2024-05-20T19:36:41.9536664Z internal/poll.(*pollDesc).wait(0xc000e82700?, 0xc0010b45b1?, 0x0)
2024-05-20T19:36:41.9537539Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2024-05-20T19:36:41.9538070Z internal/poll.(*pollDesc).waitRead(...)
2024-05-20T19:36:41.9538562Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-20T19:36:41.9539145Z internal/poll.(*FD).Read(0xc000e82700, {0xc0010b45b1, 0x1, 0x1})
2024-05-20T19:36:41.9539710Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_unix.go:164 +0x27a
2024-05-20T19:36:41.9540314Z net.(*netFD).Read(0xc000e82700, {0xc0010b45b1?, 0xc00092bf40?, 0x470930?})
2024-05-20T19:36:41.9540874Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/fd_posix.go:55 +0x25
2024-05-20T19:36:41.9541483Z net.(*conn).Read(0xc000ba80c8, {0xc0010b45b1?, 0x0?, 0xc0010b45a8?})
2024-05-20T19:36:41.9542006Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/net.go:179 +0x45
2024-05-20T19:36:41.9542468Z net/http.(*connReader).backgroundRead(0xc0010b45a0)
2024-05-20T19:36:41.9542972Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:683 +0x37
2024-05-20T19:36:41.9543557Z created by net/http.(*connReader).startBackgroundRead in goroutine 15247
2024-05-20T19:36:41.9544224Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:679 +0xba
2024-05-20T19:36:41.9544571Z 
2024-05-20T19:36:41.9544675Z goroutine 15166 [IO wait]:
2024-05-20T19:36:41.9545009Z internal/poll.runtime_pollWait(0x7fd78727e5f8, 0x72)
2024-05-20T19:36:41.9545516Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/netpoll.go:343 +0x85
2024-05-20T19:36:41.9546062Z internal/poll.(*pollDesc).wait(0xc000d25900?, 0xc000746000?, 0x0)
2024-05-20T19:36:41.9546665Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
2024-05-20T19:36:41.9547189Z internal/poll.(*pollDesc).waitRead(...)
2024-05-20T19:36:41.9547677Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_poll_runtime.go:89
2024-05-20T19:36:41.9548271Z internal/poll.(*FD).Read(0xc000d25900, {0xc000746000, 0x8000, 0x8000})
2024-05-20T19:36:41.9548862Z    /opt/hostedtoolcache/go/1.21.10/x64/src/internal/poll/fd_unix.go:164 +0x27a
2024-05-20T19:36:41.9549637Z net.(*netFD).Read(0xc000d25900, {0xc000746000?, 0x5010100000001?, 0xc00082ed78?})
2024-05-20T19:36:41.9550312Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/fd_posix.go:55 +0x25
2024-05-20T19:36:41.9550839Z net.(*conn).Read(0xc000ba9f78, {0xc000746000?, 0xc000c880a0?, 0x0?})
2024-05-20T19:36:41.9551356Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/net.go:179 +0x45
2024-05-20T19:36:41.9551879Z bufio.(*Reader).Read(0xc00189a060, {0xc000302040, 0x9, 0xc300001bc2?})
2024-05-20T19:36:41.9552541Z    /opt/hostedtoolcache/go/1.21.10/x64/src/bufio/bufio.go:244 +0x197
2024-05-20T19:36:41.9553104Z io.ReadAtLeast({0x1a9ae40, 0xc00189a060}, {0xc000302040, 0x9, 0x9}, 0x9)
2024-05-20T19:36:41.9553636Z    /opt/hostedtoolcache/go/1.21.10/x64/src/io/io.go:335 +0x90
2024-05-20T19:36:41.9554022Z io.ReadFull(...)
2024-05-20T19:36:41.9554323Z    /opt/hostedtoolcache/go/1.21.10/x64/src/io/io.go:354
2024-05-20T19:36:41.9554969Z golang.org/x/net/http2.readFrameHeader({0xc000302040, 0x9, 0xc000c88090?}, {0x1a9ae40?, 0xc00189a060?})
2024-05-20T19:36:41.9555708Z    /home/runner/go/pkg/mod/golang.org/x/net@v0.21.0/http2/frame.go:237 +0x65
2024-05-20T19:36:41.9556263Z golang.org/x/net/http2.(*Framer).ReadFrame(0xc000302000)
2024-05-20T19:36:41.9556805Z    /home/runner/go/pkg/mod/golang.org/x/net@v0.21.0/http2/frame.go:498 +0x85
2024-05-20T19:36:41.9557509Z google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000512000, 0xc000835320?)
2024-05-20T19:36:41.9558354Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/http2_client.go:1595 +0x245
2024-05-20T19:36:41.9559185Z created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 15190
2024-05-20T19:36:41.9560023Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/http2_client.go:397 +0x1e9e
2024-05-20T19:36:41.9560525Z 
2024-05-20T19:36:41.9560619Z goroutine 15472 [select]:
2024-05-20T19:36:41.9561146Z google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000b265a0)
2024-05-20T19:36:41.9561920Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:331 +0x7c
2024-05-20T19:36:41.9562643Z google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
2024-05-20T19:36:41.9563343Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:346
2024-05-20T19:36:41.9564149Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc000ae7450, {0x155e160?, 0xc0004fc300}, 0xc0009c4558?)
2024-05-20T19:36:41.9564879Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1066 +0xc9
2024-05-20T19:36:41.9565470Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2712020?)
2024-05-20T19:36:41.9566124Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:917 +0x1f
2024-05-20T19:36:41.9566818Z google.golang.org/grpc.(*clientStream).withRetry(0xc000b26360, 0xc0009c4588, 0xc0009c4558)
2024-05-20T19:36:41.9567507Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:768 +0x13a
2024-05-20T19:36:41.9568191Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc000b26360, {0x155e160?, 0xc0004fc300?})
2024-05-20T19:36:41.9568940Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:916 +0x12e
2024-05-20T19:36:41.9569961Z google.golang.org/grpc.invoke({0x1ab04b0?, 0xc000f18180?}, {0xc0008fce40?, 0x412a05?}, {0x155f420, 0xc0004fc348}, {0x155e160, 0xc0004fc300}, 0x412d85?, {0xc001639100, ...})
2024-05-20T19:36:41.9570886Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:78 +0xcb
2024-05-20T19:36:41.9571865Z google.golang.org/grpc.(*ClientConn).Invoke(0xc000814800, {0x1ab04b0, 0xc000f18180}, {0xc0008fce40, 0x32}, {0x155f420, 0xc0004fc348}, {0x155e160, 0xc0004fc300}, {0xc000842c10, ...})
2024-05-20T19:36:41.9572843Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:42 +0x2cb
2024-05-20T19:36:41.9573819Z go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0xc0013b6ab0, {0x1ab0558, 0xc0004ca000}, 0xc000a34d80, 0x1579640?, 0xc0015848b0, {0x76?, 0xc001019388?, 0x26e0980?})
2024-05-20T19:36:41.9574850Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x4e5
2024-05-20T19:36:41.9575772Z go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0xc0013b6ab0, {0x1ab0558, 0xc0004ca000}, 0xc000a34d80)
2024-05-20T19:36:41.9576566Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0x126
2024-05-20T19:36:41.9577609Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558?, 0xc0004ca000?}, 0x184f9bf?)
2024-05-20T19:36:41.9578794Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
2024-05-20T19:36:41.9579811Z go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc00183bc80, {0x1ab0558, 0xc0004ca000}, 0xc000a34d80, {0x1ab4e00, 0xc000f18090})
2024-05-20T19:36:41.9580810Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0x162
2024-05-20T19:36:41.9581905Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc0004ca000}, 0xc0013b6ab0?)
2024-05-20T19:36:41.9582991Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9584045Z go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x17706e0?, {0x1ab0558, 0xc0004ca000}, 0xc000a34d80, {0x1ab4e00?, 0xc000f18060})
2024-05-20T19:36:41.9585145Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xa8
2024-05-20T19:36:41.9586449Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x1, 0x1}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc0004ca000}, 0xc000070c00?)
2024-05-20T19:36:41.9587550Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9588389Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
2024-05-20T19:36:41.9589081Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
2024-05-20T19:36:41.9589809Z go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
2024-05-20T19:36:41.9590486Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
2024-05-20T19:36:41.9591509Z go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x1ab4c80?, 0xc0013b6c30?}, {0x7fd73c34b008?, 0xc0013b6c30?}}, {0x1ab0558, 0xc0004ca000}, 0xc000b1f590?)
2024-05-20T19:36:41.9592607Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x92
2024-05-20T19:36:41.9593789Z go.uber.org/yarpc/encoding/protobuf.(*client).Call(0xc0013b7260, {0x1ab0558?, 0xc0004ca000?}, {0x18556e8?, 0x4c?}, {0x1aa9d00?, 0xc000d7a420?}, 0x19038a0, {0xc000d7a3c0, 0x4, ...})
2024-05-20T19:36:41.9594851Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0x10c
2024-05-20T19:36:41.9596245Z github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForDecisionTask(0xc0004d0940?, {0x1ab0558?, 0xc0004ca000?}, 0xc000cfbd20?, {0xc000d7a3c0?, 0xc000cfbcf0?, 0x184f117?})
2024-05-20T19:36:41.9598420Z    /home/runner/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220713235846-fda89e95df1e/go/proto/api/v1/service_worker.pb.yarpc.go:361 +0x66
2024-05-20T19:36:41.9601134Z go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForDecisionTask({{0x1ab3c80, 0xc0010956f0}, {0x1abd1d8, 0xc001095710}, {0x1abd260, 0xc001095730}, {0x1ab7ea0, 0xc001095750}}, {0x1ab0558, 0xc0004ca000}, ...)
2024-05-20T19:36:41.9603034Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/compatibility/adapter.go:126 +0x95
2024-05-20T19:36:41.9604356Z go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForDecisionTask(0xc001586fc0, {0x1ab0558, 0xc0004ca000}, 0xc0004d0740?, {0xc000d7a3c0, 0x4, 0x6})
2024-05-20T19:36:41.9605776Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/metrics/service_wrapper.go:220 +0x9d
2024-05-20T19:36:41.9606763Z go.uber.org/cadence/internal.(*workflowTaskPoller).poll(0xc000d47980, {0x1ab0558, 0xc0004ca000})
2024-05-20T19:36:41.9607796Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:766 +0x25d
2024-05-20T19:36:41.9608579Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9609577Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9610452Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15210
2024-05-20T19:36:41.9611417Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9612014Z 
2024-05-20T19:36:41.9612119Z goroutine 15571 [chan receive]:
2024-05-20T19:36:41.9612663Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000796580, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9613667Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9614411Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9615397Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9616216Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000a75380}, 0xc000a75920)
2024-05-20T19:36:41.9617101Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9617752Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9618518Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9619766Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x7fd73c2b05c0?, {0x1a9bb20?, 0xc000ff3d20?}, 0x186e6c3?)
2024-05-20T19:36:41.9620764Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9621602Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func4({0x1a9bb20, 0xc000ff3d20})
2024-05-20T19:36:41.9622386Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:123 +0x10a
2024-05-20T19:36:41.9623251Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000a75380})
2024-05-20T19:36:41.9624153Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9625012Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc001817630?)
2024-05-20T19:36:41.9626005Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9626910Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 15556
2024-05-20T19:36:41.9627997Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9628582Z 
2024-05-20T19:36:41.9628686Z goroutine 15521 [chan receive]:
2024-05-20T19:36:41.9629228Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000796500, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9630216Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9630971Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9631980Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9632940Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000a75230}, 0xc000a758c0)
2024-05-20T19:36:41.9633946Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9634692Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9635569Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9636733Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x10?, {0x1a9bb20?, 0xc000ff3ce0?}, 0x1865c58?)
2024-05-20T19:36:41.9637868Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9638948Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func2({0x1a9bb20, 0xc000ff3ce0})
2024-05-20T19:36:41.9639823Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:75 +0x1c4
2024-05-20T19:36:41.9640774Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000a75230})
2024-05-20T19:36:41.9641829Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9642708Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc001427110?)
2024-05-20T19:36:41.9643727Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9644746Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 15556
2024-05-20T19:36:41.9645948Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9646532Z 
2024-05-20T19:36:41.9646691Z goroutine 15523 [select]:
2024-05-20T19:36:41.9647179Z google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000bdb200)
2024-05-20T19:36:41.9648057Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:331 +0x7c
2024-05-20T19:36:41.9648888Z google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
2024-05-20T19:36:41.9649817Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:346
2024-05-20T19:36:41.9650787Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc001549c70, {0x155e160?, 0xc00139ed38}, 0xc00083a4f8?)
2024-05-20T19:36:41.9651591Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1066 +0xc9
2024-05-20T19:36:41.9652219Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2712020?)
2024-05-20T19:36:41.9652915Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:917 +0x1f
2024-05-20T19:36:41.9653696Z google.golang.org/grpc.(*clientStream).withRetry(0xc000bdafc0, 0xc00083a528, 0xc00083a4f8)
2024-05-20T19:36:41.9654481Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:768 +0x13a
2024-05-20T19:36:41.9655558Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc000bdafc0, {0x155e160?, 0xc00139ed38?})
2024-05-20T19:36:41.9656544Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:916 +0x12e
2024-05-20T19:36:41.9657799Z google.golang.org/grpc.invoke({0x1ab04b0?, 0xc000bd3cb0?}, {0xc000ea8b40?, 0x412a05?}, {0x155f420, 0xc00139ed80}, {0x155e160, 0xc00139ed38}, 0x412d85?, {0xc000f06700, ...})
2024-05-20T19:36:41.9658859Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:78 +0xcb
2024-05-20T19:36:41.9659937Z google.golang.org/grpc.(*ClientConn).Invoke(0xc000814800, {0x1ab04b0, 0xc000bd3cb0}, {0xc000ea8b40, 0x32}, {0x155f420, 0xc00139ed80}, {0x155e160, 0xc00139ed38}, {0xc0011b7310, ...})
2024-05-20T19:36:41.9661106Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:42 +0x2cb
2024-05-20T19:36:41.9662242Z go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0xc0013b6ab0, {0x1ab0558, 0xc0004fa460}, 0xc000bdd680, 0x1838f92?, 0xc000e78230, {0xc18b07726dd78d61?, 0xc001018ea8?, 0x26e0980?})
2024-05-20T19:36:41.9663414Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x4e5
2024-05-20T19:36:41.9664224Z go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0xc0013b6ab0, {0x1ab0558, 0xc0004fa460}, 0xc000bdd680)
2024-05-20T19:36:41.9665117Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0x126
2024-05-20T19:36:41.9666352Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558?, 0xc0004fa460?}, 0x184f9bf?)
2024-05-20T19:36:41.9667545Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
2024-05-20T19:36:41.9668732Z go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc00183bc80, {0x1ab0558, 0xc0004fa460}, 0xc000bdd680, {0x1ab4e00, 0xc000bd3bc0})
2024-05-20T19:36:41.9669989Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0x162
2024-05-20T19:36:41.9671290Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc0004fa460}, 0xc0013b6ab0?)
2024-05-20T19:36:41.9672573Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9673726Z go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x17706e0?, {0x1ab0558, 0xc0004fa460}, 0xc000bdd680, {0x1ab4e00?, 0xc000bd3b90})
2024-05-20T19:36:41.9675034Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xa8
2024-05-20T19:36:41.9676274Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x1, 0x1}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc0004fa460}, 0xc0000a0000?)
2024-05-20T19:36:41.9677565Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9678396Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
2024-05-20T19:36:41.9679088Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
2024-05-20T19:36:41.9679816Z go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
2024-05-20T19:36:41.9680710Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
2024-05-20T19:36:41.9681923Z go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x1ab4c80?, 0xc0013b6c30?}, {0x7fd73c34b008?, 0xc0013b6c30?}}, {0x1ab0558, 0xc0004fa460}, 0xc00044c000?)
2024-05-20T19:36:41.9683106Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x92
2024-05-20T19:36:41.9684216Z go.uber.org/yarpc/encoding/protobuf.(*client).Call(0xc0013b7260, {0x1ab0558?, 0xc0004fa460?}, {0x18556fb?, 0x7?}, {0x1aa9be0?, 0xc00052c5a0?}, 0x1903898, {0xc000f512c0, 0x4, ...})
2024-05-20T19:36:41.9685262Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0x10c
2024-05-20T19:36:41.9686669Z github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForActivityTask(0xc00109b8c0?, {0x1ab0558?, 0xc0004fa460?}, 0xc0009bdcc0?, {0xc000f512c0?, 0xc0009bdc90?, 0x184f117?})
2024-05-20T19:36:41.9688167Z    /home/runner/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220713235846-fda89e95df1e/go/proto/api/v1/service_worker.pb.yarpc.go:397 +0x66
2024-05-20T19:36:41.9689975Z go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForActivityTask({{0x1ab3c80, 0xc0010956f0}, {0x1abd1d8, 0xc001095710}, {0x1abd260, 0xc001095730}, {0x1ab7ea0, 0xc001095750}}, {0x1ab0558, 0xc0004fa460}, ...)
2024-05-20T19:36:41.9691765Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/compatibility/adapter.go:121 +0x95
2024-05-20T19:36:41.9693213Z go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForActivityTask(0xc001586fc0, {0x1ab0558, 0xc0004fa460}, 0x6?, {0xc000f512c0, 0x4, 0x6})
2024-05-20T19:36:41.9694799Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/metrics/service_wrapper.go:213 +0x9d
2024-05-20T19:36:41.9696114Z go.uber.org/cadence/internal.(*activityTaskPoller).poll(0xc000fce500, {0x1ab0558, 0xc0004fa460})
2024-05-20T19:36:41.9697348Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1005 +0x33e
2024-05-20T19:36:41.9698501Z go.uber.org/cadence/internal.(*activityTaskPoller).pollWithMetrics(0xc000fce500, {0x1ab0558?, 0xc0004fa460?}, 0xc00044c000?)
2024-05-20T19:36:41.9699848Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1048 +0x42
2024-05-20T19:36:41.9701056Z go.uber.org/cadence/internal.(*activityTaskPoller).PollTask.(*activityTaskPoller).pollWithMetricsFunc.func1({0x1ab0558?, 0xc0004fa460?})
2024-05-20T19:36:41.9702628Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1042 +0x2e
2024-05-20T19:36:41.9703411Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9704280Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9705150Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15223
2024-05-20T19:36:41.9706123Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9706720Z 
2024-05-20T19:36:41.9706841Z goroutine 6567 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9707419Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000a64700, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9708425Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9709177Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9709981Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9710885Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc0013a15c0}, 0xc000bca410)
2024-05-20T19:36:41.9712030Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9712744Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9713659Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9714897Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc000fa4f00?, {0x1a9bb20?, 0xc000d38510?}, 0x18383c8?)
2024-05-20T19:36:41.9716010Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9716957Z github.com/indeedeng/iwf/service/interpreter.executeState.func2({0x1a9bb20, 0xc000d38510})
2024-05-20T19:36:41.9717891Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:579 +0x264
2024-05-20T19:36:41.9718929Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc0013a15c0})
2024-05-20T19:36:41.9719952Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9720959Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000d38280?)
2024-05-20T19:36:41.9722115Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9723134Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6558
2024-05-20T19:36:41.9724331Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9724994Z 
2024-05-20T19:36:41.9725098Z goroutine 15203 [select]:
2024-05-20T19:36:41.9725604Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000d47980, {0x1?, 0x0?}, 0xc0007ddf80)
2024-05-20T19:36:41.9726772Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9727715Z go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0xc000d47980)
2024-05-20T19:36:41.9728753Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:298 +0x65
2024-05-20T19:36:41.9729823Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc480)
2024-05-20T19:36:41.9730748Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9731590Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc480)
2024-05-20T19:36:41.9732596Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9733442Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9734392Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9734966Z 
2024-05-20T19:36:41.9735073Z goroutine 15497 [select]:
2024-05-20T19:36:41.9735667Z google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000c77320)
2024-05-20T19:36:41.9736447Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:331 +0x7c
2024-05-20T19:36:41.9737185Z google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
2024-05-20T19:36:41.9737893Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:346
2024-05-20T19:36:41.9738695Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc000bac340, {0x155e160?, 0xc000f708d0}, 0xc000b7c558?)
2024-05-20T19:36:41.9739432Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1066 +0xc9
2024-05-20T19:36:41.9740035Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2712020?)
2024-05-20T19:36:41.9740625Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:917 +0x1f
2024-05-20T19:36:41.9741314Z google.golang.org/grpc.(*clientStream).withRetry(0xc000c770e0, 0xc000b7c588, 0xc000b7c558)
2024-05-20T19:36:41.9742142Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:768 +0x13a
2024-05-20T19:36:41.9742829Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc000c770e0, {0x155e160?, 0xc000f708d0?})
2024-05-20T19:36:41.9743507Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:916 +0x12e
2024-05-20T19:36:41.9744471Z google.golang.org/grpc.invoke({0x1ab04b0?, 0xc000ef5500?}, {0xc0006f2a40?, 0x412a05?}, {0x155f420, 0xc000f70918}, {0x155e160, 0xc000f708d0}, 0x412d85?, {0xc000e12580, ...})
2024-05-20T19:36:41.9745403Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:78 +0xcb
2024-05-20T19:36:41.9746451Z google.golang.org/grpc.(*ClientConn).Invoke(0xc000814800, {0x1ab04b0, 0xc000ef5500}, {0xc0006f2a40, 0x32}, {0x155f420, 0xc000f70918}, {0x155e160, 0xc000f708d0}, {0xc0016940f0, ...})
2024-05-20T19:36:41.9747436Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:42 +0x2cb
2024-05-20T19:36:41.9748424Z go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0xc0013b6ab0, {0x1ab0558, 0xc000452af0}, 0xc000c552c0, 0x1500222?, 0xc001776d00, {0x16e6b60?, 0xc001019388?, 0x26e0980?})
2024-05-20T19:36:41.9749528Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x4e5
2024-05-20T19:36:41.9750331Z go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0xc0013b6ab0, {0x1ab0558, 0xc000452af0}, 0xc000c552c0)
2024-05-20T19:36:41.9751303Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0x126
2024-05-20T19:36:41.9752371Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558?, 0xc000452af0?}, 0x184f9bf?)
2024-05-20T19:36:41.9753467Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
2024-05-20T19:36:41.9753983Z go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc00183bc80, {0x1ab0558, 0xc000452af0}, 0xc000c552c0, {0x1ab4e00, 0xc000ef5410})
2024-05-20T19:36:41.9754353Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0x162
2024-05-20T19:36:41.9754970Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc000452af0}, 0xc0013b6ab0?)
2024-05-20T19:36:41.9755331Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9755903Z go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x17706e0?, {0x1ab0558, 0xc000452af0}, 0xc000c552c0, {0x1ab4e00?, 0xc000ef53e0})
2024-05-20T19:36:41.9756439Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xa8
2024-05-20T19:36:41.9757057Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x1, 0x1}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc000452af0}, 0xc000814c00?)
2024-05-20T19:36:41.9757429Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9757677Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
2024-05-20T19:36:41.9758002Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
2024-05-20T19:36:41.9758278Z go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
2024-05-20T19:36:41.9758559Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
2024-05-20T19:36:41.9759160Z go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x1ab4c80?, 0xc0013b6c30?}, {0x7fd73c34b008?, 0xc0013b6c30?}}, {0x1ab0558, 0xc000452af0}, 0x0?)
2024-05-20T19:36:41.9759541Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x92
2024-05-20T19:36:41.9760162Z go.uber.org/yarpc/encoding/protobuf.(*client).Call(0xc0013b7260, {0x1ab0558?, 0xc000452af0?}, {0x18556e8?, 0x4c?}, {0x1aa9d00?, 0xc0018a7b60?}, 0x19038a0, {0xc0018a7b00, 0x4, ...})
2024-05-20T19:36:41.9760558Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0x10c
2024-05-20T19:36:41.9761431Z github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkerAPIYARPCCaller).PollForDecisionTask(0xc001563b20?, {0x1ab0558?, 0xc000452af0?}, 0xc000c36d20?, {0xc0018a7b00?, 0xc000c36cf0?, 0x184f117?})
2024-05-20T19:36:41.9762109Z    /home/runner/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220713235846-fda89e95df1e/go/proto/api/v1/service_worker.pb.yarpc.go:361 +0x66
2024-05-20T19:36:41.9762965Z go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.PollForDecisionTask({{0x1ab3c80, 0xc0010956f0}, {0x1abd1d8, 0xc001095710}, {0x1abd260, 0xc001095730}, {0x1ab7ea0, 0xc001095750}}, {0x1ab0558, 0xc000452af0}, ...)
2024-05-20T19:36:41.9763602Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/compatibility/adapter.go:126 +0x95
2024-05-20T19:36:41.9764304Z go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).PollForDecisionTask(0xc001586fc0, {0x1ab0558, 0xc000452af0}, 0xc001563540?, {0xc0018a7b00, 0x4, 0x6})
2024-05-20T19:36:41.9764976Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/metrics/service_wrapper.go:220 +0x9d
2024-05-20T19:36:41.9765327Z go.uber.org/cadence/internal.(*workflowTaskPoller).poll(0xc000d47980, {0x1ab0558, 0xc000452af0})
2024-05-20T19:36:41.9765893Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:766 +0x25d
2024-05-20T19:36:41.9766081Z go.uber.org/cadence/internal.(*basePoller).doPoll.func1()
2024-05-20T19:36:41.9766643Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:256 +0x4a
2024-05-20T19:36:41.9766921Z created by go.uber.org/cadence/internal.(*basePoller).doPoll in goroutine 15203
2024-05-20T19:36:41.9767483Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:255 +0x1b7
2024-05-20T19:36:41.9767500Z 
2024-05-20T19:36:41.9767605Z goroutine 11473 [chan receive]:
2024-05-20T19:36:41.9767944Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc00060d880, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9768485Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9768668Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9769174Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9769502Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000ae5aa0}, 0xc000ae5f20)
2024-05-20T19:36:41.9770034Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9770159Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9770688Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9771214Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x1abd260?, {0x1a9bb20?, 0xc001190ce0?}, 0x185a519?)
2024-05-20T19:36:41.9771560Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9771935Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func5({0x1a9bb20, 0xc001190ce0})
2024-05-20T19:36:41.9772221Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:143 +0x1a9
2024-05-20T19:36:41.9772684Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000ae5aa0})
2024-05-20T19:36:41.9773018Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9773326Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc0010e3030?)
2024-05-20T19:36:41.9773862Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9774282Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 11468
2024-05-20T19:36:41.9774834Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9774839Z 
2024-05-20T19:36:41.9774936Z goroutine 15225 [select]:
2024-05-20T19:36:41.9775414Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000fce500, {0xe0?, 0x53?}, 0xc000f71e60)
2024-05-20T19:36:41.9776000Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9776339Z go.uber.org/cadence/internal.(*activityTaskPoller).PollTask(0xc000fce500)
2024-05-20T19:36:41.9776912Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:1071 +0xb5
2024-05-20T19:36:41.9777132Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fc9b00)
2024-05-20T19:36:41.9777758Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9777990Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fc9b00)
2024-05-20T19:36:41.9778539Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9778818Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9779367Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9779372Z 
2024-05-20T19:36:41.9779470Z goroutine 15518 [select]:
2024-05-20T19:36:41.9779758Z google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000cd2480)
2024-05-20T19:36:41.9780128Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:331 +0x7c
2024-05-20T19:36:41.9780371Z google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
2024-05-20T19:36:41.9780718Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/internal/transport/transport.go:346
2024-05-20T19:36:41.9781058Z google.golang.org/grpc.(*csAttempt).recvMsg(0xc0005f3e10, {0x155e160?, 0xc0007d0018}, 0xc0009999c0?)
2024-05-20T19:36:41.9781332Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:1066 +0xc9
2024-05-20T19:36:41.9781541Z google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2712020?)
2024-05-20T19:36:41.9781877Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:917 +0x1f
2024-05-20T19:36:41.9782186Z google.golang.org/grpc.(*clientStream).withRetry(0xc000cd2240, 0xc0009999f0, 0xc0009999c0)
2024-05-20T19:36:41.9782453Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:768 +0x13a
2024-05-20T19:36:41.9782749Z google.golang.org/grpc.(*clientStream).RecvMsg(0xc000cd2240, {0x155e160?, 0xc0007d0018?})
2024-05-20T19:36:41.9783021Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/stream.go:916 +0x12e
2024-05-20T19:36:41.9783596Z google.golang.org/grpc.invoke({0x1ab04b0?, 0xc000a74c00?}, {0xc001821d10?, 0x412a05?}, {0x155f420, 0xc0007d00c0}, {0x155e160, 0xc0007d0018}, 0x412d85?, {0xc000796240, ...})
2024-05-20T19:36:41.9783863Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:78 +0xcb
2024-05-20T19:36:41.9784483Z google.golang.org/grpc.(*ClientConn).Invoke(0xc000814800, {0x1ab04b0, 0xc000a74c00}, {0xc001821d10, 0x2e}, {0x155f420, 0xc0007d00c0}, {0x155e160, 0xc0007d0018}, {0xc000ff3870, ...})
2024-05-20T19:36:41.9784740Z    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.57.0/call.go:42 +0x2cb
2024-05-20T19:36:41.9785339Z go.uber.org/yarpc/transport/grpc.(*Outbound).invoke(0xc0013b6ab0, {0x1ab0558, 0xc00043ee00}, 0xc000c598c0, 0x5?, 0xc000707368, {0x40b7e9?, 0xc0010188f8?, 0x26e0980?})
2024-05-20T19:36:41.9785655Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:210 +0x4e5
2024-05-20T19:36:41.9786099Z go.uber.org/yarpc/transport/grpc.(*Outbound).Call(0xc0013b6ab0, {0x1ab0558, 0xc00043ee00}, 0xc000c598c0)
2024-05-20T19:36:41.9786412Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/transport/grpc/outbound.go:123 +0x126
2024-05-20T19:36:41.9787029Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558?, 0xc00043ee00?}, 0x184f9bf?)
2024-05-20T19:36:41.9787395Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:103 +0x5c
2024-05-20T19:36:41.9787914Z go.uber.org/yarpc/internal/observability.(*Middleware).Call(0xc00183bc80, {0x1ab0558, 0xc00043ee00}, 0xc000c598c0, {0x1ab4e00, 0xc000a74b10})
2024-05-20T19:36:41.9788337Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/observability/middleware.go:213 +0x162
2024-05-20T19:36:41.9788953Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x0, 0x0}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc00043ee00}, 0xc0013b6ab0?)
2024-05-20T19:36:41.9789318Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9789952Z go.uber.org/yarpc/internal/firstoutboundmiddleware.(*Middleware).Call(0x17706e0?, {0x1ab0558, 0xc00043ee00}, 0xc000c598c0, {0x1ab4e00?, 0xc000a74ae0})
2024-05-20T19:36:41.9790388Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/firstoutboundmiddleware/middleware.go:51 +0xa8
2024-05-20T19:36:41.9791010Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChainExec.Call({{0xc000690f30, 0x1, 0x1}, {0x1ab3440, 0xc0013b6ab0}}, {0x1ab0558, 0xc00043ee00}, 0xc000070c00?)
2024-05-20T19:36:41.9791393Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:107 +0xf7
2024-05-20T19:36:41.9791643Z go.uber.org/yarpc/internal/outboundmiddleware.unaryChain.Call(...)
2024-05-20T19:36:41.9791973Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/outboundmiddleware/chain.go:67
2024-05-20T19:36:41.9792257Z go.uber.org/yarpc/api/middleware.unaryOutboundWithMiddleware.Call(...)
2024-05-20T19:36:41.9792537Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/api/middleware/outbound.go:113
2024-05-20T19:36:41.9793151Z go.uber.org/yarpc/internal/request.UnaryValidatorOutbound.Call({{0x1ab4c80?, 0xc0013b6c30?}, {0x7fd73c34b008?, 0xc0013b6c30?}}, {0x1ab0558, 0xc00043ee00}, 0x53336e?)
2024-05-20T19:36:41.9793514Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/internal/request/validator_outbound.go:46 +0x92
2024-05-20T19:36:41.9794208Z go.uber.org/yarpc/encoding/protobuf.(*client).Call(0xc0013b7200, {0x1ab0558?, 0xc00043ee00?}, {0x184b747?, 0x0?}, {0x1aa94f0?, 0xc000b965f0?}, 0x1903938, {0xc00183ff80, 0x4, ...})
2024-05-20T19:36:41.9794535Z    /home/runner/go/pkg/mod/go.uber.org/yarpc@v1.60.0/encoding/protobuf/outbound.go:95 +0x10c
2024-05-20T19:36:41.9795365Z github.com/uber/cadence-idl/go/proto/api/v1.(*_WorkflowAPIYARPCCaller).QueryWorkflow(0xc000a749c0?, {0x1ab0558?, 0xc00043ee00?}, 0xc000751188?, {0xc00183ff80?, 0xc00183ff80?, 0x184f117?})
2024-05-20T19:36:41.9796053Z    /home/runner/go/pkg/mod/github.com/uber/cadence-idl@v0.0.0-20220713235846-fda89e95df1e/go/proto/api/v1/service_workflow.pb.yarpc.go:445 +0x66
2024-05-20T19:36:41.9796873Z go.uber.org/cadence/internal/compatibility.thrift2protoAdapter.QueryWorkflow({{0x1ab3c80, 0xc0010956f0}, {0x1abd1d8, 0xc001095710}, {0x1abd260, 0xc001095730}, {0x1ab7ea0, 0xc001095750}}, {0x1ab0558, 0xc00043ee00}, ...)
2024-05-20T19:36:41.9797449Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/compatibility/adapter.go:131 +0x95
2024-05-20T19:36:41.9798088Z go.uber.org/cadence/internal/common/metrics.(*workflowServiceMetricsWrapper).QueryWorkflow(0xc0013b7380, {0x1ab0558, 0xc00043ee00}, 0x0?, {0xc00183ff80, 0x4, 0x6})
2024-05-20T19:36:41.9798697Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/metrics/service_wrapper.go:353 +0x9d
2024-05-20T19:36:41.9799058Z go.uber.org/cadence/internal.(*workflowClient).QueryWorkflowWithOptions.func1()
2024-05-20T19:36:41.9799641Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow_client.go:965 +0x9c
2024-05-20T19:36:41.9800120Z go.uber.org/cadence/internal/common/backoff.Retry({0x1ab04e8, 0xc000b96550}, 0xc00099b478, {0x1a9ede0?, 0xc001821cb0}, 0x1903a78)
2024-05-20T19:36:41.9800680Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:101 +0xe5
2024-05-20T19:36:41.9801150Z go.uber.org/cadence/internal.(*workflowClient).QueryWorkflowWithOptions(0xc001000880, {0x1ab04e8?, 0xc000b96550}, 0xc00183ff20)
2024-05-20T19:36:41.9801789Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow_client.go:960 +0x3d4
2024-05-20T19:36:41.9802550Z github.com/indeedeng/iwf/service/api/cadence.queryWorkflowWithStrongConsistency(0xc001182780, {0x1ab04e8, 0xc000b96550}, {0xc000696060, 0x20}, {0xc001821c80, 0x24}, {0x1850f8a, 0x11}, {0x0, ...})
2024-05-20T19:36:41.9802793Z    /home/runner/work/iwf/iwf/service/api/cadence/client.go:233 +0x1af
2024-05-20T19:36:41.9803647Z github.com/indeedeng/iwf/service/api/cadence.(*cadenceClient).QueryWorkflow(0x75?, {0x1ab04e8?, 0xc000b96550?}, {0x1547320, 0xc0006fb810}, {0xc000696060?, 0x1001311?}, {0xc001821c80?, 0xc00010b280?}, {0x1850f8a, ...}, ...)
2024-05-20T19:36:41.9803881Z    /home/runner/work/iwf/iwf/service/api/cadence/client.go:216 +0xaa
2024-05-20T19:36:41.9804528Z github.com/indeedeng/iwf/service/api.(*serviceImpl).ApiV1WorkflowDumpPost(0xc000f08d00, {0x1ab04e8, 0xc000b96550}, {{0xc000696060, 0x20}, {0xc001821c80, 0x24}, 0x100000, 0x0})
2024-05-20T19:36:41.9804731Z    /home/runner/work/iwf/iwf/service/api/service.go:782 +0x112
2024-05-20T19:36:41.9805106Z github.com/indeedeng/iwf/service/api.(*handler).apiV1WorkflowInternalDump(0xc000691540, 0xc000c00400)
2024-05-20T19:36:41.9805309Z    /home/runner/work/iwf/iwf/service/api/handler.go:121 +0x259
2024-05-20T19:36:41.9805486Z github.com/gin-gonic/gin.(*Context).Next(...)
2024-05-20T19:36:41.9805796Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
2024-05-20T19:36:41.9806097Z github.com/gin-gonic/gin.CustomRecoveryWithWriter.func1(0xc000c00400)
2024-05-20T19:36:41.9806430Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 +0x7a
2024-05-20T19:36:41.9806601Z github.com/gin-gonic/gin.(*Context).Next(...)
2024-05-20T19:36:41.9806893Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
2024-05-20T19:36:41.9807209Z github.com/gin-gonic/gin.LoggerWithConfig.func1(0xc000c00400)
2024-05-20T19:36:41.9807531Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 +0xde
2024-05-20T19:36:41.9807699Z github.com/gin-gonic/gin.(*Context).Next(...)
2024-05-20T19:36:41.9807991Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174
2024-05-20T19:36:41.9808325Z github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000ba3860, 0xc000c00400)
2024-05-20T19:36:41.9808629Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 +0x65b
2024-05-20T19:36:41.9809052Z github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000ba3860, {0x1aa59b0?, 0xc0006de2a0}, 0xc000c00300)
2024-05-20T19:36:41.9809356Z    /home/runner/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 +0x1dd
2024-05-20T19:36:41.9809629Z net/http.serverHandler.ServeHTTP({0x1aa1d30?}, {0x1aa59b0?, 0xc0006de2a0?}, 0x6?)
2024-05-20T19:36:41.9809864Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:2938 +0x8e
2024-05-20T19:36:41.9810049Z net/http.(*conn).serve(0xc000679950, {0x1ab04b0, 0xc001586120})
2024-05-20T19:36:41.9810279Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:2009 +0x5f4
2024-05-20T19:36:41.9810454Z created by net/http.(*Server).Serve in goroutine 15192
2024-05-20T19:36:41.9810672Z    /opt/hostedtoolcache/go/1.21.10/x64/src/net/http/server.go:3086 +0x5cb
2024-05-20T19:36:41.9810677Z 
2024-05-20T19:36:41.9810851Z goroutine 15555 [chan receive]:
2024-05-20T19:36:41.9811030Z go.uber.org/cadence/internal.(*coroutineState).call(...)
2024-05-20T19:36:41.9811555Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:839
2024-05-20T19:36:41.9811855Z go.uber.org/cadence/internal.(*dispatcherImpl).ExecuteUntilAllBlocked(0xc0017c2480)
2024-05-20T19:36:41.9812394Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:923 +0x1a6
2024-05-20T19:36:41.9812759Z go.uber.org/cadence/internal.executeDispatcher({0x1ab2890, 0xc000d7aba0}, {0x1ab27e8, 0xc0017c2480})
2024-05-20T19:36:41.9813312Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:540 +0x8a
2024-05-20T19:36:41.9813706Z go.uber.org/cadence/internal.(*syncWorkflowDefinition).OnDecisionTaskStarted(0xc000b27b00?)
2024-05-20T19:36:41.9814270Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:514 +0x26
2024-05-20T19:36:41.9814728Z go.uber.org/cadence/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc0004fc7e0, 0xc000df7e00, 0x80?, 0x0)
2024-05-20T19:36:41.9815532Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_event_handlers.go:820 +0x1fb
2024-05-20T19:36:41.9815962Z go.uber.org/cadence/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc000a35140, 0xc001618d80)
2024-05-20T19:36:41.9816534Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_handlers.go:895 +0x1bbc
2024-05-20T19:36:41.9816976Z go.uber.org/cadence/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc000aec000, 0xc001618d80, 0xc00101c990)
2024-05-20T19:36:41.9817546Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_handlers.go:787 +0x42a
2024-05-20T19:36:41.9817895Z go.uber.org/cadence/internal.(*workflowTaskPoller).processWorkflowTask(0xc000d47980, 0xc001618d80)
2024-05-20T19:36:41.9818465Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:341 +0x236
2024-05-20T19:36:41.9818836Z go.uber.org/cadence/internal.(*workflowTaskPoller).ProcessTask(0xc000d47980, {0x15517a0?, 0xc001618d80?})
2024-05-20T19:36:41.9819394Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:314 +0x95
2024-05-20T19:36:41.9819744Z go.uber.org/cadence/internal.(*baseWorker).processTask(0xc000fdc480, {0x1551fe0?, 0xc000f14f40?})
2024-05-20T19:36:41.9820369Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:361 +0xe2
2024-05-20T19:36:41.9820702Z created by go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher in goroutine 15211
2024-05-20T19:36:41.9821247Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:268 +0xb1
2024-05-20T19:36:41.9821254Z 
2024-05-20T19:36:41.9821354Z goroutine 15131 [semacquire]:
2024-05-20T19:36:41.9821477Z sync.runtime_Semacquire(0xc000832e80?)
2024-05-20T19:36:41.9821686Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/sema.go:62 +0x25
2024-05-20T19:36:41.9821788Z sync.(*WaitGroup).Wait(0x1551fe0?)
2024-05-20T19:36:41.9822017Z    /opt/hostedtoolcache/go/1.21.10/x64/src/sync/waitgroup.go:116 +0x48
2024-05-20T19:36:41.9822240Z go.uber.org/cadence/internal/common/util.AwaitWaitGroup.func1()
2024-05-20T19:36:41.9822773Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/util/util.go:52 +0x25
2024-05-20T19:36:41.9823086Z created by go.uber.org/cadence/internal/common/util.AwaitWaitGroup in goroutine 14758
2024-05-20T19:36:41.9823608Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/util/util.go:51 +0x7d
2024-05-20T19:36:41.9823613Z 
2024-05-20T19:36:41.9823790Z goroutine 14795 [sleep]:
2024-05-20T19:36:41.9823881Z time.Sleep(0x219beb45)
2024-05-20T19:36:41.9824096Z    /opt/hostedtoolcache/go/1.21.10/x64/src/runtime/time.go:195 +0x125
2024-05-20T19:36:41.9824470Z go.uber.org/cadence/internal/common/backoff.(*ConcurrentRetrier).throttleInternal(0xc001173320)
2024-05-20T19:36:41.9825018Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:65 +0x8b
2024-05-20T19:36:41.9825307Z go.uber.org/cadence/internal/common/backoff.(*ConcurrentRetrier).Throttle(...)
2024-05-20T19:36:41.9825821Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/common/backoff/retry.go:51
2024-05-20T19:36:41.9826110Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc00161d9e0)
2024-05-20T19:36:41.9826675Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:291 +0x21c
2024-05-20T19:36:41.9826897Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc00161d9e0)
2024-05-20T19:36:41.9827444Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9827782Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 14758
2024-05-20T19:36:41.9828329Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9828334Z 
2024-05-20T19:36:41.9828434Z goroutine 15208 [select]:
2024-05-20T19:36:41.9828741Z go.uber.org/cadence/internal.(*basePoller).doPoll(0xc000d47980, {0x1?, 0x0?}, 0xc0011b7c20)
2024-05-20T19:36:41.9829305Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:261 +0x20a
2024-05-20T19:36:41.9829566Z go.uber.org/cadence/internal.(*workflowTaskPoller).PollTask(0xc000d47980)
2024-05-20T19:36:41.9830121Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_task_pollers.go:298 +0x65
2024-05-20T19:36:41.9830344Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc480)
2024-05-20T19:36:41.9830890Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:293 +0x284
2024-05-20T19:36:41.9831109Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc480)
2024-05-20T19:36:41.9831656Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9831994Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9832544Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9832549Z 
2024-05-20T19:36:41.9832649Z goroutine 15207 [select]:
2024-05-20T19:36:41.9833102Z golang.org/x/time/rate.(*Limiter).wait(0xc001685a90, {0x1ab04e8, 0xc0016859f0}, 0x1, {0x0?, 0xc000987c00?, 0x26e0980?}, 0x19053e0)
2024-05-20T19:36:41.9833367Z    /home/runner/go/pkg/mod/golang.org/x/time@v0.3.0/rate/rate.go:283 +0x3eb
2024-05-20T19:36:41.9833655Z golang.org/x/time/rate.(*Limiter).WaitN(0xc000c5aa80?, {0x1ab04e8, 0xc0016859f0}, 0x1?)
2024-05-20T19:36:41.9833903Z    /home/runner/go/pkg/mod/golang.org/x/time@v0.3.0/rate/rate.go:246 +0x50
2024-05-20T19:36:41.9834034Z golang.org/x/time/rate.(*Limiter).Wait(...)
2024-05-20T19:36:41.9834254Z    /home/runner/go/pkg/mod/golang.org/x/time@v0.3.0/rate/rate.go:231
2024-05-20T19:36:41.9834472Z go.uber.org/cadence/internal.(*baseWorker).pollTask(0xc000fdc480)
2024-05-20T19:36:41.9835040Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:292 +0x252
2024-05-20T19:36:41.9835260Z go.uber.org/cadence/internal.(*baseWorker).runPoller(0xc000fdc480)
2024-05-20T19:36:41.9835809Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:242 +0x8d
2024-05-20T19:36:41.9836142Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9836692Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:205 +0x85
2024-05-20T19:36:41.9836697Z 
2024-05-20T19:36:41.9836807Z goroutine 11472 [chan receive]:
2024-05-20T19:36:41.9837149Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc00060d800, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9837679Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9837870Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9838434Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9838700Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc000ae59e0}, 0xc000ae5ef0)
2024-05-20T19:36:41.9839195Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9839314Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9839924Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9840469Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x7fd73c137158?, {0x1a9bb20?, 0xc001190cc0?}, 0x186e6c3?)
2024-05-20T19:36:41.9840804Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9841185Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func4({0x1a9bb20, 0xc001190cc0})
2024-05-20T19:36:41.9841473Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:123 +0x10a
2024-05-20T19:36:41.9841936Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc000ae59e0})
2024-05-20T19:36:41.9842276Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9842580Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000c727e0?)
2024-05-20T19:36:41.9843131Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9843471Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 11468
2024-05-20T19:36:41.9844018Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9844085Z 
2024-05-20T19:36:41.9844207Z goroutine 6566 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9844547Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc000a646c0, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9845094Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9845271Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9845779Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9846039Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc0013a1470}, 0xc000bca3c0)
2024-05-20T19:36:41.9846527Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9846649Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9847174Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9847706Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0xc00180fb00?, {0x1a9bb20?, 0xc000d384e0?}, 0x18383c8?)
2024-05-20T19:36:41.9848042Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9848386Z github.com/indeedeng/iwf/service/interpreter.executeState.func2({0x1a9bb20, 0xc000d384e0})
2024-05-20T19:36:41.9848721Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:579 +0x264
2024-05-20T19:36:41.9849183Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc0013a1470})
2024-05-20T19:36:41.9849511Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9849816Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000762900?)
2024-05-20T19:36:41.9850354Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9850694Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6558
2024-05-20T19:36:41.9851299Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9851305Z 
2024-05-20T19:36:41.9851426Z goroutine 6404 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9851765Z go.temporal.io/sdk/internal.(*coroutineState).initialYield(0xc000890dc0, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9852145Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:945 +0x5f
2024-05-20T19:36:41.9852326Z go.temporal.io/sdk/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9852631Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:955
2024-05-20T19:36:41.9853070Z go.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).Await(0x1ab08a0?, {0x1ab08a0, 0xc000a8db30}, 0xc000be4000)
2024-05-20T19:36:41.9853356Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:364 +0xa5
2024-05-20T19:36:41.9853616Z go.temporal.io/sdk/internal.Await({0x1ab08a0, 0xc000a8db30}, 0xc000a8db30?)
2024-05-20T19:36:41.9853886Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/workflow.go:349 +0x58
2024-05-20T19:36:41.9854007Z go.temporal.io/sdk/workflow.Await(...)
2024-05-20T19:36:41.9854328Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/workflow/deterministic_wrappers.go:84
2024-05-20T19:36:41.9854857Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).Await(0x1547560?, {0x1a9bb20?, 0xc0004cf4e0?}, 0x185a519?)
2024-05-20T19:36:41.9855335Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:203 +0x7e
2024-05-20T19:36:41.9855714Z github.com/indeedeng/iwf/service/interpreter.NewSignalReceiver.func5({0x1a9bb20, 0xc0004cf4e0})
2024-05-20T19:36:41.9856000Z    /home/runner/work/iwf/iwf/service/interpreter/signalReceiver.go:143 +0x1a9
2024-05-20T19:36:41.9856602Z github.com/indeedeng/iwf/service/interpreter/temporal.(*workflowProvider).GoNamed.func1({0x1ab08a0?, 0xc000a8db30})
2024-05-20T19:36:41.9857067Z    /home/runner/work/iwf/iwf/service/interpreter/temporal/workflowProvider.go:180 +0xfd
2024-05-20T19:36:41.9857355Z go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine.func1(0xc000ef6ab0?)
2024-05-20T19:36:41.9857684Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1057 +0xcf
2024-05-20T19:36:41.9857994Z created by go.temporal.io/sdk/internal.(*dispatcherImpl).NewCoroutine in goroutine 6399
2024-05-20T19:36:41.9858337Z    /home/runner/go/pkg/mod/go.temporal.io/sdk@v1.25.1/internal/internal_workflow.go:1048 +0x338
2024-05-20T19:36:41.9858342Z 
2024-05-20T19:36:41.9858483Z goroutine 6558 [chan receive, 4 minutes]:
2024-05-20T19:36:41.9858821Z go.uber.org/cadence/internal.(*coroutineState).initialYield(0xc0000f0780, 0x3, {0x183a2ac, 0x5})
2024-05-20T19:36:41.9859385Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:796 +0x5f
2024-05-20T19:36:41.9859564Z go.uber.org/cadence/internal.(*coroutineState).yield(...)
2024-05-20T19:36:41.9860078Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:806
2024-05-20T19:36:41.9860329Z go.uber.org/cadence/internal.Await({0x1ab0868, 0xc001177800}, 0xc0013a1620)
2024-05-20T19:36:41.9860926Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/workflow.go:605 +0xa5
2024-05-20T19:36:41.9861051Z go.uber.org/cadence/workflow.Await(...)
2024-05-20T19:36:41.9861575Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/workflow/deterministic_wrappers.go:62
2024-05-20T19:36:41.9862124Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).Await(0x15f89a0?, {0x1a9bb20?, 0xc000a91470?}, 0xc000eb7110?)
2024-05-20T19:36:41.9862463Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:165 +0x7e
2024-05-20T19:36:41.9864148Z github.com/indeedeng/iwf/service/interpreter.executeState({_, _}, {_, _}, _, {{_, _}, {_, _}}, {{{0xc0002e564e, ...}, ...}, ...}, ...)
2024-05-20T19:36:41.9864444Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:631 +0x1e1e
2024-05-20T19:36:41.9864802Z github.com/indeedeng/iwf/service/interpreter.InterpreterImpl.func2({0x1a9bb20, 0xc000a91410})
2024-05-20T19:36:41.9865074Z    /home/runner/work/iwf/iwf/service/interpreter/workflowImpl.go:183 +0x6bb
2024-05-20T19:36:41.9865599Z github.com/indeedeng/iwf/service/interpreter/cadence.(*workflowProvider).GoNamed.func1({0x1ab0868?, 0xc0011774a0})
2024-05-20T19:36:41.9865937Z    /home/runner/work/iwf/iwf/service/interpreter/cadence/workflowProvider.go:142 +0xfd
2024-05-20T19:36:41.9866240Z go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1(0xc000f14400?)
2024-05-20T19:36:41.9866787Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:884 +0xcf
2024-05-20T19:36:41.9867130Z created by go.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine in goroutine 6551
2024-05-20T19:36:41.9867681Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_workflow.go:875 +0x2e8
2024-05-20T19:36:41.9867687Z 
2024-05-20T19:36:41.9867781Z goroutine 15152 [select]:
2024-05-20T19:36:41.9868039Z go.uber.org/cadence/internal.(*baseWorker).runTaskDispatcher(0xc000fdc900)
2024-05-20T19:36:41.9868607Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:256 +0x11d
2024-05-20T19:36:41.9868877Z created by go.uber.org/cadence/internal.(*baseWorker).Start in goroutine 15134
2024-05-20T19:36:41.9869427Z    /home/runner/go/pkg/mod/go.uber.org/cadence@v0.17.1-0.20230105221902-f50f452a8eae/internal/internal_worker_base.go:209 +0x138
2024-05-20T19:36:41.9869432Z

uber-go / cadence-client

Concurrent map writes of search attributes due to potential racing condition #1198