invalid character '1' looking for beginning of value",

slnw commented 7 months ago

Expected Behavior

It should execute the activity as usual.

Actual Behavior

Locally on arm64 it works fine, When deploying to production on some activities execution I get error:

{"message":"proto_codec_parse_message: invalid character '1' looking for beginning of value","source":"GoSDK","stackTrace":"process event for default [panic]:\ngithub.com/temporalio/roadrunner-temporal/v4/aggregatedpool.(*Workflow).OnWorkflowTaskStarted(0xc00126b340, 0x18?)\n\tgithub.com/temporalio/roadrunner-temporal/v4@v4.6.1/aggregatedpool/workflow.go:184 +0x3e7\ngo.temporal.io/sdk/internal.(*workflowExecutionEventHandlerImpl).ProcessEvent(0xc001424168, 0xc0008f7030, 0x20?, 0x1)\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_event_handlers.go:1143 +0x225\ngo.temporal.io/sdk/internal.(*workflowExecutionContextImpl).ProcessWorkflowTask(0xc001420900, 0xc0011d8690)\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_task_handlers.go:1100 +0x1608\ngo.temporal.io/sdk/internal.(*workflowTaskHandlerImpl).ProcessWorkflowTask(0xc000a91930, 0xc0011d8690, 0xc001420900, 0xc001422810)\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_task_handlers.go:889 +0x3c5\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).processWorkflowTask(0xc0008d5320, 0xc0011d8690)\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_task_pollers.go:357 +0x3c3\ngo.temporal.io/sdk/internal.(*workflowTaskPoller).ProcessTask(0xc0008d5320, {0x1a2c3a0, 0xc0011d8690})\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_task_pollers.go:321 +0x78\ngo.temporal.io/sdk/internal.(*baseWorker).processTask(0xc000a82420, {0x1a2cea0, 0xc0011e2040})\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_worker_base.go:518 +0x153\ngo.temporal.io/sdk/internal.(*baseWorker).processTaskAsync.func1()\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_worker_base.go:369 +0x45\ncreated by go.temporal.io/sdk/internal.(*baseWorker).processTaskAsync in goroutine 91\n\tgo.temporal.io/sdk@v1.26.0-rc.3/internal/internal_worker_base.go:365 +0xa5","encodedAttributes":null,"cause":null,"applicationFailureInfo":{"type":"PanicError","nonRetryable":true,"details":null}}

On docker container

{"level":"warn","ts":"2024-03-01T16:54:31.782Z","msg":"Critical attempts processing workflow task","service":"history","shard-id":1,"address":"127.0.0.1:7234","wf-namespace":"testing","wf-id":"8bfe431b-b9ce-43ea-a72c-130894bdb629","wf-run-id":"a6c94584-9be9-4737-a9f8-ad6b146a9c7b","attempt":10,"logging-call-at":"workflow_task_state_machine.go:925"}

I tried chaging temporal pgres server version to 13 and 16 both didn't work.

Steps to Reproduce the Problem

Not sure how to reproduce, the error also doesn't provide the payload, and am not sure how to see it in Temporal. Screenshot 2024-03-01 at 17 54 35

Specifications

Version: temporalio/auto-setup:1.22.6, pgres: 13 & 16
Platform: ubuntu, amd64, temporal/sdk:v2.7.6, PHP/Laravel app.

dynamic conf:

          limit.maxIDLength:
            - value: 255
              constraints: {}
          frontend.keepAliveMaxConnectionAge:
          - value: 5m
          frontend.keepAliveMaxConnectionAgeGrace:
          - value: 70s
          frontend.enableClientVersionCheck:
          - value: true
            constraints: {}
          history.persistenceMaxQPS:
          - value: 3000
            constraints: {}
          frontend.persistenceMaxQPS:
          - value: 3000
            constraints: {}
          frontend.historyMgrNumConns:
          - value: 10
            constraints: {}
          frontend.throttledLogRPS:
          - value: 20
            constraints: {}
          history.historyMgrNumConns:
          - value: 50
            constraints: {}
          history.defaultActivityRetryPolicy:
          - value:
              InitialIntervalInSeconds: 1
              MaximumIntervalCoefficient: 100.0
              BackoffCoefficient: 2.0
              MaximumAttempts: 0
          history.defaultWorkflowRetryPolicy:
          - value:
              InitialIntervalInSeconds: 1
              MaximumIntervalCoefficient: 100.0
              BackoffCoefficient: 2.0
              MaximumAttempts: 0

pfy-oleksii-storozhylov commented 7 months ago

I have the same issue. For me, the problem is reproduced with RoadRunner starting from version 2023.3.11, and it works fine with RR 2023.3.10

rustatian commented 7 months ago

Hey @slnw 👋 The problem is in the latest RR versions (2023.3.11 and 2023.3.12). We're working on the fix atm, please, use v2023.3.10 at the moment.

rustatian commented 7 months ago

Fixed via PHP-SDK v2.8.0

temporalio / roadrunner-temporal