Netflix / conductor-community

Apache License 2.0
61 stars 71 forks source link

Postgres indexing problem #252

Open rpe4a opened 11 months ago

rpe4a commented 11 months ago

Describe the bug I try to use postgres with indexing insted of elastic. And execute a simple test workflow but conductor can't update task_index and has an error:

docker-postgres-1  | 2023-07-26 18:00:56.578 UTC [2661] ERROR:  value too long for type character varying(32)
docker-postgres-1  | 2023-07-26 18:00:56.578 UTC [2661] STATEMENT:  INSERT INTO task_index (task_id, task_type, task_def_name, status, start_time, update_time, workflow_type, json_data)VALUES ($1, $2, $3, $4, $5, $6, $7, $8::JSONB) ON CONFLICT (task_id) DO UPDATE SET task_type = EXCLUDED.task_type, task_def_name = EXCLUDED.task_def_name, status = EXCLUDED.status, update_time = EXCLUDED.update_time, json_data = EXCLUDED.json_data
conductor-server   | 267641 [http-nio-8080-exec-20] ERROR com.netflix.conductor.core.dal.ExecutionDAOFacade [] - Error updating task: c979373e-f80a-4de9-a992-7ae712c0b079 in workflow: 09c197b2-d91a-4776-9aea-2766f622afb3
conductor-server   | com.netflix.conductor.core.exception.NonTransientException: ERROR: value too long for type character varying(32)
conductor-server   |    at com.netflix.conductor.postgres.dao.PostgresBaseDAO.getWithRetriedTransactions(PostgresBaseDAO.java:147) ~[conductor-postgres-persistence-3.13.8.jar!/:3.13.8]
conductor-server   |    at com.netflix.conductor.postgres.dao.PostgresBaseDAO.queryWithTransaction(PostgresBaseDAO.java:209) ~[conductor-postgres-persistence-3.13.8.jar!/:3.13.8]
conductor-server   |    at com.netflix.conductor.postgres.dao.PostgresIndexDAO.indexTask(PostgresIndexDAO.java:101) ~[conductor-postgres-persistence-3.13.8.jar!/:3.13.8]
conductor-server   |    at com.netflix.conductor.core.dal.ExecutionDAOFacade.updateTask(ExecutionDAOFacade.java:514) ~[conductor-core-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]
conductor-server   |    at com.netflix.conductor.service.ExecutionService.poll(ExecutionService.java:172) ~[conductor-core-3.14.0-SNAPSHOT.jar!/:3.14.0-SNAPSHOT]

The main problem is that task_index table has field task_type which is type varchar(32) but conductor tries to insert there wrong value. On the first screenshoot I see Task type = SIMPLE and Task name = TestWorkerDynamicSubWorkflowForkInput but on the second screenshoot on the postgres base I see Task type = TestTransferChannelWorker1 and Task name = TestTransferChannelWorker1. And here is a problem:

Details Conductor version: 3.13.8 Persistence implementation: Postgres Queue implementation: Redis Lock: Redis or Zookeeper? Workflow definition: Task definition: Event handler definition:

To Reproduce Steps to reproduce the behavior:

  1. Use postgres index insted of elastic
  2. Create workflow with task which has name more then 32 symbols;
  3. Run it
  4. See conductor server logs to catch the error.

Expected behavior Update operation will be successful.

Screenshots image image