tdbg diagnose command for stuck workflow

[dhiaayachi]

We need a tdbg command that could help dump all relevant information and do some basic analysis. tdbg workflow diagnose

Information includes: 1) Mutable state 2) Workflow History 3) Shard info (ack level, read level for all queues) 4) Task queue info (including all partitions, backlog size etc) 5) Namespace info

Diagnose part: Check if there should have workflow task scheduled or started, but didn't happen. Check if there is any timer that should already fired but did not happen. Check if there is worker polling the right task queue.

[dhiaayachi]

Thank you for your feature request! We understand the need for a comprehensive diagnostic tool within Temporal.

Currently, there isn't a single tdbg command that fulfills all these requirements. However, you can combine existing tools and techniques to achieve similar results.

To gather the information you need:

• Mutable state: Use temporal-cli describe-workflow command.
• Workflow History: Use temporal-cli describe-workflow command.
• Shard info: Use temporal-cli list-shards and temporal-cli describe-shard commands.
• Task queue info: Use temporal-cli describe-task-queue command.
• Namespace info: Use temporal-cli describe-namespace command.

For the "diagnose" part, you can use the following:

• Check for unscheduled tasks: Use the temporal-cli list-open-workflows command to see if the workflow has any pending tasks.
• Check for unfired timers: Inspect the workflow history using the temporal-cli describe-workflow command. You can find the timer events.
• Check for worker polling: Use the temporal-cli list-workers command to confirm the worker is polling the correct task queue.

We encourage you to join the Temporal community and submit a feature request for a dedicated tdbg workflow diagnose command. This will help us prioritize development efforts to improve the debugging experience.

Let me know if you have any other questions!

[dhiaayachi]

Thank you for this feature request! It sounds like you are looking for a more comprehensive debugging tool within Temporal. Currently, Temporal doesn't have a single command like tdbg workflow diagnose to perform all the actions you've outlined.

However, you can achieve most of this functionality by combining existing tools and approaches:

• Mutable State: Use the temporalctl workflow describe command to inspect a workflow's current state. Temporal CLI Reference
• Workflow History: The temporalctl workflow describe command also includes the workflow history. You can also use the temporalctl history describe command for a more detailed view of the workflow history. Temporal CLI Reference
• Shard info, Task queue info, and Namespace info: You can use the temporalctl cluster describe command to obtain information about shards, task queues, and namespaces. Temporal CLI Reference
• Diagnosis: Temporal's built-in monitoring and logging tools are valuable for debugging workflow issues. You can use the Temporal UI, the temporalctl CLI, or your custom logging solutions to analyze workflow behavior, identify potential issues, and perform additional troubleshooting. Temporal UI, Temporal CLI Reference

We appreciate your feedback and will consider it for future enhancements to Temporal's debugging capabilities.

[dhiaayachi]

Thank you for the feature request!

Currently, Temporal does not have a tdbg workflow diagnose command. However, you can use the following commands to achieve similar results:

• Workflow History:
  • temporal describe-workflow-execution --workflow-id <workflow-id>
  • temporal history --workflow-id <workflow-id>
• Shard Info:
  • temporal describe-namespace --namespace <namespace> (Includes information about the namespace, which can be helpful for troubleshooting shards)
• Task Queue Info:
  • temporal describe-task-queue --task-queue <task-queue>
• Mutable State:
  • This information is included in the temporal history output.
• Diagnose:
  • You can use the temporal describe-workflow-execution command to determine if a workflow task should have been scheduled or started, but was not.
  • You can also use the temporal describe-timers --namespace <namespace> command to see if there are any timers that should have fired but did not.
  • To check if a worker is polling the correct task queue, you can review the worker logs.

For more information on using these commands, please refer to the Temporal CLI documentation: https://docs.temporal.io/docs/temporal-cli/

[dhiaayachi]

Thank you for the feature request! We appreciate you taking the time to share this.

While we don't currently have a tdbg workflow diagnose command, you can get the information you need by using a combination of existing tools:

• Mutable State: Use temporalctl workflow describe to view the mutable state of a workflow.
• Workflow History: Use temporalctl workflow describe to view the workflow history.
• Shard Info: Use temporalctl cluster describe to view shard info.
• Task Queue Info: Use temporalctl taskqueue describe to view the task queue info.
• Namespace Info: Use temporalctl namespace describe to view the namespace info.

For the diagnose part of your request, we recommend:

• Check if there should have workflow task scheduled or started, but didn't happen: Use temporalctl workflow describe to review the history of the workflow. This should show if any tasks have been scheduled or not.
• Check if there is any timer that should already fired but did not happen: Review the workflow history using temporalctl workflow describe.
• Check if there is worker polling the right task queue: Use temporalctl taskqueue describe to confirm the task queue the worker is polling.

We will consider adding a tdbg workflow diagnose command in the future.

[dhiaayachi]

Thank you for this feature request.

While we don't have a tdbg workflow diagnose command at this time, you can achieve some of the same functionality using existing tools:

• Mutable state: The temporalctl workflow describe command can be used to view the current mutable state of a workflow. https://docs.temporal.io/docs/temporalctl/commands/workflow
• Workflow History: You can access the workflow history with the temporalctl workflow history command. https://docs.temporal.io/docs/temporalctl/commands/workflow
• Shard info, Task Queue info, and Namespace info: Use the temporalctl cluster describe command with the --shard, --taskQueue and --namespace options to get detailed information about those entities. https://docs.temporal.io/docs/temporalctl/commands/cluster

To check for potential issues with task scheduling, timers, and worker polling, you can use the following:

• Task Scheduling: Review the workflow history to ensure that the expected tasks are being scheduled.
• Timers: You can use the temporalctl namespace describe command and look for any active timers to verify they are working correctly. https://docs.temporal.io/docs/temporalctl/commands/namespace
• Worker polling: Use the temporalctl worker describe command to check that workers are polling the correct task queues. https://docs.temporal.io/docs/temporalctl/commands/worker

We appreciate your feedback, and we'll consider adding a tdbg workflow diagnose command in the future.