dhiaayachi / temporal

Temporal service
https://docs.temporal.io
MIT License
0 stars 0 forks source link

DescribeWorkflow timeout on wf with 1500 pending activities. #437

Open dhiaayachi opened 2 weeks ago

dhiaayachi commented 2 weeks ago

DescribeWorkflow timeout because it try to fetch scheduled event for each of the pending activity, only because it needs to show ActivityType for each pending activity. See the code: // TODO: move to mutable state instead of loading it from event scheduledEvent, err := mutableState.GetActivityScheduledEvent(ctx, ai.ScheduledEventId)

This lead to timeout when the number of pending activity is large.

dhiaayachi commented 1 day ago

Thank you for reporting this issue.

It seems like the workflow is timing out because it's trying to fetch the scheduled event for each pending activity to display the ActivityType. This can cause a timeout when the number of pending activities is large.

I'm currently unable to provide a solution without further context. Could you please provide more information about your workflow?

For example:

  1. What is the purpose of your workflow?
  2. Why do you need to display the ActivityType for each pending activity?
  3. How many pending activities are you dealing with?
  4. Can you provide a snippet of the code where the GetActivityScheduledEvent method is called?

Understanding your workflow and its requirements will help me provide a more targeted solution.

dhiaayachi commented 1 day ago

Thanks for reporting this issue.

This issue seems to stem from how you're loading the ActivityScheduledEvent.

I'm a bit unclear on your context here. Could you please tell me what the mutableState is? Is it the Workflow execution state?

Additionally, to better understand, could you share the following:

Once I have this information, I can provide a more accurate solution.

dhiaayachi commented 1 day ago

Thank you for reporting this issue. It seems that the Workflow is timing out because it's trying to fetch scheduled events for each pending activity to display ActivityType. This can lead to timeout when the number of pending activities is large.

To avoid this timeout, you can consider moving the fetching of the scheduled event to the mutable state instead of loading it from the event each time. This will help optimize the Workflow performance.

You can also explore using techniques such as chunking or pagination to reduce the number of events fetched at once.

Please let me know if you have any other questions.