What could be the problem with the slow opening of the temporal web?

[dhiaayachi]

I'm seeing a very slow loading of the resource http://10.0.0.57:32080/api/v1/namespaces/Answer-Flow/workflow-count?query=GROUP+BY+ExecutionStatus, which takes nearly 1 minute. Our temporal uses a MySQL database

What could be the possible reasons? Please help me！

[dhiaayachi]

Slow Loading of Workflow Count Endpoint

You're experiencing a significant delay when retrieving the workflow count using the endpoint http://10.0.0.57:32080/api/v1/namespaces/Answer-Flow/workflow-count?query=GROUP+BY+ExecutionStatus. This suggests a potential issue with your Temporal setup, particularly involving your MySQL database.

Here are some possible reasons for the slow loading time:

1. Database Performance:

• Slow Queries: The GROUP BY clause in your query can significantly impact performance, especially if your database has a large amount of data. Try simplifying the query or using appropriate indexing to improve performance.
• Database Load: If your MySQL database is under heavy load from other applications, it could be impacting the performance of your Temporal queries. Monitor your database load and consider scaling or optimizing your database to handle the increased workload.
• Database Configuration: Incorrect database configuration settings, such as low resource allocation or inefficient indexing, can contribute to slow query performance. Review your MySQL configuration for potential improvements.

2. Network Issues:

• Latency: High network latency between your Temporal server and the MySQL database can lead to slow response times for queries. Check your network connection for any performance bottlenecks and optimize if necessary.

3. Temporal Server Configuration:

• Visibility Settings: Check your Temporal server configuration for any limitations on the Visibility service. Ensure that you are not exceeding any resource limits or have enabled advanced Visibility features if required.
• Persistence Settings: Your Temporal server configuration might have settings related to the persistence layer that could affect query performance. Review your system.visibilityPersistenceEnabled configuration for potential improvements.

4. Temporal Server Load:

• High Workflow Activity: If your Temporal server is processing a large number of Workflows, it could be affecting the performance of the workflow count endpoint. Monitor your Temporal server metrics and consider scaling to handle the increased workload.

5. Temporal Server Version:

• Older Version: Older versions of Temporal might have performance limitations related to the Visibility service. Consider upgrading to the latest version of Temporal for potential performance improvements.

Debugging Steps:

1. Monitor MySQL: Use the SHOW PROCESSLIST command to see what queries are running on your MySQL server and identify any slow or long-running queries.
2. Analyze Temporal Server Logs: Check the Temporal Server logs for any errors or warnings related to the Visibility service or database connectivity.
3. Check Network Latency: Use tools like ping or traceroute to measure the network latency between your Temporal server and the MySQL database.
4. Optimize Queries: Simplify your GROUP BY query or add appropriate indexes to improve performance.
5. Profile Temporal Server: Use profiling tools to identify potential bottlenecks within your Temporal server application.
6. Temporal Server Metrics: Examine your Temporal server metrics (especially temporal_worker_task_slots_available, temporal_workflow_task_schedule_to_start_latency, temporal_activity_schedule_to_start_latency) to assess server load and resource utilization.

References:

• Temporal Visibility Documentation
• Temporal Server Configuration Reference
• MySQL Performance Tuning

By carefully investigating the potential reasons for the slow loading time and using the debugging steps provided, you can pinpoint the root cause and take appropriate actions to improve the performance of your Temporal workflow count endpoint.

[dhiaayachi]

Thanks for reporting this issue. Could you please tell me which version of the Temporal server are you using? Also, could you provide information on the workflow-count endpoint you're using, as Temporal doesn't have a built-in endpoint for that purpose. Is this endpoint from a custom service or third-party application?

[dhiaayachi]

Thanks for reporting this issue! I see that you're experiencing a slow loading time for the http://10.0.0.57:32080/api/v1/namespaces/Answer-Flow/workflow-count?query=GROUP+BY+ExecutionStatus endpoint, which is taking nearly a minute.

Could you please provide more context to help troubleshoot this issue?

1. What is the version of Temporal that you're using?
2. What is the size of your MySQL database?
3. Are you using any custom queries or filters in Temporal Web UI?
4. Have you made any recent changes to your Temporal configuration or MySQL database?

Once I have this information, I can help you find the most appropriate solution for your slow loading time.