Lots of log noise on restarting server with sqlite persistence

[dhiaayachi]

Expected Behavior

No log noise.

Actual Behavior

On 1.24.1 (i.e. latest CLI as of this writing). Doing a simple temporal server start-dev --db-filename my-db.db, interrupting it (does graceful shutdown), then running that command again gives lots of logs. This is in addition to the log noise server already gives on startup/shutdown of CLI.

[dhiaayachi]

Temporal CLI start-dev log noise on restart

This issue seems to be related to the logging behavior of the Temporal CLI start-dev command when restarting the development server with a persistent database.

Expected behavior: There should be no log noise when restarting the development server.

Actual behavior: When restarting the server with temporal server start-dev --db-filename my-db.db after interrupting it (which should trigger a graceful shutdown), there is a significant amount of log output. This is in addition to the log noise the server already generates during startup and shutdown.

Possible causes:

1. Missing or incomplete shutdown: The graceful shutdown might not be completely cleaning up resources before the server restarts. This could lead to logs being generated when the server attempts to re-use the existing resources.

2. Redundant initialization: The server might be re-initializing components that were already initialized during the previous startup, leading to additional log output.

3. Log rotation issues: The logging configuration might not be handling log rotation properly, leading to the accumulation of logs from previous sessions.

Solution:

1. Review and enhance shutdown process: Examine the graceful shutdown implementation to ensure that all resources are properly released before restarting.

2. Optimize initialization process: Identify unnecessary re-initializations during startup and streamline the process to reduce log output.

3. Implement proper log rotation: Configure log rotation mechanism in Temporal Server settings to ensure logs from previous sessions are handled correctly.

Relevant documentation:

• Temporal CLI command options reference: This documentation page details the options available when starting the development server with start-dev, including logging configuration.
• Temporal Server configuration reference: This documentation page covers all configuration options available for Temporal Server, including logging settings.
• Temporal Logging guide: This guide provides instructions on how to configure and manage logging in a self-hosted Temporal Server environment.

Further investigation:

• Inspect Temporal Server logs: Examine the server logs for any error messages or warnings related to the shutdown process or initialization.
• Debug Temporal CLI code: If possible, debug the start-dev command implementation to pinpoint the source of the excessive logging.
• Consider using a different logging library: Experiment with using a different logging library in the Temporal CLI, potentially one with more fine-grained control over log output.

Note:

The start-dev command is designed for development environments and may not have the same level of optimization or logging control as a production environment. This issue might be resolved in future releases of the Temporal CLI.

Please provide more context or information if possible, like:

• Specific error messages or warnings in the logs
• Code snippets of how you are using the start-dev command
• Version of the Temporal CLI and Temporal Server you are using

This information will help in providing a more precise diagnosis and a targeted solution.

[dhiaayachi]

Expected Behavior

No log noise.

Actual Behavior

On 1.24.1 (i.e. latest CLI as of this writing). Doing a simple temporal server start-dev --db-filename my-db.db, interrupting it (does graceful shutdown), then running that command again gives lots of logs. This is in addition to the log noise server already gives on startup/shutdown of CLI.

Thanks for reporting this issue. It seems you are experiencing a logging issue with Temporal CLI when restarting the server after a graceful shutdown. While we don't have a documented solution yet, we appreciate you bringing this to our attention. Could you please provide the following information so we can investigate this further?

• Operating System: What operating system are you using?
• Temporal CLI version: Are you using the latest version, or have you tried a different version?
• Database Type: Are you using SQLite, MySQL, PostgreSQL, or another database?
• Log Level: What is your current log level set to?
• Log Format: What log format are you using (e.g., json, text)?

The more context you can provide, the better we can understand and resolve this issue.

[dhiaayachi]

Thanks for reporting this!

This appears to be a known issue related to the way Temporal CLI handles SQLite databases with graceful shutdown. It can lead to log noise upon restarting the server.

Unfortunately, there's no workaround currently available for this behavior. However, we are actively working on improving this in upcoming releases.

For now, you can mitigate the issue by:

• Using a different database: Consider switching to a database like MySQL, PostgreSQL, or Cassandra for your development environment. This will not exhibit the same behavior.
• Enabling verbose logging: This will help you identify the source of the log noise. You can enable verbose logging using the --log-level flag with temporal server start-dev.

We appreciate your feedback and will keep you updated on any progress made regarding this issue.

[dhiaayachi]

Thanks for reporting this issue. The temporal server start-dev command uses an in-memory SQLite database by default, so stopping the server will erase all your Workflows and Task Queues. To retain that information between runs, start the server and specify a database filename using the --db-filename option, like this:

temporal server start-dev --db-filename your_temporal.db

When you stop and restart the Temporal Service and specify the same filename again, your Workflows and other information will be available. Please let me know if you have any further questions.