Timestamp in Logs is 8 Hours Behind Despite Correct Timezone Settings

ZimaBlueee commented 2 months ago

Self Checks

[X] This is only for bug report, if you would like to ask a question, please head to Discussions.
[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I deployed Dify using Docker, and have set the timezone to UTC+8 on the web interface. I also added TZ: Asia/Shanghai to the docker-compose. yaml file. The time inside the container is correct, but the timestamps in the logs are still 8 hours behind. How can this be fixed?

The time inside the container is correct. Can the backend service timezone be configured to match the container timezone?

✔️ Expected Behavior

The log timestamp for the back-end container is set to Zone 8

❌ Actual Behavior

The log timestamp of the back-end container can only be utc time

dosubot[bot] commented 2 months ago

Hey, @ZimaBlueee! 👋 I'm here to help you with bug fixes, answering questions, and becoming a contributor. Let's squash those bugs together!

To ensure that the log timestamps match the container's timezone, you need to configure both the container and the backend service correctly. Here are the steps to achieve this:

Set the Container Timezone: Ensure that the container's timezone is set correctly in your docker-compose.yml file.

version: '3.8'

services:
  web:
    image: your_image_name
    environment:
      - TZ=Asia/Shanghai
    # Other service configurations...

Configure the Backend Service Timezone: Modify the backend service to use the specified timezone for logging. This involves setting the LOG_TZ configuration in the api/configs/feature/__init__.py file and ensuring the logging configuration uses this timezone.

In api/configs/feature/__init__.py:

class LoggingConfigs(BaseModel):
    """
    Logging configs
    """

    LOG_LEVEL: str = Field(
        description='Log output level, default to INFO.'
                    'It is recommended to set it to ERROR for production.',
        default='INFO',
    )

    LOG_FILE: Optional[str] = Field(
        description='logging output file path',
        default=None,
    )

    LOG_FORMAT: str = Field(
        description='log format',
        default='%(asctime)s.%(msecs)03d %(levelname)s [%(threadName)s] [%(filename)s:%(lineno)d] - %(message)s',
    )

    LOG_DATEFORMAT: Optional[str] = Field(
        description='log date format',
        default=None,
    )

    LOG_TZ: Optional[str] = Field(
        description='specify log timezone, eg: America/New_York',
        default='Asia/Shanghai',
    )

Update the Logging Configuration: Ensure the logging configuration in api/app.py uses the specified timezone.

In api/app.py:

def create_app() -> Flask:
    app = create_flask_app_with_configs()

    app.secret_key = app.config['SECRET_KEY']

    log_handlers = None
    log_file = app.config.get('LOG_FILE')
    if log_file:
        log_dir = os.path.dirname(log_file)
        os.makedirs(log_dir, exist_ok=True)
        log_handlers = [
            RotatingFileHandler(
                filename=log_file,
                maxBytes=1024 * 1024 * 1024,
                backupCount=5
            ),
            logging.StreamHandler(sys.stdout)
        ]

    logging.basicConfig(
        level=app.config.get('LOG_LEVEL'),
        format=app.config.get('LOG_FORMAT'),
        datefmt=app.config.get('LOG_DATEFORMAT'),
        handlers=log_handlers,
        force=True
    )
    log_tz = app.config.get('LOG_TZ')
    if log_tz:
        from datetime import datetime
        import pytz
        timezone = pytz.timezone(log_tz)

        def time_converter(seconds):
            return datetime.utcfromtimestamp(seconds).astimezone(timezone).timetuple()

        for handler in logging.root.handlers:
            handler.formatter.converter = time_converter

    initialize_extensions(app)
    register_blueprints(app)
    register_commands(app)

    return app

By following these steps, you should be able to ensure that the log timestamps match the container's timezone ^[1]^[2].

To continue talking to Dosu, mention @dosu.

ZimaBlueee commented 2 months ago

I use the official docker image, no manual compilation, can not modify the code inside the container

MSZheng20 commented 2 months ago

if os.name == "nt": os.system('tzutil /s "China Standard Time"') else: os.environ['TZ'] = 'Asia/Shanghai' time.tzset()

nadirvishun commented 3 weeks ago

The method mentioned above, using os.environ['TZ']= 'Asia/Shanghai' is effective, but it is not recommended to modify it this way. The timezone designed by dify is UTC, and it is only when displaying on the frontend web that dayjs is used to convert it to the current timezone. This is an approach that supports internationalization.

There are two types of logs: one is the built-in log of Flask, and the other is the built-in log of Gunicorn. Docker environments default to using Gunicorn, but you can switch by setting the DEBUG environment variable: https://github.com/langgenius/dify/blob/603a89055cb6746650171969fbb59a310e717a8e/api/docker/entrypoint.sh#L29-L39

use flask run xxx
- .env file can add LOG_TZ=Asia/Shanghai
- docker-compose.yaml add LOG_TZ: ${LOG_TZ:-Asia/Shanghai} at &shared-api-worker-env
- then the flask log can change the timezone： https://github.com/langgenius/dify/blob/603a89055cb6746650171969fbb59a310e717a8e/api/app.py#L127-L139
usegunicorn xxx
- the flask logs are no longer valid because gunicorn has its own logging system, which is strongly associated with the project's timezone. Currently, the system is set to os.environ['TZ'] = 'UTC'
- I haven't found a good solution yet, the times in the database are all in UTC, and using UTC for the logs would also be acceptable

langgenius / dify