search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.78k stars 5.75k forks source link

[Ray Dashboard] Timestamps of logs on Dashboard are wrong #26050

Open RuiLi-5413 opened 2 years ago

RuiLi-5413 commented 2 years ago

What happened + What you expected to happen

I deployed a Ray Cluster with using Ray Kubernetes Operator on AKS and found the timestamps of logs on Ray Dashboard were different from my time zone. e.g. “2022-06-07 12:53:49,756” my time is “2022-06-07 21:53:49”.

I also tried on local K8s, the issue was same.

Versions / Dependencies

image: rayproject/ray:1.13.0

Reproduction script

None

Issue Severity

Medium: It is a significant difficulty but I can work around it.

DmitriGekhtman commented 2 years ago

cc @alanwguo, should be solvable by examining log format logic.

scv119 commented 2 years ago

@rkooo567 to triage.

mkotsalainen commented 8 months ago

We also have this problem two years later. Running date on the head node in our Ray Cluster on EKS gives "Mon Feb 19 02:58:13 PST 2024" even though server is in eu-west-1. Shouldn't it be UTC?

Louanesh commented 2 months ago

Still facing the same issue, would be great to have a parameter in the Ray dashboard for this. Thanks

rkooo567 commented 2 months ago

maybe the logger configured doesn't properly know the timestamp. Anyone interested in tackling this? I don't think we have immediate resources to fix the issue