Error messages about parsing timestamps in event entries when using a time zone offset of "Z"

[tim-goddard-flexcompute]

Fleet version: fleet version: 4.48.2 orbit version: 1.27.0 osquery version: 5.12.1

Web browser and operating system:

Ubuntu 22.04.03

---

💥  Actual behavior

Line pairs like the following appear repeatedly in log files on many of our hosts:

Jul  9 03:27:07 cell1 orbit[2568887]: 2024-07-09T03:27:07Z INF orbit version: 1.27.0
Jul  9 03:27:07 cell1 orbit[2568887]: zerolog: could not write event: fleet_logs.WriteLevel: processLogEntry parsing time: parsing time "2024-07-09T03:27:07Z" as "2006-01-02T15:04:05-07:00": cannot parse "Z" as "-07:00"

No specific harm was observed associated with this error message. This was encountered while attempting to diagnose another issue (which appears to be issue #20168 ), however these errors also appear on hosts which appear to be otherwise functioning.

The time zone on these hosts is set to UTC:

******@cell1:~$ cat /etc/timezone
UTC

🧑‍💻  Steps to reproduce

(lacking suitable non-prod environment to reproduce experimentally)

[lucasmrod]

Hi @tim-goddard-flexcompute! Thanks for the detailed report!

@dantecatalfamo @jahzielv Hi folks!

Pinging you because the error comes from fleetd_logs table (and may be fixed by recent log changes in orbit?)

[PezHub]

QA Notes - Tried to repro (after applying the fix) as best I could by setting my timezone to UTC and letting my Ubuntu device sit for a day and did not see the error mentioned above. Calling it good but please let us know if you still see the errors @tim-goddard-flexcompute once you update to 4.54.0

[PezHub]

I revisited this after smoke testing release candidate 4.54.0 on my Ubuntu PC and did encounter the error. We'll want to revisit this @dantecatalfamo

Jul 16 23:55:25 Ubuntu-PC orbit[3036]: 2024-07-16T23:55:25Z INF enroll failed, retrying error="enroll request: unknown"
Jul 16 23:55:25 Ubuntu-PC orbit[3036]: zerolog: could not write event: fleet_logs.WriteLevel: processLogEntry parsing time: parsing time "2024-07-16T23:55:25Z" as "2006-01-02T15:04:05-07:00": cannot parse "Z" as "-07:00"

[lucasmrod]

Hi @PezHub! fleetd 1.28.0 (that will contain this change) hasn't been released yet (will be released sometime this week).

We currently don't have a milestone for fleetd releases. (It makes it hard if the issue requires both server and fleetd changes because we can't assign two milestones to an issue.)

[PezHub]

ah that makes sense. Thanks @lucasmrod !

[PezHub]

checked logs again after updating fleetd to 1.28.0 and things are looking good.

[fleet-release]

Parsing time, a glitch, UTC finds its niche. Clarity in each log stitch.