It froze our production server while reading a 288MB log file

[FKPSC]

I installed this log viewer on our production server after switching to pino for logging.

It took a moment to to load but it eventually loaded. I checked a couple of entries and pressed up to reload the file.

I was waiting for the file to load but I got a notification from our monitoring service that the production api server was down. I logged in to AWS to reboot the server from their EC2 dashboard. We were back online after 12 min.

300 MB is not huge for a log file. Even if it cannot handle big files, it might be good to have some check/limits and crash with a message rather than clog the whole system.

[hedhyw]

Ohhh, I'm sorry to hear. I hope this didn't have any big consequences.

I will add some reasonable limit until stream reading is done: https://github.com/hedhyw/json-log-viewer/issues/55

[hedhyw]

Big memory & CPU optimization included in https://github.com/hedhyw/json-log-viewer/releases/tag/v0.4.0

What changed:

1. We still load the whole file (because we need to filter), but JSON parsed lazily (100 rows from config).
2. Threshold between reloads (1 second from config).
3. Reloading indicator.
4. Limited the maximum size of the file (1 Gb from config).
5. I've tested it on 600Mb file. It flies, but changes are significant, so we need to use more.

Please confirm that v0.4.0 works for you as expected!

[FKPSC]

I tried with version v0.4.0 and it still froze the whole server. I am using a staging server this time :) I am reloading the file (hitting up multiple times) and sometimes might trigger the reloading before it is done? Attached is a ~400MB redacted log file for you to replicate. redacted.example.zip

PS: is there any help or version command?

[hedhyw]

Good idea. I will add a version command.

Did you see "reloading..." when you pressed up? Was level "INFO"?

Was loading fast? Please, check the video: Screencast from 01-06-24 21:05:28.webm

[hedhyw]

v0.5.1 should print the version in the footer