Chrome hangs loading logs

[amorey]

Does it hang when you view the demo site with the same browser? https://www.kubetail.com/demo

It's probably hanging because the collection of log files you're looking at have a lot of data and the frontend is having trouble processing it. Improving support for large files is #​1 on the todo list.

[nodesocket]

Not sure how much log data. Is there a way I can get that using kubectl? Demo works fine in same Chrome browser.

[amorey]

Great, thanks for checking. If the demo is fine with the browser then it's an issue with the amount of data. Unfortunately the only way to check the size of a log with kubectl is to download the entire file (this is part of the reason why it's a non-trivial problem to fix on the frontend):

kubectl logs -n [namespace] [pod-name] | wc -l 

[nodesocket]

Honestly not that much log data. I guess MetaBase & PostgreSQL kinda has a bit with 5.9k and 3.5k but....

$ kubectl logs -n <namespace> pricesd-75b8bcf79c-gfzcd | wc -l
265
$ kubectl logs -n <namespace> pg-0 | wc -l
3529
$ kubectl logs -n <namespace> metabase-854cb558cd-d97l7 | wc -l
5963

[amorey]

Kubetail merges all the data into a single stream and adds each line as a row to a <table> so the browser is hanging dealing with ~10K table rows with 8 cells each. That's around where I've seen problems in other contexts so it isn't too surprising.

[nodesocket]

@amorey do you have any way to limit log results by relative date (e.g. last hour, or last day)? Also, can probably optimize the DOM rendering quite a bit and lazy load elements into the table which I am sure you know.

Until performance is improved, kinda hard to justify using.

[amorey]

At the API level yes but it isn't currently enabled in the frontend. As a short-term fix, I can configure the frontend to only stream new data by default - would this be helpful for you?

Also, can probably optimize the DOM rendering quite a bit and lazy load elements into the table which I am sure you know.

Yep, lots of room for improvement. Virtualization/windowing adds complexity so I wanted to be sure it was necessary before implementing it.

[nefarius]

My bold claim would be that most users would expect it to behave like tail and by default only start streaming from e.g. the past 10 or so elements form each container into the past, that should do it as a quick fix IMHO.

[nodesocket]

My bold claim would be that most users would expect it to behave like tail and by default only start streaming from e.g. the past 10 or so elements form each container into the past, that should do it as a quick fix IMHO.

That makes sense by default you start from when opened streaming (live). Would be awesome to have a way to select last hour or last 24 hours with a select dropdown or something. Thanks 🙏

[amorey]

That makes sense by default you start from when opened streaming (live). Would be awesome to have a way to select last hour or last 24 hours with a select dropdown or something. Thanks 🙏

That's the plan!

[amorey]

I just published a new release (v0.1.4) that changes the default behavior to "tailing" and adds a dropdown where you can select logs by relative time (e.g. "last 2 hours") or absolute time (e.g. "Feb 3 - Feb 9"). You can install it by upgrading the helm repo:

$ helm repo update kubetail
$ helm upgrade kubetail kubetail/kubetail --namespace kubetail

It's also live on the demo site in case you want to play around with it: https://www.kubetail.com/demo

There's still more work to do to fully support large logs but this should help you at least control the amount of data that the client retrieves from the server.

Thanks again for your help and let me know if you have any suggestions!

[nefarius]

Slight switch of arguments 👇

helm repo update kubetail

[nefarius]

Thanks again for your help and let me know if you have any suggestions!

Thanks for the quick fix, I just deployed and it works as expected! (now all we need is ANSI color support hehe)

[amorey]

@nefarius Thanks for the heads up about the helm command! I fixed it in the original comment.

[amorey]

@nefarius @nodesocket @KyteProject I just published a new release candidate (0.4.0-rc2) that addresses this issue at a deeper level by using "virtualization" to only render the records in view in the DOM. Here's the new code running on the demo site: https://www.kubetail.com/demo

To try it out on your cluster you can cherry pick the 0.4.0-rc2 tag (don't forget to use --reset-values to revert later):

helm repo update kubetail
helm upgrade kubetail kubetail/kubetail --namespace kubetail --set image.tag=0.4.0-rc2

There are still some bugs to fix before it can be merged into the trunk but the new code makes some modifications to the behavior (e.g. introduces "Jump to start", "Jump to end", "Jump to time" buttons) so it would be useful to get some feedback on that as well to make sure we're headed in the right direction.

[nefarius]

I just updated and will let it run for a while 👍

[nefarius]

There are still some bugs to fix before it can be merged into the trunk but the new code makes some modifications to the behavior (e.g. introduces "Jump to start", "Jump to end", "Jump to time" buttons) so it would be useful to get some feedback on that as well to make sure we're headed in the right direction.

IDK if that is already on your radar (probably is) but I noticed that the auto-scroll-to-bottom (whatever the right term) feature rather quickly and regularly fails to do its job and the scrolling simply stops.

[amorey]

Thanks, I'll work on that. The other known issues are:

• New pods aren't getting tailed
• Pressing "pause" then "play" will load events as they are streamed in rather than ordered by timestamp

Let me know any other issues you see big & small. I hope the test is going well!

[amorey]

I just pushed a new version (0.4.0-rc3) that fixes the bugs mentioned previously:

• Autoscroll is working more reliably (though I've seen it shut off occasionally so there's still more work to do)
• New pods will get tailed automatically
• Pressing "play" after being "pause" will order missing events by timestamp then start streaming

helm repo update kubetail
helm upgrade kubetail kubetail/kubetail --namespace kubetail --set image.tag=0.4.0-rc3

It's running live here: https://www.kubetail.com/demo

Let me know if you notice any other bugs big or small!

[amorey]

Quick update - latest version is now 0.4.0-rc4:

helm repo update kubetail
helm upgrade kubetail kubetail/kubetail --namespace kubetail --set image.tag=0.4.0-rc4

https://www.kubetail.com/demo

[amorey]

Quick update - latest version is now 0.4.0-rc5:

helm repo update kubetail
helm upgrade kubetail kubetail/kubetail --namespace kubetail --set image.tag=0.4.0-rc5

https://www.kubetail.com/demo

Please check it out and let me know if you notice any bugs. I think this release candidate reaches feature parity with the current stable version so unless there are any issues, I'll merge with the main branch and publish a new stable version soon.

[amorey]

I pushed a new stable version (0.4.0) based on the last rc. Here's the command to update (using --reset-values to clear out previous use of --set):

helm repo update kubetail
helm upgrade kubetail kubetail/kubetail --namespace kubetail --reset-values

Thanks for your help debugging this issue!

[nodesocket]

I just upgraded to 0.4.0. Will report any issues. Thanks 👍🏻

[nodesocket]

@amorey getting a JS error with 0.4.0:

[amorey]

Thanks for the screenshot. Did you get the bug on initial load? Do you see the bug with all your logs or just some? Any other details you can provide would be very helpful.

[amorey]

@nodesocket Try the new version (0.4.1):

helm repo update kubetail
helm upgrade kubetail kubetail/kubetail --namespace kubetail

Let me know if it's still giving you an error.

[nodesocket]

@amorey that seemed to fix it. Thanks!