Users were experiencing kurtosis service logs taking a long time. After running tests, I discovered that a majority of execution time during log processing was spent in the following lines:
Prior to this change, we were sending logs one at a time on an unbuffered channel - unbuffered channels block until the receiving goroutine reads the value. This was causing a lot of time being wasted waiting to send log lines across the channel.
This change implements a LogLineSender that:
uses a buffered go channel (won't block on sending line unless buffer is full)
With this change, the time to read 20 minutes of cl-lighthouse-geth logs with log level set to debug went from 1min53sec to 30.055 seconds. The time to read 2 hours 10 minutes worth of cl-lighthouse debug logs (around 3.4 gb of logs) went from 15min1sec to 3min31 sec. (As a benchmark, cat logs.json on 3.4 gb of logs takes around 2min - on my machine - so much closer) This can likely be improved further by tuning the buffer size and batch amount.
Description
Users were experiencing
kurtosis service logs
taking a long time. After running tests, I discovered that a majority of execution time during log processing was spent in the following lines:Prior to this change, we were sending logs one at a time on an unbuffered channel - unbuffered channels block until the receiving goroutine reads the value. This was causing a lot of time being wasted waiting to send log lines across the channel.
This change implements a
LogLineSender
that:With this change, the time to read 20 minutes of
cl-lighthouse-geth
logs with log level set to debug went from1min53sec
to30.055
seconds. The time to read 2 hours 10 minutes worth ofcl-lighthouse
debug logs (around 3.4 gb of logs) went from15min1sec
to3min31
sec. (As a benchmark,cat logs.json
on3.4 gb
of logs takes around2min
- on my machine - so much closer) This can likely be improved further by tuning the buffer size and batch amount.Is this change user facing?
YES
References:
https://discord.com/channels/783719264308953108/1267837033032974467/1267842228072611881