Open vpsinghg opened 2 days ago
@vpsinghg Can you try changing the config https://github.com/DiceDB/dice/blob/d999d2d6d7d40e709176b18429a54df56c715462/config/config.go#L125 in the source code itself to 10240 and try to run it again? This was a recent change that was pushed and out of memory could have happened because because the internal store impl is reserving more memory during init.
Let me know if this worked out.
@arpitbbhayani @lucifercr07
I tried to run the TestMain
integration test where StoreMapInitSize
is 1024. Still getting same error and test failed in following packages.
dice/integration_tests/commands/async/main_test.go:
dice/integration_tests/commands/http/main_test.go:
dice/integration_tests/commands/websocket/main_test.go:
dice/internal/eval/main_test.go:
go test -v -race -count=1 --run TestMain ./integration_tests/...
running with config: {{0.0.0.0 7379 300 300 0 1s 100ms 20000 0 allkeys-lfu 0.9 200000000 ./dice-master.aof true false 10 info false false 1024} {dice } {512 51200}}
{"level":"info","version":"0.0.4","port":8739,"message":"DiceDB server is running"}
Starting the test server on port 8739
{"level":"warn","message":"DiceDB is running without authentication. Consider setting a password."}
testing: warning: no tests to run
PASS
{"level":"info","message":"Skipping AOF dump."}
ok github.com/dicedb/dice/integration_tests/commands/async 3.052s [no tests to run]
{{0.0.0.0 7379 300 300 0 1s 100ms 20000 0 allkeys-lfu 0.9 200000000 ./dice-master.aof true false 10 info false false 1024} {dice } {512 51200}}
fatal error: runtime: out of memory
@vpsinghg
Can you share your system memory configs and how much free memory is left?
You can try below commands:
free -h
df -h
@lucifercr07
vikram@Vikram-LT:~/dev/real_engineering/dice$ free -h
total used free shared buff/cache available
Mem: 7.6Gi 1.7Gi 4.8Gi 15Mi 1.2Gi 5.7Gi
Swap: 2.0Gi 0B 2.0Gi
I think issue is when we create watch channel for keys in dstore. Code exits on this step in main.go
. When I run dicedb server with KeysLimit
= 200000, this runs fine.
watchChan := make(chan dstore.WatchEvent, config.DiceConfig.Server.KeysLimit)
@arpitbbhayani @lucifercr07
KeysLimit
?QueryWatchEvent
inside the watchKeys method in query manager
// watchKeys watches for changes in keys and notifies clients.
func (m *Manager) watchKeys(ctx context.Context, watchChan <-chan dstore.QueryWatchEvent) {
for {
select {
case event := <-watchChan:
m.processWatchEvent(event)
case <-ctx.Done():
return
}
}
}
@vpsinghg we'll add this as part of the config file.
@lucifercr07 I have added a PR https://github.com/DiceDB/dice/pull/888 for 1st part. Please review and let me know if there is another approach that you are thinking to avoid this issue.
OS Specification:
Operating System: linux Architecture: amd64 CPU Cores: 8 go version go1.23.0 linux/amd64
Steps I followed:
Install GoLangCI
Local Setup with Custom Config
I am unable to find any thing unusual in the
htop
command.Some memory related specifications