jkandasa commented 3 years ago

Describe the bug Facing out of memory issue. Running an instance for a week time and facing out of memory issue.

runtime: out of memory: cannot allocate 3854565376-byte block (7536640 in use)
fatal error: out of memory

goroutine 14899 [running]:
runtime.throw(0x4428e9, 0xd)
    /usr/lib/golang/src/runtime/panic.go:1117 +0x5c fp=0xee3524 sp=0xee3510 pc=0x46800
runtime.(*mcache).allocLarge(0x76f9d088, 0xe59d104c, 0x1050001, 0x0)
    /usr/lib/golang/src/runtime/mcache.go:226 +0x268 fp=0xee3554 sp=0xee3524 pc=0x26930
runtime.mallocgc(0xe59d104c, 0x424350, 0x1, 0x118a810)
    /usr/lib/golang/src/runtime/malloc.go:1078 +0x9c4 fp=0xee35b4 sp=0xee3554 pc=0x1c944
runtime.newobject(0x424350, 0x118a810)
    /usr/lib/golang/src/runtime/malloc.go:1177 +0x2c fp=0xee35c8 sp=0xee35b4 pc=0x1ca98
github.com/shirou/gopsutil/v3/process.NewProcessWithContext(0x4cbe84, 0xd14010, 0x14, 0xf15780, 0x0, 0x0)
    /home/jk/go/pkg/mod/github.com/shirou/gopsutil/v3@v3.21.4/process/process.go:183 +0x30 fp=0xee35f0 sp=0xee35c8 pc=0x37a660
github.com/shirou/gopsutil/v3/process.ProcessesWithContext(0x4cbe84, 0xd14010, 0x0, 0x0, 0x0, 0x43e8c8, 0x4)
    /home/jk/go/pkg/mod/github.com/shirou/gopsutil/v3@v3.21.4/process/process_linux.go:1090 +0xac fp=0xee3634 sp=0xee35f0 pc=0x37e968
github.com/shirou/gopsutil/v3/process.Processes(...)
    /home/jk/go/pkg/mod/github.com/shirou/gopsutil/v3@v3.21.4/process/process.go:171
github.com/mycontroller-org/backend/v2/plugin/gateway/provider/system_monitoring.(*Provider).updateProcess(0xc16930)
    /home/jk/mycontroller-org/backend/plugin/gateway/provider/system_monitoring/update_process.go:14 +0x30 fp=0xee3fbc sp=0xee3634 pc=0x3855d8
github.com/mycontroller-org/backend/v2/plugin/gateway/provider/system_monitoring.(*Provider).updateProcess-fm()
    /home/jk/mycontroller-org/backend/plugin/gateway/provider/system_monitoring/update_process.go:13 +0x28 fp=0xee3fc4 sp=0xee3fbc pc=0x38917c
github.com/robfig/cron/v3.FuncJob.Run(0xe055e0)
    /home/jk/go/pkg/mod/github.com/robfig/cron/v3@v3.0.2-0.20210106135023-bc59245fe10e/cron.go:136 +0x1c fp=0xee3fc8 sp=0xee3fc4 pc=0x2ff004
github.com/robfig/cron/v3.(*Cron).startJob.func1(0xd37500, 0x4c94bc, 0xe055e0)
    /home/jk/go/pkg/mod/github.com/robfig/cron/v3@v3.0.2-0.20210106135023-bc59245fe10e/cron.go:312 +0x58 fp=0xee3fdc sp=0xee3fc8 pc=0x304f80
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_arm.s:841 +0x4 fp=0xee3fdc sp=0xee3fdc pc=0x7c728
created by github.com/robfig/cron/v3.(*Cron).startJob
    /home/jk/go/pkg/mod/github.com/robfig/cron/v3@v3.0.2-0.20210106135023-bc59245fe10e/cron.go:310 +0x58

To Reproduce Calling this func every 30 seconds once.

Environment (please complete the following information):

[X] Linux: [paste contents of /etc/os-release and the result of uname -a]


# cat /etc/os-release 
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

uname -a

Linux rpi-170 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021 armv7l GNU/Linux


**Additional context**

$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/jk/.cache/go-build" GOENV="/home/jk/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/jk/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/jk/go" GOPRIVATE="" GOPROXY="direct" GOROOT="/usr/lib/golang" GOSUMDB="off" GOTMPDIR="" GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64" GOVCS="" GOVERSION="go1.16.3" GCCGO="gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/home/jk/mycontroller-org/backend/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1498541019=/tmp/go-build -gno-record-gcc-switches"

$ GOOS=linux GOARCH=arm go build -o mycontroller-gateway cmd/gateway/main.go


Memory details:

free -m

          total        used        free      shared  buff/cache   available

Mem: 924 221 139 46 563 599 Swap: 99 68 31


Command used to start:

./mycontroller-gateway -config gateway.yaml

shirou commented 3 years ago

updating processes takes a time and memory. How many processes were there at that time?

jkandasa commented 3 years ago

updating processes takes a time and memory. How many processes were there at that time?

That time I do not know the count. But I don't touch the system often. It is a detected server.

I just took the process count,

# ps -aux | wc -l
160

# free -m
              total        used        free      shared  buff/cache   available
Mem:            924         199         386          33         338         655
Swap:            99          88          11

pathcl commented 3 years ago

@jkandasa what raspberry pi is it? I might be able to reproduce it

jkandasa commented 3 years ago

@jkandasa what raspberry pi is it? I might be able to reproduce it

@pathcl

# cat /sys/firmware/devicetree/base/model
Raspberry Pi 2 Model B Rev 1.1

shirou / gopsutil

out of memory issue #1071

uname -a

free -m