search

joe-at-startupmedia / pmon3

Golang Production Process Manager
10 stars 0 forks source link

large amount of processes trigger CLI response memory error with default shmem tag #33

Closed joe-at-startupmedia closed 2 months ago

joe-at-startupmedia commented 2 months ago

The current solution is to use posix_mq for managing large sets of processes:

BUILD_FLAGS="-tags posix_mq,cgo_sqlite" make build_cgo

Error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x704d00]

goroutine 24 [running]:
github.com/apache/arrow/go/arrow/memory.(*Buffer).Reserve(0xc00018d1a8, 0x0?)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/apache/arrow/go/arrow/memory/buffer.go:92 +0x80
github.com/apache/arrow/go/arrow/memory.(*Buffer).resize(0xc00018d1a8, 0x15f2, 0x85?)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/apache/arrow/go/arrow/memory/buffer.go:110 +0xe7
github.com/apache/arrow/go/arrow/memory.(*Buffer).ResizeNoShrink(...)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/apache/arrow/go/arrow/memory/buffer.go:105
github.com/joe-at-startupmedia/shmemipc.(*ShmProvider).Read(0xc00018d170)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/joe-at-startupmedia/shmemipc/shmemipc.go:48 +0x18a
github.com/joe-at-startupmedia/shmemipc.(*IpcRequester).Read(...)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/joe-at-startupmedia/shmemipc/requester.go:22
github.com/joe-at-startupmedia/xipc/mem.(*Requester).Read(0xc00009fba0?)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/joe-at-startupmedia/xipc/mem/requester.go:33 +0x19
github.com/joe-at-startupmedia/xipc.WaitForProto({0x8cdde0, 0xc00021c160}, {0x8c9040, 0xc000220500})
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/joe-at-startupmedia/xipc/requester.go:84 +0x73
github.com/joe-at-startupmedia/xipc/mem.(*Requester).WaitForProto(0x1dcd6500?, {0x8c9040?, 0xc000220500?})
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/vendor/github.com/joe-at-startupmedia/xipc/mem/requester.go:66 +0x2d
pmon3/cli/cmd/base.waitForResponse(...)
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/cli/cmd/base/base.go:126
pmon3/cli/cmd/base.GetResponse.func1()
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/cli/cmd/base/base.go:74 +0xf8
created by pmon3/cli/cmd/base.GetResponse in goroutine 1
    /home/runner/work/pmon3/pmon3/__PROJECT_CHECKOUT_DIR__/cli/cmd/base/base.go:71 +0x18
joe-at-startupmedia commented 2 months ago

Offending line https://github.com/apache/arrow/blob/bc219186db40/go/arrow/memory/buffer.go#L92

Shmem dependency: https://github.com/joe-at-startupmedia/shmemipc/blob/e3a0863ffabedcb2480378732beed37df7ce4af5/go.mod#L6

joe-at-startupmedia commented 2 months ago

new stacktrace with updated dependencies:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x708540]

goroutine 9 [running]:
github.com/joe-at-startupmedia/go-arrow/arrow/memory.(*Buffer).Reserve(0xc0001fe218, 0x467a78?)
    /root/go/pkg/mod/github.com/joe-at-startupmedia/go-arrow@v0.0.1/arrow/memory/buffer.go:124 +0x80
github.com/joe-at-startupmedia/go-arrow/arrow/memory.(*Buffer).resize(0xc0001fe218, 0x15f2, 0x65?)
    /root/go/pkg/mod/github.com/joe-at-startupmedia/go-arrow@v0.0.1/arrow/memory/buffer.go:142 +0xe7
github.com/joe-at-startupmedia/go-arrow/arrow/memory.(*Buffer).ResizeNoShrink(...)
    /root/go/pkg/mod/github.com/joe-at-startupmedia/go-arrow@v0.0.1/arrow/memory/buffer.go:137
github.com/joe-at-startupmedia/shmemipc.(*ShmProvider).Read(0xc0001fe1e0)
    /root/go/pkg/mod/github.com/joe-at-startupmedia/shmemipc@v0.0.10/shmemipc.go:48 +0x18a
github.com/joe-at-startupmedia/shmemipc.(*IpcRequester).Read(...)
    /root/go/pkg/mod/github.com/joe-at-startupmedia/shmemipc@v0.0.10/requester.go:22
github.com/joe-at-startupmedia/xipc/mem.(*Requester).Read(0xc00023c000?)
    /root/go/pkg/mod/github.com/joe-at-startupmedia/xipc@v0.0.11/mem/requester.go:33 +0x19
github.com/joe-at-startupmedia/xipc.WaitForProto({0x8d3000, 0xc00020dfc0}, {0x8ce220, 0xc000240480})
    /root/go/pkg/mod/github.com/joe-at-startupmedia/xipc@v0.0.11/requester.go:84 +0x73
github.com/joe-at-startupmedia/xipc/mem.(*Requester).WaitForProto(0x1dcd6500?, {0x8ce220?, 0xc000240480?})
    /root/go/pkg/mod/github.com/joe-at-startupmedia/xipc@v0.0.11/mem/requester.go:66 +0x2d
pmon3/cli/cmd/base.waitForResponse(...)
    /root/pmon3/cli/cmd/base/base.go:126
pmon3/cli/cmd/base.GetResponse.func1()
    /root/pmon3/cli/cmd/base/base.go:74 +0xf8
created by pmon3/cli/cmd/base.GetResponse in goroutine 1
    /root/pmon3/cli/cmd/base/base.go:71 +0x189
joe-at-startupmedia commented 2 months ago

This issue was related to the MaxMsgSize only being 4096 which is surpassed with a large amount of process (>= 12) https://github.com/joe-at-startupmedia/pmon3/blob/01a9afac21cda19e2c8bc6483ec1e76b7e12a076/pmond/god/responder_helper_mem.go#L16