search

arl / statsviz

🚀 Visualise your Go program runtime metrics in real time in the browser
MIT License
3.19k stars 121 forks source link

program killed after 1-2 seconds with go 1.21.5 #124

Closed lyimmi closed 8 months ago

lyimmi commented 8 months ago

Hi,

When I build a project with go version 1.21.5 the process is killed with signal: killed after 1 or 2 seconds.
I can't catch the signal with signal.Notify, as of yet I have not found any workaround, or fix for this.

It can be reproduced with the basic example.

However when I build the project with go version 1.20.7 everything works as expected.

I'll try to dig more into this when I have time. But someone more familiar with this codebase probably could find the problem faster.

arl commented 8 months ago

Hi, thank you for reporting, however I've just tried on linux/amd64 with go 1.21.5 and the default example and i can't reproduce.

Are you running on a container? What platform? Since the process is killed by the OS, you should be able to find more information in the system log.

lyimmi commented 8 months ago

Sorry, I forgot to write the details.

I encountered this issue on Linux Ubuntu 20.04 linux/amd64 simply running go run . in the terminal, however if I build the project the process is not killed. (with go 1.20.7 go run works fine all the time)

I found nothing in the logs so far, still looking for some clues.

edit: After multiple restarts, now the basic example runs ok, but in a real project its still flaky.

edit 2: If I run my project with go run -race . the process is not killed.

arl commented 8 months ago

If with go build nothing happens but with go run it gets killed it looks like go run is killed. It does seem that what's happening is caused by something on your computer. I'd like to help but without being able to reproduce I'm not sure what I could do

when a process gets killed by the os it's either an out of memory or it did something forbidden

lyimmi commented 8 months ago

Never seen this before.
If I leave out the statsviz.Register() call, the issue goes away...

Thank you, feel free to close the issue, just tough I share this with you.

arl commented 8 months ago

Thank you, feel free to close the issue, just tough I share this with you.

You did good. Thank you. I'd really like to find ways to investigate as that's not good. Did you check for dmesg just after that happened? The reason for a process killed should be logged somewhere. Do you have specific ulimit settings? Are you running inside a container?

lyimmi commented 8 months ago

It's not running in a container, the system has default ulimit. I tailed the dmesg while running the program and there was no new record when it was killed, checked the file afterwards as well, nothing... That is what frustrates me, there is no sign of any error in the logs.

I tried the same project on an other machine Linux Ubuntu 23.10 with go version go1.21.6 linux/amd64 and it runs without any issue.
(Updated go on the previous machine too to this version, before I left work. The issue still persisted.)

Tomorrow I'll try to figure out more, if I have time.

lyimmi commented 8 months ago

I found the problem...

On our office computers we run Cortex Traps agents, for some reason this agent flags and kills the go run process when the project is using statviz.

I'm sorry for raising a false issue and thank you for your help.