Open szuecs opened 2 years ago
~14 Months ago I created a similar issue, that is maybe related, but I don't know https://github.com/golang/go/issues/42977
This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .
The complete code paths are tested and run go test -race on every change. I think it would show up or do you think that go run -race should show it instead of tests?
I tried to run it with go run -race, but I get no error if I just use curl and if I do vegeta with 500 rps I get
race: limit on 8128 simultaneously alive goroutines is exceeded, dying
exit status 66
@aclements maybe you want to check the crash log files, as you did in https://github.com/golang/go/issues/42977 , thanks
Context I run 2 binaries inside a docker container, one open source project (first crash) and another closed source (second crash). Both of them crashed with
signal SIGSEGV: segmentation violation
The first crash started with these lines:
The second crash started with these lines:
What version of Go are you using (
go version
)?The open source project runs with Go
1.17.6
. The closed source project with Go1.14.4
. Bothlinux/amd64
.Does this issue reproduce with the latest release?
I can't reproduce it, it's an event that I never observed before and we run about 400 instances across 95 clusters with more than 1M RPS.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
We run 2 processes in one docker container. One process is the open source project, which is a http reverse proxy with some more features like authnz calls to another http endpoint. The second process is the close source project which is a http server handler that does JWT validation, which is called by the first process to validate the token. At runtime both processes crashed. It's a single occurrence and I found https://github.com/golang/go/wiki/LinuxKernelSignalVectorBug which is mentioned in the second crash logs. However I could not reproduce the Bug test https://github.com/golang/go/wiki/LinuxKernelSignalVectorBug#bug-test. I tested with the same gcc version as the kernel.
Gcc version:
Kernel version:
We run all nodes without swap, for example:
The first crash logs go_1st_crash.log
The second crash logs go_2nd_crash.log
What did you expect to see?
no crash
What did you see instead?
a crash