runtime: Go randomly crashes on Windows with no stack trace, just "unknown pc" and memory/registry dump

[jasontconnell]

What version of Go are you using (go version)?

$ go version
go version go1.14.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

OS is Windows 10 Pro 10.0.18362

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\jconnell\AppData\Local\go-build
set GOENV=C:\Users\jconnell\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\jconnell\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\jconnell\go\src\github.com\jasontconnell\chashprms\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\jconnell\AppData\Local\Temp\go-build801242386=/tmp/go-build -gno-record-gcc-switches
GOROOT/bin/go version: go version go1.14.3 windows/amd64
GOROOT/bin/go tool compile -V: compile version go1.14.3

What did you do?

I don't think the code matters. It has crashed with multiple different programs. But here's an example. It's more in the "running it repeatedly" that breaks it. I've tested it on Go 1.13 and it runs fine. The earliest version that I've tried that showed the error was Go 1.14.3 as displayed in the version section of this post. It is reproducible on Go 1.15 as well.

So my manager was reporting a crash, a memory dump. I wasn't able to reproduce it locally. But then I wrote the simple program

https://play.golang.org/p/KMabbeiz1hW

And a powershell script

For ($i = 0; $i -lt 100; $i++) { Write-Host $i .\chashprms "$i is the value" }

Run go build and run.ps1.

What did you expect to see?

If I run the same code in Docker using the golang:1.15 image, it runs fine. Runs through 100 instances and always prints out the hash. I would expect to see 100 hashes printed. In Go 1.13 it works fine (the latest I tested where it works).

What did you see instead?

In Go 1.14.3 (the earliest that I tested where it crashes), I see a memory and registry dump after a few calls:

Exception 0xc0000005 0x0 0x7ffd4c110fff 0x910000 PC=0x910000

runtime: unknown pc 0x910000 stack: frame={sp:0x84f6a0, fp:0x0} stack=[0x0,0x84ff00) 000000000084f5a0: 000000000098d7c0 000000000084f610 000000000084f5b0: 0000000000000000 000000000084f6a8 000000000084f5c0: 0000000000989710 000000000084f600 000000000084f5d0: 0000000000000000 00007ffd48e60108 000000000084f5e0: 0000000000000000 0000000000963880 000000000084f5f0: 000000000084f738 00007ffd49d194d1 000000000084f600: 00007ffd49d194cb 00007ffd4bf3ba6f 000000000084f610: 00007ffd4bf200d8 0000000000975c50 000000000084f620: 0000000000000005 000000000084f670 000000000084f630: 00007ffd49d17900 00007ffd4bf51810 000000000084f640: 0000000000963880 0000000000000000 000000000084f650: 0000000000050005 00007ffd49d194cb 000000000084f660: 000000000000001b 0000000000000000 000000000084f670: 0000000000000000 00007ffd4bf200d8 000000000084f680: 0000000000000000 00007ffd49ac0000 000000000084f690: 0000000000000001 00007ffd4bf3c734 000000000084f6a0: <0000000000000001 0000000000000000 000000000084f6b0: 00007ffd0000ebb0 000000000084f7a8 000000000084f6c0: 00000000009897e0 000000000084f728 000000000084f6d0: 0000000000250024 00007ffd48e7b816 000000000084f6e0: 000000000098d690 00007ffd4c06c500 000000000084f6f0: 0000000000963880 00007ffd4c06ea5c 000000000084f700: 000000000000094c 00007ffd48deb2d8 000000000084f710: 00007ffd48de9138 00007ffd4c06c528 000000000084f720: 00007ffd4c07ec6a 00007ffd48de0000 000000000084f730: 00007ffd4c070f8c 00007ffd4bfa0d80 000000000084f740: 000000001c1dc789 000000000098d818 000000000084f750: 000000000084fab0 00000000c0000135 000000000084f760: 0000000000000000 0000000000000040 000000000084f770: 0000000000000004 00007ffd4c0852f0 000000000084f780: 0000000000000001 000000000084f900 000000000084f790: 0000000000975c50 00007ffd4bf4e2a8 runtime: unknown pc 0x910000 stack: frame={sp:0x84f6a0, fp:0x0} stack=[0x0,0x84ff00) 000000000084f5a0: 000000000098d7c0 000000000084f610 000000000084f5b0: 0000000000000000 000000000084f6a8 000000000084f5c0: 0000000000989710 000000000084f600 000000000084f5d0: 0000000000000000 00007ffd48e60108 000000000084f5e0: 0000000000000000 0000000000963880 000000000084f5f0: 000000000084f738 00007ffd49d194d1 000000000084f600: 00007ffd49d194cb 00007ffd4bf3ba6f 000000000084f610: 00007ffd4bf200d8 0000000000975c50 000000000084f620: 0000000000000005 000000000084f670 000000000084f630: 00007ffd49d17900 00007ffd4bf51810 000000000084f640: 0000000000963880 0000000000000000 000000000084f650: 0000000000050005 00007ffd49d194cb 000000000084f660: 000000000000001b 0000000000000000 000000000084f670: 0000000000000000 00007ffd4bf200d8 000000000084f680: 0000000000000000 00007ffd49ac0000 000000000084f690: 0000000000000001 00007ffd4bf3c734 000000000084f6a0: <0000000000000001 0000000000000000 000000000084f6b0: 00007ffd0000ebb0 000000000084f7a8 000000000084f6c0: 00000000009897e0 000000000084f728 000000000084f6d0: 0000000000250024 00007ffd48e7b816 000000000084f6e0: 000000000098d690 00007ffd4c06c500 000000000084f6f0: 0000000000963880 00007ffd4c06ea5c 000000000084f700: 000000000000094c 00007ffd48deb2d8 000000000084f710: 00007ffd48de9138 00007ffd4c06c528 000000000084f720: 00007ffd4c07ec6a 00007ffd48de0000 000000000084f730: 00007ffd4c070f8c 00007ffd4bfa0d80 000000000084f740: 000000001c1dc789 000000000098d818 000000000084f750: 000000000084fab0 00000000c0000135 000000000084f760: 0000000000000000 0000000000000040 000000000084f770: 0000000000000004 00007ffd4c0852f0 000000000084f780: 0000000000000001 000000000084f900 000000000084f790: 0000000000975c50 00007ffd4bf4e2a8 rax 0x7ffd48debe7c rbx 0x7ffd48debe7a rcx 0x41 rdi 0xffffffffffbadd11 rsi 0x0 rbp 0x84f900 rsp 0x84f6a0 r8 0x0 r9 0x0 r10 0x0 r11 0x94b r12 0x7ffd4bf20000 r13 0x0 r14 0x7ffd48debe7c r15 0xc000007a rip 0x910000 rflags 0x10202 cs 0x33 fs 0x53 gs 0x2b

We are a Windows shop and do .NET dev and these tools I wrote help with that, I would love to be able to keep them in Go :D For now I have to revert them to 1.13ish but I'll be watching this closely. I am available and on the Gopher slack, if you need more information I'll be happy to provide.

[davecheney]

Is there any anti virus software running on that machine that may be modifying or interposing on your actions?

[jasontconnell]

Yes we apparently run "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48". I had wondered that but Go 1.13 was fine so I failed to mention.

[davecheney]

Are you able to confirm if the problem occurs when "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48" is uninstalled or disabled?

[jasontconnell]

I am unable to modify the settings but the real time protection is not enabled. I can try running it on another computer that isn't work controlled.

[dmitshur]

/cc @aclements @randall77 @mknyszek @prattmic per owners.

[jasontconnell]

https://github.com/jasontconnell/chashprms

For reference. I'll be running it on my other computer shortly. Will report back.

[jasontconnell]

Are you able to confirm if the problem occurs when "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48" is uninstalled or disabled?

My home computer is windows 10 pro but a different build. When i ran that with 1.15 it ran fine! So you're on to something @davecheney but I'm updating windows at the moment and will run again when they're the same version.

Edit: this is taking forever. I'll have an update tomorrow.

[jasontconnell]

Home PC Update: It's updated but not the same exact build number. Still runs fine in 1.15.

I have BitDefender on my home PC but that's apparently much better behaved, if an AV is to blame. It may be partially, but I can't say 100% since it works fine in 1.13.

[ianlancetaylor]

Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1.

[alexbrainman]

Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1.

I had problem that was affected by this - #36492 - but I don't see crashes, I see hangs. But, still, you could try, it won't help.

Alex

[jasontconnell]

Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1.

I had problem that was affected by this - #36492 - but I don't see crashes, I see hangs. But, still, you could try, it won't help.

Alex

Very in depth issue! Yeah, unless I'm doing it wrong (possible) @ianlancetaylor the ascyncpreemptoff=1 setting didn't alleviate the crash. I failed to mention that it also hangs and I can't ctrl+c out of it, I have to close the powershell window.

I've now run it in Go 1.13.15 and it runs perfectly. According to webroot the version of the software we're running was released on May 13 of this year, but I don't know when it was installed. This only just started happening this week, and I was running 1.14+ since Feb 26.

I have a question out to IT about when the webroot update might have been pushed, I will edit this comment with pertinent info when I get a response.

Edit: This lines up with when the issues started happening. I can uninstall webroot after talking to IT. I will do that soon.

[jasontconnell]

AV Update: Was able to uninstall Webroot (IT granted me test permissions to remove it temporarily). Of course it runs fine.

Webroot was also causing issues compiling more than once with version 1.13.15. That update 1.1.226 in the screenshot above is most likely to blame, since I don't remember any issues (that weren't pebcak errors) before this week.

We are going to reinstall it and try to allow-list some paths where I and other devs typically use Go programs. That will not be tonight though. I'll be back.

[alexbrainman]

... But, still, you could try, it won't help.

I was meant to say "But, still, you could try, it won't hurt.". Sorry for confusion.

AV Update: Was able to uninstall Webroot (IT granted me test permissions to remove it temporarily). Of course it runs fine.

Looks like you are wining your battles.

Alex

[jasontconnell]

So... I'm not sure if this will require further digging. At least from my end. With the help of IT we re-installed webroot. It has the same Core update from a screenshot earlier that I was pretty sure would have been causing the issue. It seems to behave much better now even with 1.15, with the same program linked earlier, spitting out 100 hashes.

IT's thinking is that I didn't get the latest definitions or something. We don't have real time protection turned on so it just shouldn't have an effect either way. I am clueless as to a possible cause, yet it was consistently "randomly" reproducible.

So I'm currently not able to reproduce the originally reported issue. I've saved this issue so that if it happens again, and this is closed, I'll reference it. I hope it's just done though. I won't close it but someone on the Go team can if deemed appropriate.

Thanks for the help, I effing love Go :)

[jasontconnell]

Hold the phone, it's happening again. Bombarded with other stuff right now. It was fine until I logged back on to our VPN, I'm not sure if that had anything to do with it, probably coincidence.

[bhperry]

I am having very similar issues. Also running WebRoot, but shutting it down doesn't seem to affect anything. Originally thought it was a problem with my cgo setup, but now I've been testing with a super basic program that just prints a character and then exits. Sometimes it works, sometimes it hits the runtime: unknown pc error and hangs.

I've been developing on WSL with an X server to run a game, but I'd really like to just get it compiling natively on windows. I've tried cross-compiling as well from WSL, and get similar results. Sometimes the build works, other times it does not (with zero code changes)

Running go version 1.15.1 amd64 right now. Tried 1.13.15 and 1.14.8, but neither of those worked. 1.13 actually didn't really work at all, couldn't even run go env without it crashing.

[networkimprov]

Another report of this: #41138

[bhperry]

Ok here's something strange. I noticed that an exe I had cross-compiled (and wasn't working) suddenly started working just fine. So I started time stamping builds and saving all of them. They mostly crash when I build them. But when I go back and run them later (an hour or so, but doesn't seem to be consistent), all the sudden they work.

[networkimprov]

try this and post its output (assuming the Windows command prompt)

set GOTRACEBACK=system
your_app

[bhperry]

When I ran in windows cmd it just gave the runtime: unknown pc and stack dump. Ran in an MYSY2 bash terminal and got the following:

GOTRACEBACK=system (MSYS2)

``` Exception 0xc0000005 0x0 0x7ffb2f6f2fff 0x1beb7f80000 PC=0x1beb7f80000 signal arrived during external code execution runtime.cgocall(0x6a0040, 0xc000111ee0, 0x3fe0000000000000) C:/Go/src/runtime/cgocall.go:133 +0x55 fp=0xc000111eb0 sp=0xc000111e78 pc=0x5347f5 github.com/go-gl/glfw/v3.3/glfw._Cfunc_glfwInit(0x3fd3d3d300000000) _cgo_gotypes.go:1435 +0x50 fp=0xc000111ee0 sp=0xc000111eb0 pc=0x642c10 github.com/go-gl/glfw/v3.3/glfw.Init(0x577798, 0x6a6600) C:/Users/bhper/go/src/github.com/go-gl/glfw/v3.3/glfw/glfw.go:36 +0x29 fp=0xc000111f20 sp=0xc000111ee0 pc=0x6450a9 github.com/faiface/pixel/pixelgl.Run(0x719028) C:/Users/bhper/go/src/github.com/faiface/pixel/pixelgl/run.go:27 +0x3e fp=0xc000111f70 sp=0xc000111f20 pc=0x64ebfe main.main() C:/Users/bhper/go/src/github.com/bhperry/space-miner/main.go:117 +0x34 fp=0xc000111f88 sp=0xc000111f70 pc=0x681294 runtime.main() C:/Go/src/runtime/proc.go:204 +0x209 fp=0xc000111fe0 sp=0xc000111f88 pc=0x56ad09 runtime.goexit() C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x598b61 goroutine 2 [force gc (idle)]: runtime.gopark(0x719348, 0x82e9e0, 0x1411, 0x1) C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000045fb0 sp=0xc000045f90 pc=0x56b0da runtime.goparkunlock(...) C:/Go/src/runtime/proc.go:312 runtime.forcegchelper() C:/Go/src/runtime/proc.go:255 +0xcd fp=0xc000045fe0 sp=0xc000045fb0 pc=0x56af6d runtime.goexit() C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000045fe8 sp=0xc000045fe0 pc=0x598b61 created by runtime.init.6 C:/Go/src/runtime/proc.go:243 +0x3c goroutine 3 [GC sweep wait]: runtime.gopark(0x719348, 0x82eac0, 0x140c, 0x1) C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000047fa8 sp=0xc000047f88 pc=0x56b0da runtime.goparkunlock(...) C:/Go/src/runtime/proc.go:312 runtime.bgsweep(0xc00000c150) C:/Go/src/runtime/mgcsweep.go:163 +0xb2 fp=0xc000047fd8 sp=0xc000047fa8 pc=0x5558d2 runtime.goexit() C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000047fe0 sp=0xc000047fd8 pc=0x598b61 created by runtime.gcenable C:/Go/src/runtime/mgc.go:217 +0x67 goroutine 4 [GC scavenge wait]: runtime.gopark(0x719348, 0x82ebe0, 0x140d, 0x1) C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000055f78 sp=0xc000055f58 pc=0x56b0da runtime.goparkunlock(...) C:/Go/src/runtime/proc.go:312 runtime.bgscavenge(0xc00000c150) C:/Go/src/runtime/mgcscavenge.go:265 +0xe5 fp=0xc000055fd8 sp=0xc000055f78 pc=0x5538a5 runtime.goexit() C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000055fe0 sp=0xc000055fd8 pc=0x598b61 created by runtime.gcenable C:/Go/src/runtime/mgc.go:218 +0x89 goroutine 5 [finalizer wait]: runtime.gopark(0x719348, 0x8618c8, 0x1ffff1410, 0x1) C:/Go/src/runtime/proc.go:306 +0xfa fp=0xc000049f58 sp=0xc000049f38 pc=0x56b0da runtime.goparkunlock(...) C:/Go/src/runtime/proc.go:312 runtime.runfinq() C:/Go/src/runtime/mfinal.go:175 +0xb2 fp=0xc000049fe0 sp=0xc000049f58 pc=0x54a5f2 runtime.goexit() C:/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000049fe8 sp=0xc000049fe0 pc=0x598b61 created by runtime.createfing C:/Go/src/runtime/mfinal.go:156 +0x68 rax 0x7ffb2a7b0158 rbx 0x7ffb2a7b0156 rcx 0x77 rdi 0xffffffffffbadd11 rsi 0x0 rbp 0x0 rsp 0xb75e9fed90 r8 0x97c r9 0x97c r10 0x97c r11 0x97c r12 0xc000007a r13 0x0 r14 0x7ffb2a7b0158 r15 0x7ffb2f4f0000 rip 0x1beb7f80000 rflags 0x10202 cs 0x33 fs 0x53 gs 0x2b exit status 2 ```

Running with set GOTRACEBACK=all did work on windows cmd, so here's that as well:

GOTRACEBACK=all (win cmd)

``` Exception 0xc0000005 0x0 0x7ffb2f6f2fff 0x212ba160000 PC=0x212ba160000 signal arrived during external code execution github.com/go-gl/glfw/v3.3/glfw._Cfunc_glfwInit(0x3fd3d3d300000000) _cgo_gotypes.go:1435 +0x50 github.com/go-gl/glfw/v3.3/glfw.Init(0xaf7798, 0xc26600) C:/Users/bhper/go/src/github.com/go-gl/glfw/v3.3/glfw/glfw.go:36 +0x29 github.com/faiface/pixel/pixelgl.Run(0xc99028) C:/Users/bhper/go/src/github.com/faiface/pixel/pixelgl/run.go:27 +0x3e main.main() C:/Users/bhper/go/src/github.com/bhperry/space-miner/main.go:117 +0x34 rax 0x7ffb2a7b0158 rbx 0x7ffb2a7b0156 rcx 0x77 rdi 0xffffffffffbadd11 rsi 0x0 rbp 0x0 rsp 0xa1821feb50 r8 0x97c r9 0x97c r10 0x97c r11 0x97c r12 0xc000007a r13 0x0 r14 0x7ffb2a7b0158 r15 0x7ffb2f4f0000 rip 0x212ba160000 rflags 0x10202 cs 0x33 fs 0x53 gs 0x2b exit status 2 ```

[networkimprov]

Ben, that looks like a different problem from the one originally posted here. Can you file a new issue, and include:

1. a complete program that produces the error
2. the logs you posted above

EDIT: also try it with this env var: GODEBUG=asyncpreemptoff=1

[bhperry]

with GODEBUG=asyncpreemptoff=1 I'm just getting the stack dump (same whether or not GOTRACEBACK is set, not sure if that matters)

asyncpreemptoff

``` Exception 0xc0000005 0x0 0x7ffb2f6f0fff 0x19fc1a40000 PC=0x19fc1a40000 runtime: unknown pc 0x19fc1a40000 stack: frame={sp:0x31b43fe010, fp:0x0} stack=[0x0,0x31b43ff6f0) 00000031b43fdf10: 00000031b43fdf58 00000031b43fdf80 00000031b43fdf20: 00000031b43fdf48 00000031b43fdf40 00000031b43fdf30: 00000031b43fdf44 0000019f9af50000 00000031b43fdf40: 0000000000000000 0000000000000000 00000031b43fdf50: 0000000000000000 0000000000000005 00000031b43fdf60: 00000031b43fe0a8 00007ffb2d2a9f38 00000031b43fdf70: 0000019f9ad84250 00007ffb2f5146ce 00000031b43fdf80: 00007ffb2f4f00e8 0000019f9ad980b0 00000031b43fdf90: 00007ffb2d2a9f32 00000031b43fdfe0 00000031b43fdfa0: 004f0044004e0049 0053005c00530057 00000031b43fdfb0: 0000019f9ad84250 0000000000000000 00000031b43fdfc0: 0000019f9adc3660 006c006400050005 00000031b43fdfd0: 00007ffb2d2a9f32 0000000000000000 00000031b43fdfe0: 00007ffb00000000 00007ffb2f4f00e8 00000031b43fdff0: 0000000000000000 0000000000000000 00000031b43fe000: 0000000000000001 00007ffb2f513783 00000031b43fe010: <0000019f00000001 0000000000000000 00000031b43fe020: 0000000000000000 00000031b43fe118 00000031b43fe030: 0000000000000000 0000000000000000 00000031b43fe040: 0000000000000000 0000000000000000 00000031b43fe050: 0000019f9ad980b0 00007ffb2f640f00 00000031b43fe060: 0000019f9ad84250 00007ffb2f643520 00000031b43fe070: 000000000000097d 00007ffb2c32ccb8 00000031b43fe080: 00007ffb2c32a148 00007ffb2f640f28 00000031b43fe090: 00007ffb2f653d4f 00007ffb2c320000 00000031b43fe0a0: 00007ffb2f645b14 00007ffb2f570aa0 00000031b43fe0b0: 0000000000000000 0000000000000000 00000031b43fe0c0: 0000000000000000 0000000000000000 00000031b43fe0d0: 0000019f9ad97e50 0000000000000040 00000031b43fe0e0: 0000000000000003 00007ffb2f65a3f0 00000031b43fe0f0: 0000000000000001 00000031b43fe300 00000031b43fe100: 0000019f9adc3660 00007ffb2f551448 runtime: unknown pc 0x19fc1a40000 stack: frame={sp:0x31b43fe010, fp:0x0} stack=[0x0,0x31b43ff6f0) 00000031b43fdf10: 00000031b43fdf58 00000031b43fdf80 00000031b43fdf20: 00000031b43fdf48 00000031b43fdf40 00000031b43fdf30: 00000031b43fdf44 0000019f9af50000 00000031b43fdf40: 0000000000000000 0000000000000000 00000031b43fdf50: 0000000000000000 0000000000000005 00000031b43fdf60: 00000031b43fe0a8 00007ffb2d2a9f38 00000031b43fdf70: 0000019f9ad84250 00007ffb2f5146ce 00000031b43fdf80: 00007ffb2f4f00e8 0000019f9ad980b0 00000031b43fdf90: 00007ffb2d2a9f32 00000031b43fdfe0 00000031b43fdfa0: 004f0044004e0049 0053005c00530057 00000031b43fdfb0: 0000019f9ad84250 0000000000000000 00000031b43fdfc0: 0000019f9adc3660 006c006400050005 00000031b43fdfd0: 00007ffb2d2a9f32 0000000000000000 00000031b43fdfe0: 00007ffb00000000 00007ffb2f4f00e8 00000031b43fdff0: 0000000000000000 0000000000000000 00000031b43fe000: 0000000000000001 00007ffb2f513783 00000031b43fe010: <0000019f00000001 0000000000000000 00000031b43fe020: 0000000000000000 00000031b43fe118 00000031b43fe030: 0000000000000000 0000000000000000 00000031b43fe040: 0000000000000000 0000000000000000 00000031b43fe050: 0000019f9ad980b0 00007ffb2f640f00 00000031b43fe060: 0000019f9ad84250 00007ffb2f643520 00000031b43fe070: 000000000000097d 00007ffb2c32ccb8 00000031b43fe080: 00007ffb2c32a148 00007ffb2f640f28 00000031b43fe090: 00007ffb2f653d4f 00007ffb2c320000 00000031b43fe0a0: 00007ffb2f645b14 00007ffb2f570aa0 00000031b43fe0b0: 0000000000000000 0000000000000000 00000031b43fe0c0: 0000000000000000 0000000000000000 00000031b43fe0d0: 0000019f9ad97e50 0000000000000040 00000031b43fe0e0: 0000000000000003 00007ffb2f65a3f0 00000031b43fe0f0: 0000000000000001 00000031b43fe300 00000031b43fe100: 0000019f9adc3660 00007ffb2f551448 rax 0x7ffb2c32d85c rbx 0x7ffb2c32d85a rcx 0x41 rdi 0xffffffffffbadd11 rsi 0x0 rbp 0x7ffb2c409f00 rsp 0x31b43fe010 r8 0x0 r9 0x0 r10 0x0 r11 0x97c r12 0xc000007a r13 0x0 r14 0x7ffb2c32d85c r15 0x7ffb2f4f0000 rip 0x19fc1a40000 rflags 0x10206 cs 0x33 fs 0x53 gs 0x2b exit status 1 ```

Running a super simple program, GOTRACEBACK and GODEBUG don't seem to make a difference. It also fails more inconsistently than a program using cgo, but still fairly regularly

Code

``` package main import "fmt" func main() { fmt.Println("s") } ```

Crash

``` $ GODEBUG=asyncpreemptoff=1 GOTRACEBACK=system go run main.go Exception 0xc0000005 0x0 0x7ffb2f6f0fff 0x166364e0000 PC=0x166364e0000 runtime: unknown pc 0x166364e0000 stack: frame={sp:0x2a6dbfe5e0, fp:0x0} stack=[0x0,0x2a6dbffcc0) 0000002a6dbfe4e0: 0000002a6dbfe528 0000002a6dbfe550 0000002a6dbfe4f0: 0000002a6dbfe518 0000002a6dbfe510 0000002a6dbfe500: 0000002a6dbfe514 0000016611190000 0000002a6dbfe510: 0000000000000000 0000000000000000 0000002a6dbfe520: 0000000000000000 0000000000000005 0000002a6dbfe530: 0000002a6dbfe678 00007ffb2d2a9f38 0000002a6dbfe540: 0000016610fc4210 00007ffb2f5146ce 0000002a6dbfe550: 00007ffb2f4f00e8 0000016610ff9290 0000002a6dbfe560: 00007ffb2d2a9f32 0000002a6dbfe5b0 0000002a6dbfe570: 004f0044004e0049 0053005c00530057 0000002a6dbfe580: 0000016610fc4210 0000000000000000 0000002a6dbfe590: 0000016610fdf990 006c006400050005 0000002a6dbfe5a0: 00007ffb2d2a9f32 0000000000000000 0000002a6dbfe5b0: 00007ffb00000000 00007ffb2f4f00e8 0000002a6dbfe5c0: 0000000000000000 0000000000000000 0000002a6dbfe5d0: 0000000000000001 00007ffb2f513783 0000002a6dbfe5e0: <0000016600000001 0000000000000000 0000002a6dbfe5f0: 0000000000000000 0000002a6dbfe6e8 0000002a6dbfe600: 0000000000000000 0000000000000000 0000002a6dbfe610: 0000000000000000 0000000000000000 0000002a6dbfe620: 0000016610ff9290 00007ffb2f640f00 0000002a6dbfe630: 0000016610fc4210 00007ffb2f643520 0000002a6dbfe640: 000000000000097d 00007ffb2c32ccb8 0000002a6dbfe650: 00007ffb2c32a148 00007ffb2f640f28 0000002a6dbfe660: 00007ffb2f653d4f 00007ffb2c320000 0000002a6dbfe670: 00007ffb2f645b14 00007ffb2f570aa0 0000002a6dbfe680: 0000000000000000 0000000000000000 0000002a6dbfe690: 0000000000000000 0000000000000000 0000002a6dbfe6a0: 0000016610ff4d90 0000000000000040 0000002a6dbfe6b0: 0000000000000003 00007ffb2f65a3f0 0000002a6dbfe6c0: 0000000000000001 0000002a6dbfe900 0000002a6dbfe6d0: 0000016610fdf990 00007ffb2f551448 runtime: unknown pc 0x166364e0000 stack: frame={sp:0x2a6dbfe5e0, fp:0x0} stack=[0x0,0x2a6dbffcc0) 0000002a6dbfe4e0: 0000002a6dbfe528 0000002a6dbfe550 0000002a6dbfe4f0: 0000002a6dbfe518 0000002a6dbfe510 0000002a6dbfe500: 0000002a6dbfe514 0000016611190000 0000002a6dbfe510: 0000000000000000 0000000000000000 0000002a6dbfe520: 0000000000000000 0000000000000005 0000002a6dbfe530: 0000002a6dbfe678 00007ffb2d2a9f38 0000002a6dbfe540: 0000016610fc4210 00007ffb2f5146ce 0000002a6dbfe550: 00007ffb2f4f00e8 0000016610ff9290 0000002a6dbfe560: 00007ffb2d2a9f32 0000002a6dbfe5b0 0000002a6dbfe570: 004f0044004e0049 0053005c00530057 0000002a6dbfe580: 0000016610fc4210 0000000000000000 0000002a6dbfe590: 0000016610fdf990 006c006400050005 0000002a6dbfe5a0: 00007ffb2d2a9f32 0000000000000000 0000002a6dbfe5b0: 00007ffb00000000 00007ffb2f4f00e8 0000002a6dbfe5c0: 0000000000000000 0000000000000000 0000002a6dbfe5d0: 0000000000000001 00007ffb2f513783 0000002a6dbfe5e0: <0000016600000001 0000000000000000 0000002a6dbfe5f0: 0000000000000000 0000002a6dbfe6e8 0000002a6dbfe600: 0000000000000000 0000000000000000 0000002a6dbfe610: 0000000000000000 0000000000000000 0000002a6dbfe620: 0000016610ff9290 00007ffb2f640f00 0000002a6dbfe630: 0000016610fc4210 00007ffb2f643520 0000002a6dbfe640: 000000000000097d 00007ffb2c32ccb8 0000002a6dbfe650: 00007ffb2c32a148 00007ffb2f640f28 0000002a6dbfe660: 00007ffb2f653d4f 00007ffb2c320000 0000002a6dbfe670: 00007ffb2f645b14 00007ffb2f570aa0 0000002a6dbfe680: 0000000000000000 0000000000000000 0000002a6dbfe690: 0000000000000000 0000000000000000 0000002a6dbfe6a0: 0000016610ff4d90 0000000000000040 0000002a6dbfe6b0: 0000000000000003 00007ffb2f65a3f0 0000002a6dbfe6c0: 0000000000000001 0000002a6dbfe900 0000002a6dbfe6d0: 0000016610fdf990 00007ffb2f551448 rax 0x7ffb2c32d85c rbx 0x7ffb2c32d85a rcx 0x41 rdi 0xffffffffffbadd11 rsi 0x0 rbp 0x7ffb2c409f00 rsp 0x2a6dbfe5e0 r8 0x0 r9 0x0 r10 0x0 r11 0x97c r12 0xc000007a r13 0x0 r14 0x7ffb2c32d85c r15 0x7ffb2f4f0000 rip 0x166364e0000 rflags 0x10206 cs 0x33 fs 0x53 gs 0x2b ```

Is that more related to this issue? Happy to open a new one if you think I should.

[networkimprov]

Hopefully we'll get some pointers from the runtime team on investigating these reports...

cc @aclements @randall77 @prattmic

[networkimprov]

We have confirmation in #41138 that Webroot causes problems for Go. Uninstalling Webroot is a solution.

EDIT: If that fixes your problems, could you retitle the issue to identify Webroot?

[bhperry]

Well that's annoying. I tried turning off webroot protection and it still had the problem, but I didn't notice that webroot still has some background processes up even when it is "off".

I've suspected webroot of doing other crummy things in the past too (like slowing down my old laptop to the point where it was basically useless after the license expired). Guess it's time to find a different AV.

I'll try uninstalling and see what happens.

[bhperry]

wow... uninstalled Webroot and now everything works perfectly. Thanks for the help!

[davecheney]

Thanks for confirming.

[jasontconnell]

Apologies! Yeah our IT team allowed me an exception on webroot. Now we're in a transition to sophos. Hopefully that one behaves better.

I've had 1.13 installed in a c:\go-lock folder and have been building my programs that get distributed to other team members since then with that version. I haven't heard any issues since.

So, confirming a few months later that uninstalling webroot fixed my issue.

On Wed, Sep 9, 2020, 9:19 PM Liam notifications@github.com wrote:

Dave, the issue was opened by @jasontconnell https://github.com/jasontconnell who has yet to confirm that his problem is fixed.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/40878#issuecomment-689912992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABUVYYHFZW65QNCWFUJ243SFASTXANCNFSM4QEM3H3Q .

[jimitheat]

Holy fucking shit... I have web root as well and I was going crazy. I upgraded from 1.15 to 1.16 and it failed after installing the whole OS.

I've just turned off webroot and it compiles. OMG I was going nuts.

[bhperry]

I know the feels

[jasontconnell]

Months later follow up part 2. We've switched over to Sophos for our AV. After much revolt by the whole dev team due to performance issues, and it deleting apps I've written in Go, I am able to confirm no issues with Go 1.16 and Sophos. Glad you found this @jimitheat and for @davecheney thinking it could be AV.

Man I hate AV, at the beginning of our Sophos conversion last week, it was scanning every executable before it allowed it to execute, so saving a file in VS Code took like 5-6 seconds since it's doing gofmt and the linting and all that. Brutal.

[JacomusP]

@jasontconnell Hey do you have any other AV suggestions other than Sophos? Or does anybody else have some suggestions?

[alexbrainman]

Hey do you have any other AV suggestions other than Sophos? Or does anybody else have some suggestions?

I use Microsoft Defender. Every copy of Windows 10 come with this antivirus preinstalled. IMHO Microsoft Defender is the least painful antivirus to use.

Still I disable the antivirus when developing. Makes my computer run quicker.

Alex