Closed jasontconnell closed 4 years ago
Is there any anti virus software running on that machine that may be modifying or interposing on your actions?
Yes we apparently run "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48". I had wondered that but Go 1.13 was fine so I failed to mention.
Are you able to confirm if the problem occurs when "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48" is uninstalled or disabled?
I am unable to modify the settings but the real time protection is not enabled. I can try running it on another computer that isn't work controlled.
https://github.com/jasontconnell/chashprms
For reference. I'll be running it on my other computer shortly. Will report back.
Are you able to confirm if the problem occurs when "WebRoot SecureAnywhere Endpoint Protection v9.0.28.48" is uninstalled or disabled?
My home computer is windows 10 pro but a different build. When i ran that with 1.15 it ran fine! So you're on to something @davecheney but I'm updating windows at the moment and will run again when they're the same version.
Edit: this is taking forever. I'll have an update tomorrow.
Home PC Update: It's updated but not the same exact build number. Still runs fine in 1.15.
I have BitDefender on my home PC but that's apparently much better behaved, if an AV is to blame. It may be partially, but I can't say 100% since it works fine in 1.13.
Please try running your program with the environment variable GODEBUG=asyncpreemptoff=1
.
Please try running your program with the environment variable
GODEBUG=asyncpreemptoff=1
.
I had problem that was affected by this - #36492 - but I don't see crashes, I see hangs. But, still, you could try, it won't help.
Alex
Please try running your program with the environment variable
GODEBUG=asyncpreemptoff=1
.I had problem that was affected by this - #36492 - but I don't see crashes, I see hangs. But, still, you could try, it won't help.
Alex
Very in depth issue! Yeah, unless I'm doing it wrong (possible) @ianlancetaylor the ascyncpreemptoff=1 setting didn't alleviate the crash. I failed to mention that it also hangs and I can't ctrl+c out of it, I have to close the powershell window.
I've now run it in Go 1.13.15 and it runs perfectly. According to webroot the version of the software we're running was released on May 13 of this year, but I don't know when it was installed. This only just started happening this week, and I was running 1.14+ since Feb 26.
I have a question out to IT about when the webroot update might have been pushed, I will edit this comment with pertinent info when I get a response.
Edit: This lines up with when the issues started happening. I can uninstall webroot after talking to IT. I will do that soon.
AV Update: Was able to uninstall Webroot (IT granted me test permissions to remove it temporarily). Of course it runs fine.
Webroot was also causing issues compiling more than once with version 1.13.15. That update 1.1.226 in the screenshot above is most likely to blame, since I don't remember any issues (that weren't pebcak errors) before this week.
We are going to reinstall it and try to allow-list some paths where I and other devs typically use Go programs. That will not be tonight though. I'll be back.
... But, still, you could try, it won't help.
I was meant to say "But, still, you could try, it won't hurt.". Sorry for confusion.
AV Update: Was able to uninstall Webroot (IT granted me test permissions to remove it temporarily). Of course it runs fine.
Looks like you are wining your battles.
Alex
So... I'm not sure if this will require further digging. At least from my end. With the help of IT we re-installed webroot. It has the same Core update from a screenshot earlier that I was pretty sure would have been causing the issue. It seems to behave much better now even with 1.15, with the same program linked earlier, spitting out 100 hashes.
IT's thinking is that I didn't get the latest definitions or something. We don't have real time protection turned on so it just shouldn't have an effect either way. I am clueless as to a possible cause, yet it was consistently "randomly" reproducible.
So I'm currently not able to reproduce the originally reported issue. I've saved this issue so that if it happens again, and this is closed, I'll reference it. I hope it's just done though. I won't close it but someone on the Go team can if deemed appropriate.
Thanks for the help, I effing love Go :)
Hold the phone, it's happening again. Bombarded with other stuff right now. It was fine until I logged back on to our VPN, I'm not sure if that had anything to do with it, probably coincidence.
I am having very similar issues. Also running WebRoot, but shutting it down doesn't seem to affect anything. Originally thought it was a problem with my cgo setup, but now I've been testing with a super basic program that just prints a character and then exits. Sometimes it works, sometimes it hits the runtime: unknown pc
error and hangs.
I've been developing on WSL with an X server to run a game, but I'd really like to just get it compiling natively on windows. I've tried cross-compiling as well from WSL, and get similar results. Sometimes the build works, other times it does not (with zero code changes)
Running go version 1.15.1 amd64 right now. Tried 1.13.15 and 1.14.8, but neither of those worked. 1.13 actually didn't really work at all, couldn't even run go env
without it crashing.
Another report of this: #41138
Ok here's something strange. I noticed that an exe I had cross-compiled (and wasn't working) suddenly started working just fine. So I started time stamping builds and saving all of them. They mostly crash when I build them. But when I go back and run them later (an hour or so, but doesn't seem to be consistent), all the sudden they work.
try this and post its output (assuming the Windows command prompt)
set GOTRACEBACK=system
your_app
When I ran in windows cmd it just gave the runtime: unknown pc
and stack dump. Ran in an MYSY2 bash terminal and got the following:
Running with set GOTRACEBACK=all
did work on windows cmd, so here's that as well:
Ben, that looks like a different problem from the one originally posted here. Can you file a new issue, and include:
EDIT: also try it with this env var: GODEBUG=asyncpreemptoff=1
with GODEBUG=asyncpreemptoff=1
I'm just getting the stack dump (same whether or not GOTRACEBACK is set, not sure if that matters)
Running a super simple program, GOTRACEBACK and GODEBUG don't seem to make a difference. It also fails more inconsistently than a program using cgo, but still fairly regularly
Is that more related to this issue? Happy to open a new one if you think I should.
Hopefully we'll get some pointers from the runtime team on investigating these reports...
cc @aclements @randall77 @prattmic
We have confirmation in #41138 that Webroot causes problems for Go. Uninstalling Webroot is a solution.
EDIT: If that fixes your problems, could you retitle the issue to identify Webroot?
Well that's annoying. I tried turning off webroot protection and it still had the problem, but I didn't notice that webroot still has some background processes up even when it is "off".
I've suspected webroot of doing other crummy things in the past too (like slowing down my old laptop to the point where it was basically useless after the license expired). Guess it's time to find a different AV.
I'll try uninstalling and see what happens.
wow... uninstalled Webroot and now everything works perfectly. Thanks for the help!
Thanks for confirming.
Apologies! Yeah our IT team allowed me an exception on webroot. Now we're in a transition to sophos. Hopefully that one behaves better.
I've had 1.13 installed in a c:\go-lock folder and have been building my programs that get distributed to other team members since then with that version. I haven't heard any issues since.
So, confirming a few months later that uninstalling webroot fixed my issue.
On Wed, Sep 9, 2020, 9:19 PM Liam notifications@github.com wrote:
Dave, the issue was opened by @jasontconnell https://github.com/jasontconnell who has yet to confirm that his problem is fixed.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/40878#issuecomment-689912992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABUVYYHFZW65QNCWFUJ243SFASTXANCNFSM4QEM3H3Q .
Holy fucking shit... I have web root as well and I was going crazy. I upgraded from 1.15 to 1.16 and it failed after installing the whole OS.
I've just turned off webroot and it compiles. OMG I was going nuts.
I know the feels
Months later follow up part 2. We've switched over to Sophos for our AV. After much revolt by the whole dev team due to performance issues, and it deleting apps I've written in Go, I am able to confirm no issues with Go 1.16 and Sophos. Glad you found this @jimitheat and for @davecheney thinking it could be AV.
Man I hate AV, at the beginning of our Sophos conversion last week, it was scanning every executable before it allowed it to execute, so saving a file in VS Code took like 5-6 seconds since it's doing gofmt and the linting and all that. Brutal.
@jasontconnell Hey do you have any other AV suggestions other than Sophos? Or does anybody else have some suggestions?
Hey do you have any other AV suggestions other than Sophos? Or does anybody else have some suggestions?
I use Microsoft Defender. Every copy of Windows 10 come with this antivirus preinstalled. IMHO Microsoft Defender is the least painful antivirus to use.
Still I disable the antivirus when developing. Makes my computer run quicker.
Alex
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?OS is Windows 10 Pro 10.0.18362
go env
OutputWhat did you do?
I don't think the code matters. It has crashed with multiple different programs. But here's an example. It's more in the "running it repeatedly" that breaks it. I've tested it on Go 1.13 and it runs fine. The earliest version that I've tried that showed the error was Go 1.14.3 as displayed in the version section of this post. It is reproducible on Go 1.15 as well.
So my manager was reporting a crash, a memory dump. I wasn't able to reproduce it locally. But then I wrote the simple program
https://play.golang.org/p/KMabbeiz1hW
And a powershell script
For ($i = 0; $i -lt 100; $i++) { Write-Host $i .\chashprms "$i is the value" }
Run go build and run.ps1.
What did you expect to see?
If I run the same code in Docker using the golang:1.15 image, it runs fine. Runs through 100 instances and always prints out the hash. I would expect to see 100 hashes printed. In Go 1.13 it works fine (the latest I tested where it works).
What did you see instead?
In Go 1.14.3 (the earliest that I tested where it crashes), I see a memory and registry dump after a few calls:
Exception 0xc0000005 0x0 0x7ffd4c110fff 0x910000 PC=0x910000
runtime: unknown pc 0x910000 stack: frame={sp:0x84f6a0, fp:0x0} stack=[0x0,0x84ff00) 000000000084f5a0: 000000000098d7c0 000000000084f610 000000000084f5b0: 0000000000000000 000000000084f6a8 000000000084f5c0: 0000000000989710 000000000084f600 000000000084f5d0: 0000000000000000 00007ffd48e60108 000000000084f5e0: 0000000000000000 0000000000963880 000000000084f5f0: 000000000084f738 00007ffd49d194d1 000000000084f600: 00007ffd49d194cb 00007ffd4bf3ba6f 000000000084f610: 00007ffd4bf200d8 0000000000975c50 000000000084f620: 0000000000000005 000000000084f670 000000000084f630: 00007ffd49d17900 00007ffd4bf51810 000000000084f640: 0000000000963880 0000000000000000 000000000084f650: 0000000000050005 00007ffd49d194cb 000000000084f660: 000000000000001b 0000000000000000 000000000084f670: 0000000000000000 00007ffd4bf200d8 000000000084f680: 0000000000000000 00007ffd49ac0000 000000000084f690: 0000000000000001 00007ffd4bf3c734 000000000084f6a0: <0000000000000001 0000000000000000 000000000084f6b0: 00007ffd0000ebb0 000000000084f7a8 000000000084f6c0: 00000000009897e0 000000000084f728 000000000084f6d0: 0000000000250024 00007ffd48e7b816 000000000084f6e0: 000000000098d690 00007ffd4c06c500 000000000084f6f0: 0000000000963880 00007ffd4c06ea5c 000000000084f700: 000000000000094c 00007ffd48deb2d8 000000000084f710: 00007ffd48de9138 00007ffd4c06c528 000000000084f720: 00007ffd4c07ec6a 00007ffd48de0000 000000000084f730: 00007ffd4c070f8c 00007ffd4bfa0d80 000000000084f740: 000000001c1dc789 000000000098d818 000000000084f750: 000000000084fab0 00000000c0000135 000000000084f760: 0000000000000000 0000000000000040 000000000084f770: 0000000000000004 00007ffd4c0852f0 000000000084f780: 0000000000000001 000000000084f900 000000000084f790: 0000000000975c50 00007ffd4bf4e2a8 runtime: unknown pc 0x910000 stack: frame={sp:0x84f6a0, fp:0x0} stack=[0x0,0x84ff00) 000000000084f5a0: 000000000098d7c0 000000000084f610 000000000084f5b0: 0000000000000000 000000000084f6a8 000000000084f5c0: 0000000000989710 000000000084f600 000000000084f5d0: 0000000000000000 00007ffd48e60108 000000000084f5e0: 0000000000000000 0000000000963880 000000000084f5f0: 000000000084f738 00007ffd49d194d1 000000000084f600: 00007ffd49d194cb 00007ffd4bf3ba6f 000000000084f610: 00007ffd4bf200d8 0000000000975c50 000000000084f620: 0000000000000005 000000000084f670 000000000084f630: 00007ffd49d17900 00007ffd4bf51810 000000000084f640: 0000000000963880 0000000000000000 000000000084f650: 0000000000050005 00007ffd49d194cb 000000000084f660: 000000000000001b 0000000000000000 000000000084f670: 0000000000000000 00007ffd4bf200d8 000000000084f680: 0000000000000000 00007ffd49ac0000 000000000084f690: 0000000000000001 00007ffd4bf3c734 000000000084f6a0: <0000000000000001 0000000000000000 000000000084f6b0: 00007ffd0000ebb0 000000000084f7a8 000000000084f6c0: 00000000009897e0 000000000084f728 000000000084f6d0: 0000000000250024 00007ffd48e7b816 000000000084f6e0: 000000000098d690 00007ffd4c06c500 000000000084f6f0: 0000000000963880 00007ffd4c06ea5c 000000000084f700: 000000000000094c 00007ffd48deb2d8 000000000084f710: 00007ffd48de9138 00007ffd4c06c528 000000000084f720: 00007ffd4c07ec6a 00007ffd48de0000 000000000084f730: 00007ffd4c070f8c 00007ffd4bfa0d80 000000000084f740: 000000001c1dc789 000000000098d818 000000000084f750: 000000000084fab0 00000000c0000135 000000000084f760: 0000000000000000 0000000000000040 000000000084f770: 0000000000000004 00007ffd4c0852f0 000000000084f780: 0000000000000001 000000000084f900 000000000084f790: 0000000000975c50 00007ffd4bf4e2a8 rax 0x7ffd48debe7c rbx 0x7ffd48debe7a rcx 0x41 rdi 0xffffffffffbadd11 rsi 0x0 rbp 0x84f900 rsp 0x84f6a0 r8 0x0 r9 0x0 r10 0x0 r11 0x94b r12 0x7ffd4bf20000 r13 0x0 r14 0x7ffd48debe7c r15 0xc000007a rip 0x910000 rflags 0x10202 cs 0x33 fs 0x53 gs 0x2b
We are a Windows shop and do .NET dev and these tools I wrote help with that, I would love to be able to keep them in Go :D For now I have to revert them to 1.13ish but I'll be watching this closely. I am available and on the Gopher slack, if you need more information I'll be happy to provide.