search

beego / beego

beego is an open-source, high-performance web framework for the Go programming language.
Other
31.57k stars 5.61k forks source link

The server crashes with "fatal error: concurrent map read and map write" #1682

Closed cristian-sima closed 8 years ago

cristian-sima commented 8 years ago

I recently upgraded to Go 1.6 and SOMETIMES (I have no clues why) my server crashes with this log.

I think it is because Go 1.6 introduces the detection of concurrent map write and read (more here)

Here is the full log of the problem

fatal error: concurrent map read and map write

goroutine 193 [running]:
runtime.throw(0xce20e0, 0x21)
C:/Go/src/runtime/panic.go:530 +0x97 fp=0xc0822e7460 sp=0xc0822e7448
runtime.mapaccess2_faststr(0x9d3b80, 0xc082059da0, 0xc082385350, 0x2e, 0x1, 0xc0                                                                                                                82385290)
C:/Go/src/runtime/hashmap_fast.go:307 +0x62 fp=0xc0822e74c0 sp=0xc0822e7                                                                                                                460
github.com/astaxie/beego.openFile(0xc082385290, 0x29, 0x16a4800, 0xc0821c2540, 0                                                                                                                xc30e60, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:101 +0x107 fp                                                                                                                =0xc0822e7620 sp=0xc0822e74c0
github.com/astaxie/beego.serverStaticRouter(0xc082307230)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:67 +0x34b fp=                                                                                                                0xc0822e77c0 sp=0xc0822e7620
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc082059ef0, 0x3500858                                                                                                                , 0xc082409e10, 0xc082936700)
W:/go-workspace/src/github.com/astaxie/beego/router.go:637 +0x11fc fp=0x                                                                                                                c0822e7b60 sp=0xc0822e77c0
net/http.serverHandler.ServeHTTP(0xc0820b0180, 0x3500858, 0xc082409e10, 0xc08293                                                                                                                6700)
C:/Go/src/net/http/server.go:2081 +0x1a5 fp=0xc0822e7bc0 sp=0xc0822e7b60
net/http.(*conn).serve(0xc0820b1480)
C:/Go/src/net/http/server.go:1472 +0xf35 fp=0xc0822e7f88 sp=0xc0822e7bc0
runtime.goexit()
C:/Go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc0822e7f90 sp=0xc0822e7f88
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2137 +0x455

goroutine 1 [chan receive, 61 minutes]:
github.com/astaxie/beego.(*App).Run(0xc082003480)
W:/go-workspace/src/github.com/astaxie/beego/app.go:182 +0xac2
github.com/astaxie/beego.Run(0x0, 0x0, 0x0)
W:/go-workspace/src/github.com/astaxie/beego/beego.go:67 +0x16a
main.main()
W:/go-workspace/src/github.com/cristian-sima/contabilitate/main.go:132 +                                                                                                                0x8e3

goroutine 5 [syscall, 61 minutes]:
os/signal.signal_recv(0x0)
C:/Go/src/runtime/sigqueue.go:116 +0x139
os/signal.loop()
C:/Go/src/os/signal/signal_unix.go:22 +0x1f
created by os/signal.init.1
C:/Go/src/os/signal/signal_unix.go:28 +0x3e

goroutine 194 [semacquire]:
sync.runtime_Semacquire(0x109b0bc)
C:/Go/src/runtime/sema.go:47 +0x2d
sync.(*Mutex).Lock(0x109b0b8)
C:/Go/src/sync/mutex.go:83 +0x1cb
github.com/astaxie/beego.openFile(0xc0823848d0, 0x28, 0x16a4800, 0xc0821c22a0, 0                                                                                                                xc30e60, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:105 +0x1a6
github.com/astaxie/beego.serverStaticRouter(0xc082384750)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:67 +0x34b
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc082059ef0, 0x3500858                                                                                                                , 0xc0824088f0, 0xc082936380)
W:/go-workspace/src/github.com/astaxie/beego/router.go:637 +0x11fc
net/http.serverHandler.ServeHTTP(0xc0820b0180, 0x3500858, 0xc0824088f0, 0xc08293                                                                                                                6380)
C:/Go/src/net/http/server.go:2081 +0x1a5
net/http.(*conn).serve(0xc0820b0480)
C:/Go/src/net/http/server.go:1472 +0xf35
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2137 +0x455

goroutine 195 [runnable]:
github.com/astaxie/beego/context.(*BeegoOutput).Header(0xc0822a41e0, 0xca7350, 0                                                                                                                x10, 0xc30e60, 0x4)
W:/go-workspace/src/github.com/astaxie/beego/context/output.go:53
github.com/astaxie/beego.serverStaticRouter(0xc082384060)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:77 +0x6ec
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc082059ef0, 0x3500858                                                                                                                , 0xc082383c70, 0xc08238f0a0)
W:/go-workspace/src/github.com/astaxie/beego/router.go:637 +0x11fc
net/http.serverHandler.ServeHTTP(0xc0820b0180, 0x3500858, 0xc082383c70, 0xc08238                                                                                                                f0a0)
C:/Go/src/net/http/server.go:2081 +0x1a5
net/http.(*conn).serve(0xc0820b0500)
C:/Go/src/net/http/server.go:1472 +0xf35
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2137 +0x455

goroutine 80 [runnable]:
sync.runtime_Semacquire(0x109b0bc)
C:/Go/src/runtime/sema.go:47 +0x2d
sync.(*Mutex).Lock(0x109b0b8)
C:/Go/src/sync/mutex.go:83 +0x1cb
github.com/astaxie/beego.openFile(0xc082384540, 0x21, 0x16a4800, 0xc0821c21e0, 0                                                                                                                xc30e60, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:105 +0x1a6
github.com/astaxie/beego.serverStaticRouter(0xc0823843c0)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:67 +0x34b
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc082059ef0, 0x3500858                                                                                                                , 0xc0824084e0, 0xc08238e0e0)
W:/go-workspace/src/github.com/astaxie/beego/router.go:637 +0x11fc
net/http.serverHandler.ServeHTTP(0xc0820b0180, 0x3500858, 0xc0824084e0, 0xc08238                                                                                                                e0e0)
C:/Go/src/net/http/server.go:2081 +0x1a5
net/http.(*conn).serve(0xc0820b1380)
C:/Go/src/net/http/server.go:1472 +0xf35
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2137 +0x455

goroutine 138 [IO wait, 3 minutes]:
net.runtime_pollWait(0x16b4c18, 0x72, 0xc082130b38)
C:/Go/src/runtime/netpoll.go:160 +0x67
net.(*pollDesc).Wait(0xc08243cef0, 0x72, 0x0, 0x0)
C:/Go/src/net/fd_poll_runtime.go:73 +0x41
net.(*ioSrv).ExecIO(0xc082450828, 0xc08243cde0, 0xc39ef0, 0x8, 0xc08245c620, 0xc                                                                                                                082433200, 0x0, 0x0)
C:/Go/src/net/fd_windows.go:183 +0x177
net.(*netFD).acceptOne(0xc08243cd80, 0xc08238ed20, 0x2, 0x2, 0xc08243cde0, 0x980                                                                                                                00004, 0x0, 0x0)
C:/Go/src/net/fd_windows.go:583 +0x26c
net.(*netFD).accept(0xc08243cd80, 0x0, 0x0, 0x0)
C:/Go/src/net/fd_windows.go:613 +0x173
net.(*TCPListener).AcceptTCP(0xc082450838, 0x454bc0, 0x0, 0x0)
C:/Go/src/net/tcpsock_posix.go:254 +0x54
net/http.tcpKeepAliveListener.Accept(0xc082450838, 0x0, 0x0, 0x0, 0x0)
C:/Go/src/net/http/server.go:2425 +0x48
net/http.(*Server).Serve(0xc0820b0180, 0x16b4cd8, 0xc082450838, 0x0, 0x0)
C:/Go/src/net/http/server.go:2117 +0x130
net/http.(*Server).ListenAndServe(0xc0820b0180, 0x0, 0x0)
C:/Go/src/net/http/server.go:2098 +0x13d
github.com/astaxie/beego.(*App).Run.func4(0xc082003480, 0xc082002240, 0x5, 0xc08                                                                                                                21b05b0)
W:/go-workspace/src/github.com/astaxie/beego/app.go:174 +0x47d
created by github.com/astaxie/beego.(*App).Run
W:/go-workspace/src/github.com/astaxie/beego/app.go:180 +0xa9f

goroutine 78 [semacquire]:
sync.runtime_Semacquire(0x109b0bc)
C:/Go/src/runtime/sema.go:47 +0x2d
sync.(*Mutex).Lock(0x109b0b8)
C:/Go/src/sync/mutex.go:83 +0x1cb
github.com/astaxie/beego.openFile(0xc082384bd0, 0x25, 0x16a4800, 0xc0821c23c0, 0                                                                                                                xc30e60, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:105 +0x1a6
github.com/astaxie/beego.serverStaticRouter(0xc082306690)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:67 +0x34b
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc082059ef0, 0x3500858                                                                                                                , 0xc082409860, 0xc082936460)
W:/go-workspace/src/github.com/astaxie/beego/router.go:637 +0x11fc
net/http.serverHandler.ServeHTTP(0xc0820b0180, 0x3500858, 0xc082409860, 0xc08293                                                                                                                6460)
C:/Go/src/net/http/server.go:2081 +0x1a5
net/http.(*conn).serve(0xc0820b0400)
C:/Go/src/net/http/server.go:1472 +0xf35
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2137 +0x455

goroutine 79 [semacquire]:
sync.runtime_Semacquire(0x109b0bc)
C:/Go/src/runtime/sema.go:47 +0x2d
sync.(*Mutex).Lock(0x109b0b8)
C:/Go/src/sync/mutex.go:83 +0x1cb
github.com/astaxie/beego.openFile(0xc082384fc0, 0x2b, 0x16a4800, 0xc0821c2420, 0                                                                                                                xc30e60, 0x4, 0x0, 0x0, 0x0, 0x0, ...)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:105 +0x1a6
github.com/astaxie/beego.serverStaticRouter(0xc08223dce0)
W:/go-workspace/src/github.com/astaxie/beego/staticfile.go:67 +0x34b
github.com/astaxie/beego.(*ControllerRegister).ServeHTTP(0xc082059ef0, 0x3500858                                                                                                                , 0xc082409ad0, 0xc082936540)
W:/go-workspace/src/github.com/astaxie/beego/router.go:637 +0x11fc
net/http.serverHandler.ServeHTTP(0xc0820b0180, 0x3500858, 0xc082409ad0, 0xc08293                                                                                                                6540)
C:/Go/src/net/http/server.go:2081 +0x1a5
net/http.(*conn).serve(0xc0820b0b00)
C:/Go/src/net/http/server.go:1472 +0xf35
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2137 +0x455
ysqi commented 8 years ago

hi @cristian-sima it is a bug. read and write static file map error:

staticFileMap = make(map[string]*serveContentHolder)

and @astaxie Maybe need to modify the static file server logic has to ensure efficient and safe.

cristian-sima commented 8 years ago

@astaxie Can you please fix this bug. It is really difficult to stop and start the server for 20 times per day. Thanks

iyidan commented 8 years ago

I have come across this problem too. my go version is go1.6 and my system is mac osx 10.11.3 (15D21)

I use the bee command to create a project named beegoDemo $ bee new beegoDemo

and then run the project then, use ab to benchmark

$ ab -n 10000 -c 10 http://localhost:8080/

And the problem appeared

`fatal error: concurrent map read and map write

goroutine 407 [running]: runtime.throw(0x6a1a80, 0x21) /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc820341478 sp=0xc820341460 runtime.mapaccess1_faststr(0x476ce0, 0xc82012c210, 0x62b810, 0x9, 0xc8201058e8) /usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc8203414d8 sp=0xc820341478 github.com/astaxie/beego.(_Controller).RenderBytes(0xc8201c31e0, 0x0, 0x0, 0x0, 0x0, 0x0) /Users/liwei/Documents/mygo/src/github.com/astaxie/beego/controller.go:265 +0x1893 fp=0xc820341760 sp=0xc8203414d8 github.com/astaxie/beego.(_Controller).Render(0xc8201c31e0, 0x0, 0x0) /Users/liwei/Documents/mygo/src/github.com/astaxie/beego/controller.go:183 +0x4e fp=0xc8203417c0 sp=0xc820341760 github.com/astaxie/beego.(_ControllerRegister).ServeHTTP(0xc82012c300, 0xd900d0, 0xc8201c2f70, 0xc820346000) /Users/liwei/Documents/mygo/src/github.com/astaxie/beego/router.go:784 +0x1d7c fp=0xc820341b60 sp=0xc8203417c0 net/http.serverHandler.ServeHTTP(0xc82007c180, 0xd900d0, 0xc8201c2f70, 0xc820346000) /usr/local/go/src/net/http/server.go:2081 +0x19e fp=0xc820341bc0 sp=0xc820341b60 net/http.(_conn).serve(0xc82007d100) /usr/local/go/src/net/http/server.go:1472 +0xf2e fp=0xc820341f88 sp=0xc820341bc0 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820341f90 sp=0xc820341f88 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2137 +0x44e

goroutine 1 [chan receive]: github.com/astaxie/beego.(*App).Run(0xc82000b5c0) /Users/liwei/Documents/mygo/src/github.com/astaxie/beego/app.go:182 +0xabb github.com/astaxie/beego.Run(0x0, 0x0, 0x0) /Users/liwei/Documents/mygo/src/github.com/astaxie/beego/beego.go:67 +0x163 main.main() /Users/liwei/Documents/mygo/src/beegoDemo/main.go:9 +0x28

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1

goroutine 5 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:116 +0x132 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1 /usr/local/go/src/os/signal/signal_unix.go:28 +0x37`

outman commented 8 years ago

@iyidan Just in DEV mode .

astaxie commented 8 years ago

it's fixed in develop branch

ssh-nkar commented 8 years ago

@astaxie curious.. what was the fix?

JessonChan commented 8 years ago

@shiv123 this commit https://github.com/astaxie/beego/commit/226e54e0d8d81b287a67949d9559572be5ccc363