Panic after batch executing a number of statements

[GoogleCodeExporter]

I am using the odbc driver to insert a large amount of records obtained from a 
webservice. I create a prepared statement and then execute that statement for 
every object I receive from the webservice. After a while the program panics 
with the following stack trace.

unexpected fault address 0xc180219d3e
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xc180219d3e pc=0x7feead01464]

goroutine 1 [syscall]:
[fp=0x3916d8] return()
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/runtime/asm_amd64.s:508
[fp=0x391720] runtime.cgocall(0x427420, 0x391730)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/runtime/cgocall.c:162 +0x133
[fp=0x391768] syscall.Syscall(0x7feeb110f00, 0x1, 0x1cae50, 0x0, 0x0, ...)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/runtime/zsyscall_windows_windows_amd64.c:74 +0x55
[fp=0x3917b0] code.google.com/p/odbc/api.SQLExecute(0x1cae50, 0x1cae50)
        Y:/gocode/src/code.google.com/p/odbc/api/zapi_windows.go:89 +0x5e
[fp=0x391848] code.google.com/p/odbc.(*ODBCStmt).Exec(0xc0801d6f00, 
0xc08022c3c0, 0xf, 0xf, 0x494ac0, ...)
        Y:/gocode/src/code.google.com/p/odbc/odbcstmt.go:110 +0x1b4
[fp=0x3918a8] code.google.com/p/odbc.(*Stmt).Exec(0xc080145780, 0xc08022c3c0, 
0xf, 0xf, 0x0, ...)
        Y:/gocode/src/code.google.com/p/odbc/stmt.go:60 +0x203
[fp=0x3919a0] database/sql.resultFromStatement(0xc0800bab10, 0xc0800a92a0, 
0xc08008cfc0, 0xc080145780, 0xc08022c1e0, ...)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/database/sql/sql.go:1077 +0x27b
[fp=0x391a38] database/sql.(*Stmt).Exec(0xc0801a2c00, 0xc08022c1e0, 0xf, 0xf, 
0x0, ...)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/database/sql/sql.go:1056 +0x1c3
[fp=0x391f90] main.main()
        Y:/gocode/src/dbtest/main.go:99+0xb8f
[fp=0x391fb8] runtime.main()
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/runtime/proc.c:182 +0x99
[fp=0x391fc0] runtime.goexit()
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/runtime/proc.c:1223

goroutine 4 [syscall]:
syscall.Syscall6(0x775ca4d0, 0x5, 0x310, 0xc0800e7e60, 0xc080000fa0, ...)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/runtime/zsyscall_windows_windows_amd64.c:97 +0x55
syscall.GetQueuedCompletionStatus(0x310, 0xc0800e7e60, 0xc080000fa0, 
0xc080000f98, 0xffffffff, ...)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/syscall/zsyscall_windows_amd64.go:507 +0x9f
net.(*resultSrv).Run(0xc080000f90)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/net/fd_windows.go:150 +0x151
created by net.startServer
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/net/fd_windows.go:285 +0x10f

goroutine 251 [chan receive]:
net/http.(*persistConn).readLoop(0xc080120a00)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/net/http/transport.go:761 +0x64e
created by net/http.(*Transport).dialConn
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/net/http/transport.go:511 +0x577

goroutine 264 [chan send]:
dbtest/xsc.Resource.parseObjectStream(0xc0800dfb80, 0xc08010b800, 0x1b, 
0x10001, 0xc0801bd6a0, ...)
        Y:/gocode/src/dbtest/xsc.go:316 +0x65c
created by dbtest/xsc.Resource.Data
        Y:/gocode/src/dbtest/xsc.go:236 +0x212

goroutine 252 [select]:
net/http.(*persistConn).writeLoop(0xc080120a00)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/net/http/transport.go:774 +0x272
created by net/http.(*Transport).dialConn
        C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist675583351/go/src/pkg/net/http/transport.go:512 +0x58e

What version of the product are you using? On what operating system?

odbc (commit fdeace4e5288)
go version 1.1.2 windows/amd64
Windows Server 2008 R2 Standard SP1
Microsoft SQL Server 2008

Please provide any additional information below.

So far I haven't found a way to reliably reproduce this problem. The panic 
happens every time (with the same input data), but always with different insert 
values. Sometimes it would crash after a second, sometimes after 10. A 
statement that crashed before was usually successfully executed the next run. 
But the panic always happens at some point.

Since I'm working with some sensitive data I cannot provide the input data nor 
the source code, but I'm working on creating a smaller program that reproduces 
this problem (no luck so far). Since I have not much experience yet with cgo, 
any suggestions or ideas on how to debug this is very much appreciated.

Original issue reported on code.google.com by stemmert...@gmail.com on 5 Sep 2013 at 12:27

[GoogleCodeExporter]

> ... So far I haven't found a way to reliably reproduce this problem.

That is not good. It looks like memory corruption - it could be happening 
anywhere: in your program, in odbc package, in Go stdlib or in Windows dlls. 
Reproducible example will make hunting it down easier.

> ... Since I have not much experience yet with cgo, 

Looking at your stack trace I don't see cgo is used here. Does your program use 
any other packages that use cgo?

> ... any suggestions or ideas on how to debug this is very much appreciated.

Like I said earlier, I suspect it is memory corruption. If I am correct, it is 
very difficult to find the problem because the crash is caused by either Go 
garbage collector or by Windows memory manager freeing / reallocating memory. 
That means it crashes long way after the bug being hit.

One obvious thing for you to try is to use "latest" Go version. You said, you 
are using "go version 1.1.2". As far as I remember, there were some 
database/sql fixes that could fix your problem. It is certainly worth a try.

Alex

Original comment by alex.bra...@gmail.com on 6 Sep 2013 at 12:45

[GoogleCodeExporter]

Thanks for the reply. I wrongly assumed the driver used cgo, because of the 
cgocall.c in the stacktrace. There are no other packages that use cgo.

I've reduced the problem to this:
https://gist.github.com/jstemmer/6461407

Some observations so far:
- The more parameters I use in my named query, the quicker the panic triggers.
- The longer the string variable I use as the parameters, the quicker the panic 
triggers (after executing the statement about ~25 times on my machine).
- Calling runtime.GC() after each execute seems to solve it (at least in this 
case; it runs the code in the gist every time without a panic).
- Panic happens both when compiling on windows or crosscompiling from linux to 
windows.
- Tried with go tip (version devel +601c402eb321), doesn't seem to change 
anything except triggering the panic faster (after executing the statement ~15 
times).
- The same code runs fine with the mysql driver 
(github.com/go-sql-driver/mysql) on both windows and linux.

Original comment by stemmert...@gmail.com on 6 Sep 2013 at 9:30

[GoogleCodeExporter]

I've worked around this issue by not using the sql parameters at all. I created 
the entire sql statement as a single string, including escaping each of the 
values where necessary, and executing it with the db.Exec function. This works 
great for millions of inserts, which makes me believe the problem has something 
to do with the way the parameters are handled.

After trying a few different things, I've managed to prevent the panic from 
happening by supplying a nil for the last parameter of SQLBindParameter (in 
param.go:105) when dealing with string parameters. This is possible since the 
strings are null terminated. It seems to fix the problem at first when running 
the different tests, but when I try to use it in my main application I 
encounter other errors. For example "[Microsoft][ODBC SQL Server Driver][SQL 
Server]Conversion failed when converting date and/or time from character 
string", which is strange since this worked fine before. No more panics 
however, which is good.

Unfortunately I don't have much time to investigate further to try and fix 
these issues. I've found another go odbc driver at 
https://github.com/LukeMauldin/lodbc which runs fine without any significant 
problems.

Original comment by stemmert...@gmail.com on 8 Sep 2013 at 1:34

[GoogleCodeExporter]

> I've reduced the problem to this:
> https://gist.github.com/jstemmer/6461407

Thank you for that. I can see the problem know. Here is a fix 
https://codereview.appspot.com/13253046/ Please, review and let me know, if you 
like.

> Unfortunately I don't have much time to investigate further to try and fix 
these issues. I've found another go odbc driver at 
https://github.com/LukeMauldin/lodbc which runs fine without any significant 
problems.

Fair enough. Luckily all these drivers are interchangeable. It should just be a 
matter of changing import statement.

Alex

Original comment by alex.bra...@gmail.com on 9 Sep 2013 at 2:18

• Changed state: Accepted

[GoogleCodeExporter]

Thanks for the patch. Some more issues popped up after applying the patch, I 
tracked most of them down. I've added some comments at 
https://codereview.appspot.com/13253046/

Original comment by stemmert...@gmail.com on 9 Sep 2013 at 10:07

[GoogleCodeExporter]

This issue was closed by revision d64013ec9967.

Original comment by alex.bra...@gmail.com on 13 Sep 2013 at 2:09

• Changed state: Fixed