db.Query breaks the application over ab benchmark

[onaxon]

I have divided the mdb.go example in two apps:

• mdb1 for the DB load http://play.golang.org/p/gvLXYfFCif
• mdb2 for the querys http://play.golang.org/p/ePg2uTHy21

When I test the mdb2 app manually with the browser, curl, etc it works fine, but when y put them on load, for example:

ab -n 100 -c 10 -k http://app.url/

ab exits with "...apr_socket_recv: Connection reset by peer (54)", and the app crashes with:

Exception 0xc00000fd 0x0 0x3150263c
PC=0x31e4d284fr French

signal arrived during cgo execution

syscall.Syscall9(0x4bcc6bd6, 0x9, 0x315922a0, 0x5, 0x6d65f0, ...de German
)
es Spanish
        /usr/local/go/src/pkg/runtime/zsyscall_windows_windows_386.c:123 +0x49
github.com/mattn/go-ole.invoke(0x315922a0, 0x5, 0x1, 0x0, 0x0, ...)
        /Users/pmd/Dropbox/Code/go/src/github.com/mattn/go-ole/idispatch.go

I have test it wit wrk with similar errors:

wrk -t2 -c10 -d5s http://app.url

Thanks.

[mattn]

Could you please try to use mutex.Lock/Unlock between querying?

[onaxon]

I'm learning Go and not started with concurrency, so do not know how to use mutex. I've tried:

var mu sync.Mutex
mu.Lock()
rows , err: = db.Query ("select id, name from languages")
mu.Unlock()

And also putting the Unlock after the rows.Next loop, but without success.

I thought that, although unlikely, could be a problem of "database/sql", so I used the same example with the PostgreSQL driver ( bmizerany/pq ) and a similar DB, but all ab test passed flawless.

As I understand, each http request runs in its own goroutine, but can share the database connection. When I run requests manually there are no concurrency, as the process is fast enough to have ended when the next request arrive, and therefore do not share the connection. But a program like ab is able to send many more requests per second, and can do so concurrently, so it is probable to run a new db.Query without having received the previous query data.

pq has had a similar problem, but only with transactions, maybe this can throw some light:

https://groups.google.com/forum/#!topic/golang-nuts/94NKlXjbN1g https://github.com/lib/pq/issues/81

[mattn]

Ok, I'll look into it.

[mattn]

I didn't get error with following code.

package main

import (
    "fmt"
    "net/http"
    "database/sql"
    _ "github.com/mattn/go-adodb"
    "sync"
)

var mutex sync.Mutex

func rootHandler(w http.ResponseWriter, r *http.Request) {
    mutex.Lock()
    defer mutex.Unlock()

    db, err := sql.Open("adodb", "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=./example.mdb;")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer db.Close()

    rows, err := db.Query("select id, name from languages")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer rows.Close()

    for rows.Next() {
        var id string
        var name string
        rows.Scan(&id, &name)
        fmt.Println(id, name)
    }

}

func main() {

    http.HandleFunc("/", rootHandler)

    fmt.Println("About to listen on 8081")
    err := http.ListenAndServe(":8081", nil)
    if err != nil {
        fmt.Println("ListenAndServe: ",err)
        return
    }

}

[onaxon]

Yes, your code works, the key is to declare the mutex variable globally and make mutex.Lock() at the beginning of the handler.

Thank you.

To not set a connection on each request, and get higher requests per second, I made a test declaring a global variable db * sql.DB and calling sql.Open in main, but then, after 500-600 requests throws error "db.Query error: An exception occurred".

See code: http://play.golang.org/p/hB0XHYi7Kc

[mattn]

Looks good

[neclepsio]

I have the same exception, but use no currency at all. The crash is repetitive, and depending on the order I execute the queries. I.e.: if I execute query1 before and query2 after, it crashes.

If I execute the same exact query2 before and query1 after, it works. The crash occours only if, after db.Query(...), I read the data using rows.Next(). If I just execute the db.Query(...), it does not crash.

Can you help me diagnose the issue?

Stack is:

Exception 0xc00000fd 0x0 0x375b2000 0x673dc7c8
PC=0x673dc7c8

syscall.Syscall9(0x670f73b6, 0x9, 0x32e987e8, 0x5, 0x126843c0, 0x410, 0x1, 0x126679cc, 0x1298aa40, 0x12667af8, ...)
    F:/Windows/Go/Go17/src/runtime/syscall_windows.go:185 +0x4a
github.com/go-ole/go-ole.invoke(0x32e987e8, 0x5, 0x1, 0x0, 0x0, 0x0, 0x1298aa40, 0x0, 0x0)
    F:/Home-Mirror/Programming/Go-Workspace/src/github.com/go-ole/go-ole/idispatch_windows.go:179 +0x534
github.com/go-ole/go-ole.(*IDispatch).Invoke(0x32e987e8, 0x5, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    F:/Home-Mirror/Programming/Go-Workspace/src/github.com/go-ole/go-ole/idispatch.go:27 +0x57
github.com/go-ole/go-ole.(*IDispatch).InvokeWithOptionalArgs(0x32e987e8, 0x4eb247, 0x7, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    F:/Home-Mirror/Programming/Go-Workspace/src/github.com/go-ole/go-ole/idispatch.go:69 +0x97
github.com/go-ole/go-ole/oleutil.CallMethod(0x32e987e8, 0x4eb247, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    F:/Home-Mirror/Programming/Go-Workspace/src/github.com/go-ole/go-ole/oleutil/oleutil.go:51 +0x5c
github.com/mattn/go-adodb.(*AdodbStmt).Query(0x12984660, 0x5679fc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    F:/Home-Mirror/Programming/Go-Workspace/src/github.com/mattn/go-adodb/adodb.go:197 +0xaa
database/sql.rowsiFromStatement(0x547ce0, 0x1267c140, 0x547f40, 0x12984660, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    F:/Windows/Go/Go17/src/database/sql/sql.go:1655 +0x263
database/sql.(*DB).queryConn(0x12682060, 0x1267c140, 0x12a3e818, 0x4f2f6a, 0x32, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    F:/Windows/Go/Go17/src/database/sql/sql.go:1119 +0x356
database/sql.(*DB).query(0x12682060, 0x4f2f6a, 0x32, 0x0, 0x0, 0x0, 0x12667d01, 0x0, 0x0, 0x0)
    F:/Windows/Go/Go17/src/database/sql/sql.go:1079 +0xd4
database/sql.(*DB).Query(0x12682060, 0x4f2f6a, 0x32, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    F:/Windows/Go/Go17/src/database/sql/sql.go:1062 +0x7d
main.readTools(0x12682060)
    F:/Home-Mirror/Programming/Go-Workspace/src/idp/modifiche/main.go:98 +0x115
main.main()
    F:/Home-Mirror/Programming/Go-Workspace/src/idp/modifiche/main.go:130 +0xfd

goroutine 18 [chan receive]:
database/sql.(*DB).connectionOpener(0x12682060)
    F:/Windows/Go/Go17/src/database/sql/sql.go:730 +0x3c
created by database/sql.Open
    F:/Windows/Go/Go17/src/database/sql/sql.go:493 +0x231
eax     0x375b2000
ebx     0x7f4
ecx     0x375af03c
edx     0xff
edi     0x32dd6170
esi     0x32fc7460
ebp     0x0
esp     0x375cf428
eip     0x673dc7c8
eflags  0x10206
cs      0x23
fs      0x53
gs      0x2b

[neclepsio]

My problem is indeed a duplicate #18, which is in turn a duplicate of https://github.com/golang/go/issues/9457: adding import _ "runtime/cgo" solves it. Maybe you should document that, so you can save others the time to find out? (And also, speaking of saving time: importing runtime/cgo requires TDM-GCC and does not work with MinGW).

[mattn]

At the first, I don't still reproduce this. So I can't put the workaround in README.md .

[mattn]

Anyway, it seems a way to save my (or your's) time. updated README. Thanks.