search

brianfoshee / aquaponics-data

0 stars 0 forks source link

Application Error - Too many database connections #24

Closed nathanprayzo closed 9 years ago

nathanprayzo commented 9 years ago

The error copied below was surfaced while stress testing the application. While it appears to be the result of heroku postgres hobby-dev restricting max db connections to 20, db connection limits may still be worth accounting for in the application.

2014-12-28T05:47:56.255801+00:00 app[web.1]: 2014/12/28 05:47:56 http: panic serving 10.228.26.250:42161: pq: too many connections for role "fidoqlepzkocvo" 2014-12-28T05:47:56.255805+00:00 app[web.1]: goroutine 83822 [running]: 2014-12-28T05:47:56.255806+00:00 app[web.1]: net/http.func��011() 2014-12-28T05:47:56.255808+00:00 app[web.1]: /usr/local/go/src/pkg/net/http/server.go:1100 +0xb7 2014-12-28T05:47:56.255810+00:00 app[web.1]: runtime.panic(0x8592a0, 0xc20826aea0) 2014-12-28T05:47:56.255811+00:00 app[web.1]: /usr/local/go/src/pkg/runtime/panic.c:248 +0x18d 2014-12-28T05:47:56.255813+00:00 app[web.1]: main.func��002(0x7f50a12f75c8, 0xc208110780, 0xc2083d6b60) 2014-12-28T05:47:56.255815+00:00 app[web.1]: /tmp/build_6e43d5e740a3c2539f1f6ef12ebd4721/.heroku/g/src/github.com/crakalakin/aquaponics-data/main.go:101 +0x43f 2014-12-28T05:47:56.255817+00:00 app[web.1]: net/http.HandlerFunc.ServeHTTP(0xc208001a90, 0x7f50a12f75c8, 0xc208110780, 0xc2083d6b60) 2014-12-28T05:47:56.255818+00:00 app[web.1]: /usr/local/go/src/pkg/net/http/server.go:1235 +0x40 2014-12-28T05:47:56.092634+00:00 heroku[router]: at=info method=POST path="/devices/MockClient4136/readings" host=gowebz.herokuapp.com request_id=e6a8ba9d-4940-4f63-8d71-e6ef3ed7359e fwd="72.91.164.227" dyno=web.1 connect=1ms service=69ms status=201 bytes=349 2014-12-28T05:47:56.255820+00:00 app[web.1]: github.com/gorilla/mux.(_Router).ServeHTTP(0xc2080199a0, 0x7f50a12f75c8, 0xc208110780, 0xc2083d6b60) 2014-12-28T05:47:56.255822+00:00 app[web.1]: /tmp/build_6e43d5e740a3c2539f1f6ef12ebd4721/.heroku/g/src/github.com/crakalakin/aquaponics-data/Godeps/_workspace/src/github.com/gorilla/mux/mux.go:98 +0x292 2014-12-28T05:47:56.255823+00:00 app[web.1]: net/http.(_ServeMux).ServeHTTP(0xc2080226c0, 0x7f50a12f75c8, 0xc208110780, 0xc2083d6b60) 2014-12-28T05:47:56.255825+00:00 app[web.1]: /usr/local/go/src/pkg/net/http/server.go:1511 +0x1a3 2014-12-28T05:47:56.255827+00:00 app[web.1]: net/http.serverHandler.ServeHTTP(0xc208004f00, 0x7f50a12f75c8, 0xc208110780, 0xc2083d6b60) 2014-12-28T05:47:56.255828+00:00 app[web.1]: /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f 2014-12-28T05:47:56.255830+00:00 app[web.1]: net/http.(_conn).serve(0xc20825dd00) 2014-12-28T05:47:56.255831+00:00 app[web.1]: /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e 2014-12-28T05:47:56.255833+00:00 app[web.1]: created by net/http.(_Server).Serve 2014-12-28T05:47:56.255834+00:00 app[web.1]: /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

brianfoshee commented 9 years ago

@letsgitgrowing the max is 20 connections. Go has a connection pool that the database/sql package uses under the hood, and we can tune it to our liking http://golang.org/pkg/database/sql/#DB.SetMaxOpenConns and http://golang.org/pkg/database/sql/#DB.SetMaxIdleConns. We can also use pgbouncer (https://github.com/heroku/heroku-buildpack-pgbouncer) which is a very performant postgres connection proxy and pool.

The problem may have been that the row limit was reached, though. I forwarded you the email from heroku.

nathanprayzo commented 9 years ago

@crakalakin After setting MaxOpenConns to 20 and MaxIdleConns to 20, almost all of the application metrics improved.

maxopenconns 20 -benchmarch-hour1

brianfoshee commented 9 years ago

@letsgitgrowing looks great!