Volatile db, pool stretching and other goodies

[haizaar]

Hi Frank, This is kind of a bold code change. Feature summary:

• Pool reconnects to database if connections become broken
• Pool now supports stretching - up to max_size connections will be open if all initial connections are in use
• Switched to subscription based model using Futures for connections waiting
• Explicit managing of free/busy connections - one connection allocated per operation. This how it was working before anyway, but this approach also eliminates subtle transaction related bugs that were reported earlier.
• Finally, PEP8 compliance :)

I hope you'll have a time to review it soon.

Unitests pass (except for Mogrify test, which also fails on your master and 1.0.0). Connections resiliency was tested by me manually - I'm not sure how to create unit tests for this, since it involves shutting down database server (unless we'll launch our own psql server during unittest).

[whardier]

You may be able to kill the socket object for a connection by exposing it to the test system through the pool.

[haizaar]

I'm not sure what you mean. Can you be more verbose?

You may be able to kill the socket object for a connection by exposing it to the test system through the pool.

— Reply to this email directly or view it on GitHubhttps://github.com/FSX/momoko/pull/60#issuecomment-34221080 .

[whardier]

First off.. great commit. Looks very nice.

You were looking for a way to test connection issues properly. You may be able to get access to the underlaying socket or tell the psycopg2 implementation to disconnect or possibly cause it to fault somehow. It would be interesting to see if there were a method to force psycopg2 to raise a connection error or timeout.

Otherwise I can suggest a tornado based TCP/IP based proxy method for dealing with this and incorporating that into testing.

[whardier]

There are some tools I used in VoIP testing as well that help including setting up a virtual switch using VDE2 and using the wirefilter connector to emulate loss and drops .. optionally killing it as needed for complete loss.

[FSX]

Oh, I think something like this would be useful: http://vaurien.readthedocs.org/en/1.8/index.html

It doesn't speak the PostgeSQL protocol, but maybe the TCP handler maybe be enough for us.

[haizaar]

Nice approach indeed! That should be a way to write proper unit tests!

I'm looking into set_session and server-side cursors support, and then this.

On Wed, Feb 5, 2014 at 10:37 PM, Frank Smit notifications@github.comwrote:

Oh, I think something like this would be useful: http://vaurien.readthedocs.org/en/1.8/index.html

It doesn't speak the PostgeSQL protocol, but maybe the TCP handler maybe be enough for us.

— Reply to this email directly or view it on GitHubhttps://github.com/FSX/momoko/pull/60#issuecomment-34237990 .

Zaar

[haizaar]

At the end I'm just closing connection to "kill" them. Works pretty good to check reconnect logic.

You are welcome to comment on contents of tests.py on my latest master.

Zaar

On Thu, Feb 6, 2014 at 5:12 PM, Zaar Hai haizaar@haizaar.com wrote:

Nice approach indeed! That should be a way to write proper unit tests!

I'm looking into set_session and server-side cursors support, and then this.

On Wed, Feb 5, 2014 at 10:37 PM, Frank Smit notifications@github.comwrote:

Oh, I think something like this would be useful: http://vaurien.readthedocs.org/en/1.8/index.html

It doesn't speak the PostgeSQL protocol, but maybe the TCP handler maybe be enough for us.

— Reply to this email directly or view it on GitHubhttps://github.com/FSX/momoko/pull/60#issuecomment-34237990 .

Zaar

Zaar

[FSX]

Whoah!