Closed brotskydotcom closed 8 years ago
Going to try to fix this by adding redis, to see if its free level add-on works across dyno idle and perhaps even restart.
So the fix doesn't work due to concurrent redis access problems. Reopening.
So the concurrency problems were solved with 7bdceea but there is a new issue: dropped redis connections. When I connect to the redis server from the command line, I can see the server's client connection drop:
herring.redistogo.com:9763> client list
id=75 addr=10.180.86.27:60505 fd=6 name= age=143 idle=140 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=76 addr=50.136.220.100:56694 fd=7 name= age=5 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
herring.redistogo.com:9763> client list
id=76 addr=50.136.220.100:56694 fd=7 name= age=69 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
herring.redistogo.com:9763>
After this happens, the server crashes trying to talk down the broken connection. Reopening.
Fixed this with 12cde9f (merged with pull #14 and fixed with pull #15 fa9b98b). Now every redis operation is preceded with a PING to see if the connection has dropped, and if so it is properly closed and reopened. Now we get server logs that look like this:
Nov 26 09:59:02 susen-staging heroku/web.1: Starting process with command `susen`
Nov 26 09:59:05 susen-staging app/web.1: 2015/11/26 17:59:04 Connected to redis at "redis://redistogo:693c0d12329f8c825669c7338ad993da@herring.redistogo.com:9763/0"
Nov 26 09:59:07 susen-staging heroku/router: at=info method=POST path="/api/assign/" host=susen-staging.herokuapp.com request_id=cb075e59-17df-498b-9145-236649f31050 fwd="50.136.220.100" dyno=web.1 connect=1ms service=14ms status=200 bytes=369
Nov 26 09:59:07 susen-staging app/web.1: 2015/11/26 17:59:06 Added session 54e9399d-1c07-4ca4-80e2-92e5f1610466 step 4.
Nov 26 09:59:07 susen-staging app/web.1: 2015/11/26 17:59:06 Reloaded session 54e9399d-1c07-4ca4-80e2-92e5f1610466, puzzle "3-star", through step 3.
Nov 26 10:03:56 susen-staging heroku/router: at=info method=POST path="/api/assign/" host=susen-staging.herokuapp.com request_id=f295fc89-e575-4b74-a452-4a4135bac1a4 fwd="50.136.220.100" dyno=web.1 connect=2ms service=12ms status=200 bytes=339
Nov 26 10:03:56 susen-staging app/web.1: 2015/11/26 18:03:55 Lost connection to redis: EOF
Nov 26 10:03:56 susen-staging app/web.1: 2015/11/26 18:03:55 Reconnected to redis at "redis://redistogo:693c0d12329f8c825669c7338ad993da@herring.redistogo.com:9763/0"
Nov 26 10:03:56 susen-staging app/web.1: 2015/11/26 18:03:55 Added session 54e9399d-1c07-4ca4-80e2-92e5f1610466 step 5.
I don't know whether these connection failures happen with paid levels of Redis To Go or if they are a consequence of using the free level. I also need to do research on whether redigo can give me a heads-up when the connection closes, so I don't have to always PING first. But this does seem really fixed for now. Closing the issue.
Leaving a solver window open and idle for a few hours (or a few days) causes the free Heroku dyno to go idle and shut down. This flushes the server-side session, although the cookie persists on the client side.