nicktacular / php-mongo-session

A PHP session handler with a Mongo DB backend.
MIT License
18 stars 6 forks source link

session lockout on write concern exception #7

Closed rocksfrow closed 9 years ago

rocksfrow commented 10 years ago

Okay so lock() is updated to properly ignore the duplicate key and drops out on the wtimeout exception correctly (instead of retrying for $locktimeout).

Testing the write concern support I've added, I spotted a potential issue. In the event that a write concern error occurs, the lock remains on the nodes it did properly write to.

mongodb: "wtimeout causes write operations to return with an error after the specified limit, even if the required write concern is not fulfilled. When these write operations return, MongoDB does not undo successful data modifications performed before the write concern exceeded the wtimeout time limit."

I confirmed that firing off an unlock() in the event of a replication timeout (wtimeout), the lock is successfully removed, although you will end up with a second wtimeout occuring on that unlock event.

I have these changes local still, but I confirmed it works. As is, this works. The only negative is that you end up waiting for wtimeout twice in that event.

So if you have wTimeoutMS set to 5000, you'll wait 10 seconds total for both the lock and subsequent unlock to timeout.

                   if(preg_match('/duplicate key error/i', $e->getMessage())){
                    //duplicate key may occur during lock race, try again
                    continue;
+                 }elseif(preg_match('/replication timed out/i', $e->getMessage())){
+                   //replication error, force unlock to prevent lock-out
+                   $this->lockAcquired = true;
+                   $this->unlock($sid);
                  }else{
                    //log exception and fail lock
                    $this->log('exception: ' . $e->getMessage());
rocksfrow commented 10 years ago

I created this issue because I want to look into this replication time out exception to see if any data is available to see how many nodes it DID write to...

IE, I have 2 nodes, so using w=3 causes it to timeout writing to a 3rd missing node. I want to detect it wrote to 2 nodes, decrease the write concern to 2 and THEN call unlock(). That should remove the locks, and avoid the double timeout.

This will only be possible if any such data is available from the exception. Otherwise, I will commit and push up what I have now, which at least fixes the lockout issue.

rocksfrow commented 10 years ago

Actually I don't need, or care how many nodes it wrote to. I will override the write concern to 0 and unlock in this situation. That way it simply removes any that were written and then dies.

rocksfrow commented 10 years ago

The changes are pushed: https://github.com/rocksfrow/php-mongo-session/commit/b773eb5dc40984fcf83025c4d3d76a87e432338f

rocksfrow commented 10 years ago

So after stress testing this fix, my code is working as expected and no sessions end up locked out.

BUT, the stress testing revealed a different exception occuring that ALSO results in the lockout:

"Read timed out after reading 0 bytes, waited for 5.000000 seconds"

It's an exception on the ->insert() though.... I am wondering if this is an issue on my end or just another potential exception in this scenario. If so I need to catch it as well as the replication timed out.

I could just match on 'timed out', so it'll match all timeouts, and kick off an unlock() in that scenario... but I prefer not to try unlocking unless necessary.

nicktacular commented 9 years ago

This sounds like it's related to #9 -- is this correct? Can we close this and keep investigating there?

rocksfrow commented 9 years ago

@nicktacular Yes let's close this issue out anyway as it's not related to the original write concern exception I was receiving. I am not positive this Read time out is what's causing the stuck locks, but I suspect so as well. If I confirm otherwise I will open a separate issue.