Session lock record with PHP 7.0.9

[CassianoCarraro]

I'm using a Memcached session handler normally in PHP 5.6. But with PHP 7 session don't retrieved properly and the follow warning message was triggered in PHP error log: PHP Warning: session_start(): Unable to clear session lock record.

Any suggestion?

[terryzwt]

I fixed the problem using below config. php:5.6.18

[session]
session.lazy_write = 0

[memcached]
memcached.sess_locking = On
memcached.sess_lock_wait = 150000
memcached.sess_lock_max_wait = 30;
memcached.sess_lock_expire = 0;
memcached.sess_consistent_hash = Off
memcached.sess_remove_failed = 1
memcached.sess_number_of_replicas = 0
memcached.sess_binary = Off
memcached.sess_randomize_replica_read = Off
memcached.sess_connect_timeout = 1000
memcached.sess_sasl_username = NULL
memcached.sess_sasl_password = NULL
memcached.compression_type = "fastlz"
memcached.compression_factor = "1.3"
memcached.compression_threshold = 2000
memcached.serializer = "igbinary"
memcached.use_sasl = Off
memcached.store_retry_count = 2

[lichunqiang]

@tvlooy can you share your full configuretions?

After try the current master branch, and below settings, it does not work for me.

@sodabrew I tried the lastest commit, but not works.

Is any update for this @all ?

[tvlooy]

$ php -v
PHP 7.1.8 (cli) (built: Aug  7 2017 13:46:59) ( NTS )
Copyright (c) 1997-2017 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.1.8, Copyright (c) 1999-2017, by Zend Technologies
$ php -i | grep ^memcached
memcached
memcached support => enabled
memcached.compression_factor => 1.3 => 1.3
memcached.compression_threshold => 2000 => 2000
memcached.compression_type => fastlz => fastlz
memcached.default_binary_protocol => no value => no value
memcached.default_connect_timeout => 0 => 0
memcached.default_consistent_hash => no value => no value
memcached.serializer => igbinary => igbinary
memcached.sess_binary_protocol => no value => no value
memcached.sess_connect_timeout => 1000 => 1000
memcached.sess_consistent_hash => 1 => 1
memcached.sess_lock_expire => 0 => 0
memcached.sess_lock_max_wait => not set => not set
memcached.sess_lock_retries => 2000 => 2000
memcached.sess_lock_wait => not set => not set
memcached.sess_lock_wait_max => 150 => 150
memcached.sess_lock_wait_min => 150 => 150
memcached.sess_locking => 1 => 1
memcached.sess_number_of_replicas => 0 => 0
memcached.sess_persistent => no value => no value
memcached.sess_prefix => memc.sess.key. => memc.sess.key.
memcached.sess_randomize_replica_read => no value => no value
memcached.sess_remove_failed_servers => no value => no value
memcached.sess_sasl_password => no value => no value
memcached.sess_sasl_username => no value => no value
memcached.sess_server_failure_limit => 0 => 0
memcached.store_retry_count => 2 => 2

[terryzwt]

Finally I fixed the problem in symfony. reference to this: https://labs.madisoft.it/scaling-symfony-sessions-with-memcached/

The main idea is add a wrapper to addServer to void multiple memcache session create.

[lichunqiang]

hi @tvlooy can you take out the memcached version and libmemcached version?

@terryzwt I just using single server now,

I haid tried memcached version 3.0.0 and 3.0.3 and the master branch, all failed. I have no idea what issued the problem..

[tvlooy]

php-memcached = 3.0.3 libmemcached = 1.0.16 memcached = 1.4.15

[terryzwt]

@lichunqiang you can try

telnet your-memcache-server-ip 11211

then type "stats", check the "curr_connections" session to get how many connection to memcache server. Then you will be realize the problem.

[lichunqiang]

@terryzwt 你的意思是连接数导致的问题？但是我现在是单机进行测试的。英语不好，还是中文方便。。

[tvlooy]

thanks Google for Google translate ;-) the number of connections can be a problem but I don't see how terryzwt got to that conclusion ...

@terryzwt about your blog article. Unlike Memcached, Redis does not have session locking support. I just wanted to point that out to you maybe you can elaborate on that in your next blog article. I would be interested in your findings

[terryzwt]

@lichunqiang 我在实际场景中,memcache连接数超过4000,就很容易出现lock问题.后来在代码层面进行连接复用,大概控制在200左右,基本lock问题就不会出现了.链接复用的方式就是上面告诉你的博客链接里的方法.在addServer外面封装一层.

@tvlooy it's not my blog :) My blog is www.dplor.com, most of the articles are Chinese.

[lichunqiang]

@tvlooy Hi, This can resolved the problem, but casued a new issue, the session_start in php slow log.

It is casued by time of the settings?

[fishfacemcgee]

@tvlooy With the settings you have, have you seen any issues with sessions started for a request that eventually hits a fatal error staying locked until the lock expiration time (which would be the max execution time)? I'm using similar settings on a staging environment and the locking issue is effectively resolved except in this edge case. I don't want to assume that fatal errors are impossible in my app so I'm trying to figure out if there's anything I can do besides setting the lock expiration time to something lower.

[adv4000]

For me with PHP7.1 Change in /etc/php-7.1.d/50-memcached.ini from memcached.sess_locking = On to memcached.sess_locking = Off

Looks like fixed this error.

[Hotte512]

Yeah... but no more session locking :(

Hope it'll be fixed someday...

[tvlooy]

@lichunqiang if that is slow, that means your memcache is slow. It could be that the server where memcache runs is under load @fishfacemcgee no I didn't. What kind of setup are you running? @adv4000 yeah. It's pretty normal that problems with a feature (session locking) go away when you turn off that feature

[SvenRtbg]

Reading through the comments, I see several similarities to what I experienced.

The question was asked if the new default settings in #350 work better. Yes, they do. Waiting at least 1000ms (as the old default for sess_lock_wait_min) is ridiculous: PHP scripts usually are executed within 100ms if they do basic stuff - why would anyone be willing to wait an entire second? My experience is that you should fine-tune the minimum value to something that is slightly larger than the minimum execution time of a script on your server. I have chosen 100ms.

The maximum waiting time or the reason for exponential backoff has no good explanation, but I'm fine with it existing, I suppose that you would be able somehow to overload a Memcached server with lock checks. However, the current default of 2000ms is also too big for my liking, I chose 400ms.

The thing that is not documented very good is how these two values interact with the sess_lock_retries and the sess_lock_expire values. My understanding is that the sess_lock_expire value should be greater than max_execution_time, and that the retries should also be a reasonably high number.

Why should sess_lock_expire be greater than max_execution_time? Because max_execution_time does not count every time PHP is using - but sess_lock_expire does. If PHP waits for a remote system to answer, like databases, that time is not counted when evaluating the max_execution_time, so you can end up executing scripts for longer than the max_execution_time suggests. The lock however is based on wall time. My suggestion would be to set the sess_lock_expire value to at least 1.5 times the value of max_execution_time, just to ensure that an aborted script still is within the time the lock exists.

Regarding sess_lock_retries, this value should also be high enough to cover for edge cases. Assume that you have three long running requests coming in at the same time - each run for 55 seconds. The lock expiry time is set to 100 seconds. Min/max wait time is set to 100ms.

If you want to retry for the whole lifetime of a lock, you'd have to set sess_lock_retries to 100s/100ms = 1000. However, the first request locks the session for 55 seconds, then releases the lock. The second request aquires the lock after around 55.1 seconds, creating a new lock that would last for another 100 seconds. It will release that lock after 110 seconds (55s * 2 requests). However, the third request has given up after it retried for 100 seconds, or 1000 times. It will corrupt the session because it overrides the existing lock, reads the session in its currently stored state (which is after the first script ran), and will write back it's new state, overwriting any changes that the second script had made.

I understand that having "long running web request scripts" is not ideal, but that's what I have to deal with - and the documentation should give some hints regarding the configuration considerations for these values.

So TL;DR:

• __sess_lock_wait_min__ = aim for a resonably low value that is slightly greater than your average "fast" script execution time.
• __sess_lock_wait_max__ = aim for a value that looks like your Memcached server can handle it if you experience some heavy load situations.
• sess_lock_expire = aim for a high enough value that should at least be max_execution_time, but possibly multiple times that time. Consider if you'd be likely to experience long running scripts and concurrent calls for the same session id. As an upper boundary for this value, consider that a single session will be unresponsive for at least this long if things go really bad.
• sess_lock_retries = higher is better. There is nothing wrong with retrying for longer than the lock expiry time is configured. Remember that if you stop retrying, you'll possibly corrupt the existing session because the script will start using the session in it's existing state despite it being locked.

[onassar]

Did anyone come to a conclusive, consistent answer on this one? I'm running: php-dev: 1:7.2+60ubuntu1 (resulting in 7.2.5-0ubuntu0.18.04.1) libmemcached: 1.0.18-4.2 php-memcached: 3.0.1+2.2.0-1build2

My sessions are running through AWS ElastiCache

After reading this thread and this post, I'm at a loss as to what the best way to: 1) Reproduce this issue consistently 2) Resolve the issue

[fishfacemcgee]

@onassar I don't think there is a way to "solve" the issue from the config side of things besides not using locked sessions. At least I wasn't able to; I could just minimize the frequency and severity of the locking by using the values proposed as new defaults in #350. The problem (based off the ma.ttias.be post you linked) is that Request B has to wait for Request A's session to end. Depending on how your application interacts with Sessions, this may require the request to end, which, depending on the speed of your requests, may be a while.

In my case, every request always started a session, and never closed it until the request ended. What we've been doing in our app to address the issue is change both behaviors:

• Only start session on requests that actually need to use $_SESSION
• Close the session (at least the request's ability to write to the session) as soon as you're done using it, especially if it's early in the request.

If you know for certain that your application does not rely on the session data in a way that mandates locking, you could just disable session locking and call it day. We're pretty sure we can, but aren't comfortable making that risk so we're switching everything over to this new system before investigating disabling locking entirely.

[onassar]

Thanks for the writeup @fishfacemcgee At the moment, I've simply set session.lazy_write to false and am going to monitor our server to see what happens. The reason I presumed a possible "solution" was because I just upgraded from PHP 5.6.x when I started noticing this. So my presumption was that something in the newer libraries (PHP, libmemcached, php-memcached, Ubuntu 18.04, newer apache, etc etc) isn't playing well together.

Or is that the default session and memcached configuration options in my PHP 5.6.x install were simply so "loose" that they didn't cause the issue to show it's head?

Ideally, I'd like to say that I know exactly when the writing is happening; but that would require a fair amount of refactoring to ensure that's the case, which could then introduce application/middle-wear bugs. So a rock and a hard place.

A few posts in this thread list session.lazy_write being set to false as not the "correct" way. Tbh, I don't understand what that setting does. PHP's docs on that are somewhat vague:

session.lazy_write, when set to 1, means that session data is only rewritten if it changes. Defaults to 1, enabled.

If you could enlighten me as to why setting this to false would fix the locking issue, I'd be all ears. :)

[woolardfa]

Just an FYI... have tested PHP 7.2 (7.2.6), the underlying issue where session_start returns true when it in fact failed to acquire the session lock, and subsequently corrupts the existing session data, has been fixed.

A call to session_start() now returns false when it failed to acquire the session lock.

[onassar]

Hmmm I'm running 7.2.7 and still running into the issue..

[Eimantas123]

php 7.2.15, libmemcached version 1.0.18 problems still occur.

[csev]

Just an update - I am using AWS Elasticache as my Memcache and PHP 7.1. Setting session.lazy_write = Off got rid of the message. I am 100% guessing here, but this behavior suggests that with lazy_write on, it holds the lock a little while longer - even though it knows the request is done and that the session is unmodified. When it writes it back the lock is released. Good thing my memcache is pretty fast / underutilized for now :)

[csev]

Update. I was able to watch error logs and tweak all these values live and figure out what worked and what did not. Turning off session.lazy_write only reduced the frequency of the errors. Also trying to set the _min and _max to 150 as described above does not stop the errors. The only thing that stopped the errors are to turn ini_set('memcached.sess_locking', '0'); - and then turn lazy_write back on. I read that Redis does not lock at all. So this feels like the simplest / best path. (This is PHP 7.1 and libmemcached 1.0.18 ). Hope this helps.

[woolardfa]

@csev Good news that you've determined your simplest/best path, and do not want to discourage you, but do want to offer some things to consider if you choose to keep using memcached for sessions.

Firstly, I'd recommend getting to a current version of PHP where the fundamental bug is fixed. The bug is not in libmemcached, or in php-memcached; it's in PHP.

The issue is not so much how php_memcached is configured (i.e. _min, _max), because like you said, changing how those are configured didn't eliminate the errors--and they weren't meant to, they were just meant to a) mimic the old session acquisition timings, and b) try to reduce the frequency of the errors based on how one particular application, Moodle, was designed.

The issue is more about how sessions are handled at the application level, and whether your application code uses PHP's session management on almost every browser request, like Moodle currently does, (page requests, and subsequent AJAX requests), or if it's more selective. Also, does the application code detect whether the $_SESSION super global is valid or not (most all the time we take that for granted, and that's the patch I added to Moodle's memcached session startup, until PHP was fixed).

So, you're right, going with no session locks will eliminate the errors, because that particular lock/semaphore entry won't be created to begin with, and no attempt will be made to remove it. If you're application will work with no concurrency checks (last to touch it wins), then yeah, that's the ticket.

[csev]

@woolardfa - Thanks a ton - My application is Tsugi - an EdTech app like Moodle. It can survive with "eventual consistency" and "last write wins". I am encouraged by your suggestion to move beyond 7.1 - what version of PHP is this fixed in? I am a little gun shy as, the road through and to 7.0 and 7.1 was kind of rocky, stuff was not getting fixed, and even the apt repos were wonky and non-standard. So when I got something working I just breathed a sigh of relief and froze my infrastructure. But that was a year ago and I am sure things are better so I am heartened to give it another go if I know what version to go to - I am nervous about jumping to the absolute latest. Again, thanks.

[SvenRtbg]

Nobody has been doing this in this issue, but I think in order to get the full picture of a problem report you should add at least all the settings I mentioned in my comment above: https://github.com/php-memcached-dev/php-memcached/issues/269#issuecomment-366285778

sess_lock_wait_min, sess_lock_wait_max, sess_lock_expire, sess_lock_retries. Note that the _min and _max times are basically irrelevant for the functioning of the session in general, they are being used to fine-tune timing behavior. The most important settings are lock_expire (also compared to your max_execution_time) and lock_retries. With these set the wrong way, you may end up overwriting your session unintentionally.

[woolardfa]

@csev The pain points in Moodle (all handled by moodlehq) were mostly in the 5.x -> 7.x migration. I imagine there ought to be very few going from 7.1 -> 7.3. We've tried to keep our sites fairly up-to-date, and didn't see any issues when we moved from 7.1.x. to 7.2.x.

I believe the bug was fixed in 7.2.6 onward, so the current stable of either 7.2 or 7.3 will be prudent... but then, like I said earlier, it will just mean that your application code can then correctly detect whether the session (lock) was acquired or not before it proceeds.

[csev]

@woolardfa I upgraded to 7.3 and everything is working fine and I am using the default settings and seeing no lock issues. Thanks for your advice.

[murraycollingwood]

I'm using php 7.3 and it's still a problem. I've gone with the config changes to see how this impacts the incidence. If it remains a problem then I'm seriously considering turning locking off.

[murraycollingwood]

I'm using php 7.3 and it's still a problem. The config changes _wait _min _max made no difference, in fact I think it got worse. I have now turned off locking and it seems to have solved the locking problem. I also enabled lazy_write, although I'm not sure if that was necessary.

However, as discussed above, turning off locking is dependent upon how you use your session data. I think most of my usage should be fine, I guess the next few days will be telling.

[dytyniuk]

Hello everyone.

I have read carefully all the comments and solutions proposed: starting with memcached.sess_lock_* settings changed and up to sessions locks turned off.

Frankly, I'm strongly convinced, the issue is not PHP-related, while specific to this extension. Details are the following:

• an application is running in Docker image based on php/7.3.16-fpm-alpine with both memcache and memcached extensions installed (we need both, since this is our internal "base" image for all PHP apps)
• All off memcached.* and memcache.* settings are left default:

  
  memcache support => enabled
  memcache.allow_failover => 1 => 1
  memcache.chunk_size => 32768 => 32768
  memcache.compress_threshold => 20000 => 20000
  memcache.default_port => 11211 => 11211
  memcache.hash_function => crc32 => crc32
  memcache.hash_strategy => consistent => consistent
  memcache.lock_timeout => 15 => 15
  memcache.max_failover_attempts => 20 => 20
  memcache.prefix_host_key => 0 => 0
  memcache.prefix_host_key_remove_subdomain => 0 => 0
  memcache.prefix_host_key_remove_www => 1 => 1
  memcache.prefix_static_key => no value => no value
  memcache.protocol => ascii => ascii
  memcache.redundancy => 1 => 1
  memcache.session_prefix_host_key => 0 => 0
  memcache.session_prefix_host_key_remove_subdomain => 0 => 0
  memcache.session_prefix_host_key_remove_www => 1 => 1
  memcache.session_prefix_static_key => no value => no value
  memcache.session_redundancy => 2 => 2
  memcache.session_save_path => no value => no value

memcached support => enabled libmemcached version => 1.0.18 memcached.compression_factor => 1.3 => 1.3 memcached.compression_threshold => 2000 => 2000 memcached.compression_type => fastlz => fastlz memcached.default_binary_protocol => Off => Off memcached.default_connect_timeout => 0 => 0 memcached.default_consistent_hash => Off => Off memcached.serializer => php => php memcached.sess_binary_protocol => On => On memcached.sess_connect_timeout => 0 => 0 memcached.sess_consistent_hash => On => On memcached.sess_consistent_hash_type => ketama => ketama memcached.sess_lock_expire => 0 => 0 memcached.sess_lock_max_wait => not set => not set memcached.sess_lock_retries => 5 => 5 memcached.sess_lock_wait => not set => not set memcached.sess_lock_wait_max => 150 => 150 memcached.sess_lock_wait_min => 150 => 150 memcached.sess_locking => On => On memcached.sess_number_of_replicas => 0 => 0 memcached.sess_persistent => Off => Off memcached.sess_prefix => memc.sess.key. => memc.sess.key. memcached.sess_randomize_replica_read => Off => Off memcached.sess_remove_failed_servers => Off => Off memcached.sess_sasl_password => no value => no value memcached.sess_sasl_username => no value => no value memcached.sess_server_failure_limit => 0 => 0 memcached.store_retry_count => 2 => 2

* as soon as you set `session.save_handler=memcached` and put any memcached hostname to `session.save_path`, consecutive XHR calls to the backed start to fail with `session_lock` error
* but, as soon as you change your `session.save_handler` to "legacy" `memcache` (`session.save_handler=memcache`) and modily `session.save_path` a bit:

session.save_handler=memcache session.save_path=tcp://:/ # is still required even if you use default 11211

Everything works as it should.

So, either default settings for `memcached` are missing proper tuning compared to `memcache` ones. Or there's something wrong with sessions locking on `memcached` side. Again, I do think so, since "legacy" `memcache` works like a charm 🤷‍♂️ 

[michnovka]

I still dont think this is resolved. I had to fall back to using files for sessions

[martinsoenen]

Hello,

I also have this problem with PHP version 7.4.33 and memcached 1.6.19 . Is there anything we can do ? Disabling session.lazy_write did not help us.

[Misosooup]

Same here, still getting this issue with 7.4.33