Recurring 'Lock request time out period exceeded' exception

[m4nickroll]

Production site - load balanced using Azure. Seemingly random occurrences of 'Lock request time out period exceeded' exception. Resolved by scaling the database tier up or down. Returns within days however, preventing editing. Running 8.8.0 and following the recommended config from https://our.umbraco.com/documentation/getting-started/setup/server-setup/load-balancing/azure-web-apps. Screenshot of successful publish then the first 'Lock request time out period exceeded' exception a few minutes later:

[nul800sebastiaan]

@m4nickroll Oh no!

Generally, this COULD be an indication that your load balancing setup is not quite right, most of all, it is possible your backoffice is not isolated and accessible from multiple servers.

However, we've been fixing some of these errors, some occurred due to transient errors on SQL Azure, which might be the problem in your case. The latest fix for that is available in the recently released version 8.10.0: https://github.com/umbraco/Umbraco-CMS/pull/9543 - this fix looks directly related to your error. I would recommend you try out the latest version (8.10.1) now and see if that helps. We also intend to patch this for 8.8.x but I don't have a release date for a patch yet I'm afraid.

[m4nickroll]

@nul800sebastiaan thanks for your response.

Our backoffice is on a dedicated master app and /umbraco is not available from either of our two front-end apps. Master/Replica server role is also set correctly.

We have a separate issue with Umbraco Form submissions taking a while before producing a 504. The last time we had the lock timeout issue we noticed a (failed) form submission just before the lock timeouts began. Would that shed any light?

That issue certainly seems relevant though - will take a look at 8.10

[SimonAntony]

@nul800sebastiaan hello stranger :-)

I've been getting this more and more since 8.9 and it's progressively getting worse (even after upgrading to 8.11 today). Setup is Dedicated server running IIS/SQL Server 2017 (so no azure).

Always happens after doing a few publish events (updating content) - only way to fix it is to restart SQL Server which is far from ideal, god knows how i'd do this if it was Azure.

Log entry:

2021-01-28 11:49:01,926 [P15776/D2/T17] ERROR  Umbraco.Core.Persistence.UmbracoDatabase - Exception (3909210e).
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
ClientConnectionId:9df42156-ac71-4e50-91b4-e78a159e0483
Error Number:1222,State:51,Class:16
2021-01-28 11:49:02,112 [P15776/D2/T17] ERROR  Umbraco.Web.Editors.ContentController - Unhandled controller exception occurred for request 'http://site.com.local/umbraco/backoffice/UmbracoApi/Content/PostSave'
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) in D:\a\1\s\src\Umbraco.Core\Persistence\FaultHandling\RetryPolicy.cs:line 172
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, TimeSpan timeout, Int32[] lockIds) in D:\a\1\s\src\Umbraco.Core\Persistence\SqlSyntax\SqlServerSyntaxProvider.cs:line 272
   at Umbraco.Core.Services.Implement.ContentService.Save(IContent content, Int32 userId, Boolean raiseEvents) in D:\a\1\s\src\Umbraco.Core\Services\Implement\ContentService.cs:line 782
   at Umbraco.Web.Editors.ContentController.SaveAndNotify(ContentItemSave contentItem, Func`2 saveMethod, Int32 variantCount, Dictionary`2 notifications, SimpleNotificationModel globalNotifications, String invariantSavedLocalizationKey, String variantSavedLocalizationKey, String cultureForInvariantErrors, Boolean& wasCancelled) in D:\a\1\s\src\Umbraco.Web\Editors\ContentController.cs:line 924
   at Umbraco.Web.Editors.ContentController.PostSaveInternal(ContentItemSave contentItem, Func`2 saveMethod, Func`2 mapToDisplay) in D:\a\1\s\src\Umbraco.Web\Editors\ContentController.cs:line 712
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__6.MoveNext()
ClientConnectionId:9df42156-ac71-4e50-91b4-e78a159e0483
Error Number:1222,State:51,Class:16

[Shazwazza]

Just some info: when you see Lock request time out period exceeded this isn't an SQL connection timeout or load balancing problem, or a SQL problem. We use distributed locks to synchronize access to resources. Like a c# lock, but distributed so it's a lock that can be used by all active processes. The reason why this exception can occur can be many things:

• long running bulk DB operations having a write lock while other requests are waiting for a read lock
• having tons of service calls making write locks while others are waiting for read locks - this occurred with membership access a lot which we fixed in several places for this one https://github.com/umbraco/Umbraco-CMS/issues/8433
• service calls making write locks that might not need them
• poor query performance holding a write lock

Currently the timeout value is hard coded and isn't very long and there is a plan to increase this timeout to more lenient (like a c# lock).

However, there are things you can do to mitigate the problem but this depends on your website:

• Don't use Umbraco.Core.Services. in your front-end rendering code. I realize in some edge cases this may not be possible but any usages of this should be reviewed to ensure this is absolutely required. Almost all Umbraco.Core.Services. (especially IContentService) will require read and write distributed locks on all DB queries. In the case of https://github.com/umbraco/Umbraco-CMS/issues/8433 it was due to the front-end membership requiring read locks on each request which we've fixed.
• Definitely don't 'write' on a front-end read-only request

only way to fix it is to restart SQL Server

To me this indicates something that is holding a write lock for a very very very long time. Perhaps like a long running uSync or Deploy operation executing while people are trying to edit content. I'm unsure if that sounds accurate but restarting SQL server is very extreme since a write lock will always be released after the SQL operation holding it - unless there's code somewhere outside of core that is not utilizing these locks correctly (i.e. in a using).

[SimonAntony]

Thanks for the reply @Shazwazza - appreciate it. For the site i'm having issues with (the only v8 one i'm currently on), its a pretty standard build, no custom calls to write to/from services, no members etc - however it is running uSync both local and server (which both use a shared db at the moment whilst being developed which is server side).

Basically all i'm doing is publishing content, after what is usually between 5/8 publishes, I need to then restart SQL as get the timeout lock.

I'm going to disable uSync and see if it improves, i'll update back when I know one way or another.

[SimonAntony]

@Shazwazza Update, removed usync, clean deploy to live and still getting the timeout issues :(

[Shazwazza]

@SimonAntony

which both use a shared db at the moment whilst being developed which is server side

How much writing activity is taking place on both? Sharing a DB on writing means sharing distributed locks too. This is also multiple master instances writing to a DB which we don't support (but I think you are aware and prob undertand the risks). Having to restart the SQL server is still something new to me. Locks should always expire so this doesn't seem right.

Can you replicate this behavior locally without a shared DB? i.e. if you are running the exact same code locally with the same database locally (without other connections to the DB).

Basically all i'm doing is publishing content, after what is usually between 5/8 publishes, I need to then restart SQL as get the timeout lock.

Are you only seeing these errors on live?

[SimonAntony]

@Shazwazza

How much writing activity is taking place on both? Sharing a DB on writing means sharing distributed locks too. This is also multiple master instances writing to a DB which we don't support (but I think you are aware and prob undertand the risks).

Very little - it's a new build so setup is local build, db is on a vpn connection on the server so not much activity at all really. Only occurs on content saves and usually after 6/8 publishes.

The server holds a staging url so I can do more testing, however hardly any content publishes happen on there really.

Multiple master, yup i'm content with that - this is always the way i've done new builds (it's actually my own new company site in this case) so strange it only started to happen since 8.9.1

Can you replicate this behavior locally without a shared DB? i.e. if you are running the exact same code locally with the same database locally (without other connections to the DB).

Not tried it recently but will have a go - if I remove the staging site from the equation completely, this is effectively what i'm doing (albeit SQL is remote but that could be an Azure db connection anyway).

Are you only seeing these errors on live?

It's not live yet but I do see them on the staging version as noted above.

[timmather]

We are having a similar problem with a 8.6+ build (now on 8.12). Single server setup iis and sql on the same box. After 6/7 saves we get this error, worse with multiple editors, nothing special running at all, tried increasing timeout using writelock setting, content started to disappear across the site. It’s a big problem we have two sites waiting to go live this week with 20 editors and they can’t use the site. Never been a problem before on over 80 deployments.

[timlaughlin]

This has become a problem issue for us as well. We are creating nodes while importing content from a legacy system. Everything goes well for many new nodes and out of no where it will throw this exception and doesn't seem to recover quickly.

[brannmark]

We are seeing this problem on Umbraco 8.12.2 running against a SQL Server 2019. Prior to the lock we can see in our logs that it throws an exception on Umbraco.Web.Scheduling.IBackgroundTask everytime a lock situation begins. Only way to get the site back is to kill the SQL server processes for the database that are in a waiting state.

Are you also seeing this error before the errors on lock? Only occurs the first time when a lock situation is triggered, after that only the lock errors are logged.

[Shazwazza]

@brannmark that stack trace looks like the problem of ExecutionContext leaking. Make sure you are running the very latest Examine version and also try 8.13.0 - specifically for this https://github.com/umbraco/Umbraco-CMS/pull/9953

[brannmark]

@Shazwazza Thanks for your input. Will try to upgrade the site today and see if that solves the problem. Will report back if that solved it.

[brannmark]

@Shazwazza Upgrade to 8.13.0 seams like it might have solved the issue. Thanks for the feedback.

[drummerboy46]

@Shazwazza is ExecutionContext leaking a problem in Umbraco 7 too? We have a very similar problem on our version 7 build, it seems to kick off one or twice a month, ALWAYS when editors are trying to publish.

2021-05-12 09:26:05,178 [P6044/D3/T25] ERROR Umbraco.Core.Persistence.UmbracoDatabase - Exception (b2be5c7f). System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TrySetMetaData(_SqlMetaDataSet metaData, Boolean moreInfo) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at Umbraco.Core.Persistence.PetaPocoCommandExtensions.<>c__DisplayClass5_0.<ExecuteReaderWithRetry>b__0() at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func1 func) at Umbraco.Core.Persistence.Database.d741.MoveNext() ClientConnectionId:b7301278-5080-4c82-98ee-f50d68241fef Error Number:-2,State:0,Class:11 2021-05-12 09:26:05,178 [P6044/D3/T25] DEBUG Umbraco.Core.Sync.DatabaseServerMessenger - Complete (took 30003ms) 2021-05-12 09:26:05,178 [P6044/D3/T25] ERROR Umbraco.Web.BatchedDatabaseServerMessenger+InstructionProcessing - Failed (will repeat). System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TrySetMetaData(_SqlMetaDataSet metaData, Boolean moreInfo) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at Umbraco.Core.Persistence.PetaPocoCommandExtensions.<>cDisplayClass5_0.b0() at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func) at Umbraco.Core.Persistence.Database.d741.MoveNext() at System.Collections.Generic.List1..ctor(IEnumerable1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions() at Umbraco.Core.Sync.DatabaseServerMessenger.Sync() at Umbraco.Web.BatchedDatabaseServerMessenger.InstructionProcessing.TryPerformRun() at Umbraco.Web.BatchedDatabaseServerMessenger.InstructionProcessing.PerformRun() ClientConnectionId:b7301278-5080-4c82-98ee-f50d68241fef Error Number:-2,State:0,Class:11`

[lindeberg]

@drummerboy46 Possibly related issue?: https://github.com/umbraco/Umbraco-CMS/issues/10362

[lindeberg]

Related Our-forum posts with people getting this issue fairly recently as well:

• Examine index problems & members can not login "Lock request time out period exceeded." error
• Cannot acquire MainDom/Lock request timeout
• System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
• Lock request time out period exceeded exceptions from MemberService (v8.8.0)
• SQL lock request timeout
• Lock request time out period exceeded.

[deekoulos]

getting this exception also.

Version 8.14.3 Examine 1.1.0

Exception:
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.ObtainWriteLock(IDatabase db, TimeSpan timeout, Int32 lockId)
   at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, Int32[] lockIds)
   at Umbraco.Core.Scoping.Scope.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`1 obtainLock, Action`2 obtainLockTimeout, Nullable`1 timeout, Int32[] lockIds)
   at Umbraco.Core.Scoping.Scope.WriteLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)
   at Umbraco.Core.Scoping.Scope.WriteLock(Int32[] lockIds)
   at Umbraco.Core.Services.Implement.MemberService.Save(IMember member, Boolean raiseEvents)
   at Fb.Core.Controllers.RenderMvc.ProfileController.Index(ContentModel model)
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c.<BeginInvokeSynchronousActionMethod>b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_0.<InvokeActionMethodFilterAsynchronouslyRecursive>b__0()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_6.<BeginInvokeAction>b__4()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__1(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
ClientConnectionId:fb0cb4ca-a4be-40a9-8170-48e725f14047
Error Number:1222,State:51,Class:16

Exception:
System.Data.SqlClient.SqlException (0x80131904): Lock request time out period exceeded.
The statement has been terminated.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Umbraco.Core.Persistence.FaultHandling.RetryPolicy.ExecuteAction[TResult](Func`1 func)
   at NPoco.Database.ExecuteNonQueryHelper(DbCommand cmd)
   at NPoco.Database.Execute(String sql, CommandType commandType, Object[] args)
   at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.ObtainWriteLock(IDatabase db, TimeSpan timeout, Int32 lockId)
   at Umbraco.Core.Persistence.SqlSyntax.SqlServerSyntaxProvider.WriteLock(IDatabase db, Int32[] lockIds)
   at Umbraco.Core.Scoping.Scope.LockInner(Guid instanceId, Dictionary`2& locks, HashSet`1& locksSet, Action`1 obtainLock, Action`2 obtainLockTimeout, Nullable`1 timeout, Int32[] lockIds)
   at Umbraco.Core.Scoping.Scope.WriteLockInner(Guid instanceId, Nullable`1 timeout, Int32[] lockIds)
   at Umbraco.Core.Scoping.Scope.WriteLock(Int32[] lockIds)
   at Umbraco.Core.Services.Implement.MemberService.Save(IMember member, Boolean raiseEvents)
   at Fb.Core.Controllers.RenderMvc.ProfileController.Index(ContentModel model)
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c.<BeginInvokeSynchronousActionMethod>b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_0.<InvokeActionMethodFilterAsynchronouslyRecursive>b__0()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_2.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_6.<BeginInvokeAction>b__4()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3_1.<BeginInvokeAction>b__1(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.<>c.<BeginExecuteCore>b__152_1(IAsyncResult asyncResult, ExecuteCoreState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.<>c.<BeginProcessRequest>b__20_1(IAsyncResult asyncResult, ProcessRequestState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
ClientConnectionId:fb0cb4ca-a4be-40a9-8170-48e725f14047
Error Number:1222,State:51,Class:16

[TommyKolodziejczyk]

Hi, I found one more case for sql lock issue connected with BatchedDatabaseServerMessenger and loadbalancing

when we are importing in background task (initiated via http request), changes are propagated to frontend server at end of request. sometimes it can be huge number od nodes to be updated, looks like system bloks everything on flush batch

when we used modifed custom meesenger to triger flush after every save. issue is not visible but i assume db is still locked but only for one update and is processed before timeout.

public class CustomServerMessenger : ICustomServerMessenger
    {
        private readonly IServerMessenger _umbracoMessenger;
        private readonly ILogger _logger;

        public CustomServerMessenger(IServerMessenger umbracoMessenger, ILogger logger)
        {
            _umbracoMessenger = umbracoMessenger;
            _logger = logger;
        }

        public void FlushBatch()
        {
            if (_umbracoMessenger is BatchedDatabaseServerMessenger batchedDatabaseServerMessenger)
            {
                var messengers = (IList) HttpContext.Current.Items["BatchedDatabaseServerMessenger"];
                if (messengers != null && messengers.Count > 0) {
                    _logger.Debug<CustomServerMessenger>("BatchedDatabaseServerMessenger -> Cache refresh for {messengers.Count} items");
                    batchedDatabaseServerMessenger.FlushBatch();
                }
            }
        }

        public void Dispose()
        {
            FlushBatch();
        }
    }
}

when we use hangfire the FlushBatch it's even not triggered ( #10908 ), when we triggering it we have sam scenario as above: one time at the end -> lock issues on every save -> limited issue with lock

---

additional info: similar scenario we see in queue mechanism we talked about with @KevinJump, as solution for sql lock if batch size is 100 nodes we have 3 minutes of lock issue after finishing one request. .FlushBatch is triggered at end of batch

[HARVS1789UK]

I am continually seeing this issue in my Umbraco v8 site, previously running v8.11.1 now updated to v8.18.4 and still seeing the issue.

My site is hosted in Azure (App Service, Azure SQL and Blob Storage etc) and the frontend is load balanced (and to my knowledge, set up correctly, explicit Master/Replica servers specified and all data manipulation requested go to the Master install) but I don't think that's related as I see this issue when only a single instance of the frontend App Service is running still.

I was hoping that my upgrade to v8.18.4 (meaning I now have a configurable SQL Lock Timeout property, defaulting to 5s) plus all associated NuGet packages (like Examine) would have resolved this, but it seems not.

Hard to identify which requests/queries/transactions are causing the timeout to be hit, as I can't profile Azure SQL but from taking a look at my 'slow running queries' amongst a couple of sizeable SQL queries that seem to be dealing with getting/working out lists of contentTree items, considering which version is newest etc, I can also see read and write operations on the umbracoLock table frequently falling into the >5s lock timeout e.g:

Read from umbracoLock: (@0 int)UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0

Write to umbracoLock: (@0 int)UPDATE umbracoLock WITH (REPEATABLEREAD) SET value = (CASE WHEN (value=1) THEN -1 ELSE 1 END) WHERE id=@0

Due to the use of parameterized queries, I can't tell which lock value(s) this relates to, but if I was guessing it'd be contentTree.

This is an issue for me on a client production site, which although not insubstantial is hardly a beast in terms of user base, site content, volume of read/write operations. The site is also running with decent infrastructure (not a running on a potato) so I can't imaging I am pushing the boundaries of what Umbraco, SQL and the other Azure infrastructure can/should support?

As demonstrated by @lindeberg 's comment above, listing at least 6 blog posts, plus a number of other GitHub issues raising the same problem, I am far from alone here, yet this issue has been open for 18 months and hasn't even been given an assignee or any tags/milestones yet.....come on Umbraco, surely someone from the core team must be able to offer some more actionable feedback for us to work with?

[theworkstim]

The only way we could rectify this problem was to have each of our developers run their own db locally. If we start to use more than one front end (website running the same code) on the same db (never a problem in the past) we experience this error.

What I’m struggling with most is Umbraco lack of response to what is a big problem for a lot of people developers. With focus on 9 and 10 I fear this will now be left unresolved.

[Shazwazza]

Having several devs connected to the same database means you are effectively load balancing with multiple primary servers which has never been supported.

[VahidCh013]

I have the same issue with Umbraco 8.6.4. I can't edit any contents but when I try to create a new content it will work then I can edit other contents. But after a while the issue happens again and to fix this I need to follow the above mentioned process.