search

OctopusDeploy / Issues

| Public | Bug reports and known issues for Octopus Deploy and all related tools
https://octopus.com
162 stars 20 forks source link

Exceptions when getting the value of a Lazy<T> are being cached - SpacesCache, RandomConstant #7848

Closed akirayamamoto closed 2 years ago

akirayamamoto commented 2 years ago

Team

Severity

a few customers were affected

Version

Present on 2021.2.3085

Latest Version

I could reproduce the problem in the latest build

What happened?

When getting the value of a Lazy object and an exception is thrown, this exception will be cached.

This happens when using the LazyThreadSafetyMode.ExecutionAndPublication (default) and the value factory constructor)).

This problem will exacerbate a transient connection to the database for example.

When this happens, the exception will continue to be thrown every time we try to access the Lazy value even if the database connection has been restored.

We observed this issue on the SpacesCache and the RandomConstant classes.

Reproduction

N/A

Error and Stacktrace

Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)\
   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)\
   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\
   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\
   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)\
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\
   at Microsoft.Data.SqlClient.SqlConnection.Open()\
   at Nevermore.Transient.RetryPolicy.<>c__DisplayClass26_0.<ExecuteAction>b__0()\
   at Nevermore.Transient.RetryPolicy.ExecuteAction[TResult](Func`1 func)\
   at Nevermore.Transient.RetryPolicy.ExecuteAction(Action action)\
   at Nevermore.Transient.DbConnectionExtensions.OpenWithRetry(DbConnection connection, RetryPolicy retryPolicy)\
   at Nevermore.Transient.DbConnectionExtensions.OpenWithRetry(DbConnection connection)\
   at Nevermore.Advanced.ReadTransaction.Open()\
   at Nevermore.RelationalStore.BeginReadTransaction(RetriableOperation retriableOperation, String name)\
   at Nevermore.TableColumnNameResolvers.JsonLastTableColumnNameResolver.GetColumnNames(String schemaName, String tableName)\
   at Nevermore.TableColumnsCache.<>c__DisplayClass0_0.<GetOrAdd>b__0(String _)\
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)\
   at Nevermore.TableColumnsCache.GetOrAdd(String schemaName, String tableName, Func`3 valueFactory)\
   at Nevermore.TableColumnNameResolvers.CachingTableColumnNameResolver.GetColumnNames(String schemaName, String tableName)\
   at Nevermore.Advanced.ReadTransaction.GetColumnNames(String schemaName, String tableName)\
   at Nevermore.Advanced.QueryBuilders.TableSourceQueryBuilder`1.CreateSimpleTableSource()\
   at Nevermore.Advanced.QueryBuilders.TableSourceQueryBuilder`1.CreateSelectBuilder()\
   at Nevermore.Advanced.QueryBuilders.SourceQueryBuilder`1.get_Builder()\
   at Nevermore.Advanced.QueryBuilders.SourceQueryBuilder`1.CalculatedColumn(String expression, String columnAlias)\
   at Octopus.Core.RelationalStorage.Caching.LatestAndCountsFetcher.GetLatestAndCountQuery[TModel](IRawReadQueryExecutor trn, SpacePartitionsDefinition partitions)+MoveNext() in ./source/Octopus.Core/RelationalStorage/Caching/LatestAndCountsFetcher.cs:line 148\
   at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)\
   at System.Collections.Generic.SparseArrayBuilder`1.ReserveOrAdd(IEnumerable`1 items)\
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.ToArray()\
   at Octopus.Core.RelationalStorage.Caching.LatestAndCountsFetcher.UnionQueries(IEnumerable`1[] latestAndCountQueries) in ./source/Octopus.Core/RelationalStorage/Caching/LatestAndCountsFetcher.cs:line 196\
   at Octopus.Core.RelationalStorage.Caching.LatestAndCountsFetcher.GetLatestAndCounts[T](IRawReadQueryExecutor queryExecutor, SpacePartitionsDefinition partitions) in ./source/Octopus.Core/RelationalStorage/Caching/LatestAndCountsFetcher.cs:line 28\
   at Octopus.Core.RelationalStorage.Caching.NonCloningRawFullTableCache.Get[T](IRawReadQueryExecutor trn, SpacePartitionsDefinition partitions, Func`2 filter) in ./source/Octopus.Core/RelationalStorage/Caching/NonCloningRawFullTableCache.cs:line 35\
   at Octopus.Core.RelationalStorage.Caching.CloningRawFullTableCache.Get[T](IRawReadQueryExecutor trn, SpacePartitionsDefinition partitions, Func`2 filter) in ./source/Octopus.Core/RelationalStorage/Caching/CloningRawFullTableCache.cs:line 21\
   at Octopus.Server.Spaces.SpacesCache.LoadSpaces() in ./source/Octopus.Server/Spaces/SpacesCache.cs:line 57\
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)\
--- End of stack trace from previous location ---\
   at System.Lazy`1.CreateValue()\
   at Octopus.Server.Spaces.SpacesCache.get_DefaultSpace() in ./source/Octopus.Server/Spaces/SpacesCache.cs:line 31\
   at Octopus.Server.Web.Infrastructure.Spaces.LazyAllSpacePartitions.CreateSpacePartitions() in ./source/Octopus.Server/Web/Infrastructure/Spaces/LazyAllSpacePartitions.cs:line 54\
   at Octopus.Server.Web.Infrastructure.Spaces.LazyAllSpacePartitions.CreatePartitions() in ./source/Octopus.Server/Web/Infrastructure/Spaces/LazyAllSpacePartitions.cs:line 34\
   at Octopus.Server.Web.Infrastructure.Spaces.LazyAllSpacePartitions.SingleSpecificSpacePartition() in ./source/Octopus.Server/Web/Infrastructure/Spaces/LazyAllSpacePartitions.cs:line 30\
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)\
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)\
   at System.Lazy`1.CreateValue()\
   at Octopus.Server.Web.Routes.Api.SpaceScopedRoutes.Route(String template) in ./source/Octopus.Server/Web/Routes/Api/SpaceScopedRoutes.cs:line 16\
   at Octopus.Server.Web.Api.Actions.SpaceScopedHomeLinksProvider.get_SpaceLinks()+MoveNext() in ./source/Octopus.Server/Web/Api/Actions/SpaceScopedHomeLinksProvider.cs:line 103\
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)\
   at Octopus.Server.Web.Api.Actions.HomeResponder.<>c.<SpaceLinks>b__11_1(SpaceScopedHomeLinksProvider links) in ./source/Octopus.Server/Web/Api/Actions/HomeResponder.cs:line 86\
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do[T](IUnitOfWork unitOfWork, Func`1 action, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 23\
   at Octopus.Server.Web.Infrastructure.Spaces.SpaceContextSwitcher.PerformInSpaceContext[TDependency,TReturn](SpacePartitionsDefinition definition, Func`2 action) in ./source/Octopus.Server/Web/Infrastructure/Spaces/SpaceContextSwitcher.cs:line 42\
   at Octopus.Server.Web.Api.Actions.HomeResponder.ExecuteRegistered() in ./source/Octopus.Server/Web/Api/Actions/HomeResponder.cs:line 63\
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.ExecuteRegistered(CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Infrastructure/Api/CustomResponder.cs:line 344\
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException[TResult](Task`1 task)\
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.Respond(TDescriptor options, NancyContext context) in ./source/Octopus.Server/Web/Infrastructure/Api/CustomResponder.cs:line 292\
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.Respond(Object options, NancyContext context, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Infrastructure/Api/CustomResponder.cs:line 287\
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass14_0.<get_Routes>b__1(Object _, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Infrastructure/OctopusNancyModule.cs:line 158\
   at Nancy.Routing.Route`1.Invoke(DynamicDictionary parameters, CancellationToken cancellationToken)\
   at Nancy.Routing.DefaultRouteInvoker.Invoke(Route route, CancellationToken cancellationToken, DynamicDictionary parameters, NancyContext context)\
   at Nancy.Routing.DefaultRequestDispatcher.Dispatch(NancyContext context, CancellationToken cancellationToken)\
   at Nancy.NancyEngine.InvokeRequestLifeCycle(NancyContext context, CancellationToken cancellationToken, IPipelines pipelines)\
   at Octopus.NancyOwin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary`2 environment, CancellationToken cancellationToken) in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 174\
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 110\
--- End of stack trace from previous location ---\
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()\
--- End of stack trace from previous location ---\
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47\
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47\
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 53\
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 49\
   at Octopus.Server.Web.Middleware.PrivateSpaceToggleMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/PrivateSpaceToggleMiddleware.cs:line 56\
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 61\
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, IUserAuthenticator userAuthenticator, IAuthCookieService authCookieService, IWebAuthCache authCache, ILogger logger) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 57\
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\
   at Octopus.Server.Web.Middleware.LegacyRequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/LegacyRequestLoggerMiddleware.cs:line 42\
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 76\
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 98\
ClientConnectionId:00000000-0000-0000-0000-000000000000

More Information

.NET Runtime GitHub issue.

Workaround

Restart the Octopus server.

octoreleasebot commented 2 years ago

Release Note: Fixed an issue where a transient database connection issue could persist even after the transient issue has gone.

Octobob commented 1 year ago

:tada: The fix for this issue has been released in:

Release stream Release
2022.4 2022.4.5481
2023.1+ all releases