search

OctopusDeploy / Issues

| Public | Bug reports and known issues for Octopus Deploy and all related tools
https://octopus.com
162 stars 20 forks source link

Modify VCS space used logic to account for file being removed while its getting calculated. #7186

Closed zentron closed 2 years ago

zentron commented 2 years ago

Team

Severity

No response

Version

2021.3.5962

Latest Version

No response

What happened?

Transient error where the git directory is being removed while something is trying to calculate the size. We already have a handler for the whole directory being deleted, but it seems as though if the file itself is deleted while its size is being calculated it can throw a different FileNotFound exception

In cases such as this we will just ignore counting that file size.

Worst case the directory is slightly larger than it would be if it was calculated again since its possible that files which were counted no longer exist. One could argue that due to the latent race condition (we want to avoid expensive locks for its current usage) it could have been sized up completely before anything was removed and hence could have been larger still. This is really only a problem if the timing issue occurs right when the git repo is at the limits of available space and so is a reasonable handling

Reproduction

Manually delete a git repository at the same time that another is being cloned/checked out. Fairly un-likey to occur in normal usage, discovered during tests where non-standard cleanup takes place.

Error and Stacktrace

ERROR  Unhandled error on request: POST http://localhost:36349/api/Spaces-48/projects/Projects-41/git/convert 0HMCJ9JOPJHFU:00000011 by 063110ba47864e2b8c8fa0ea89a11f81 : An exception was thrown while activating λ:Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> λ:Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.EntityTracking.EntityTrackingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Database.ProjectPathDecorators.ProjectPathDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Git.NameAsIdDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Git.GitDocumentStore`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> λ:Octopus.Core.Git.Transactions.IGitTransaction -> λ:Octopus.Core.Git.Transactions.IGitTransaction -> λ:Octopus.Core.Git.Transactions.IGitTransaction.
Autofac.Core.DependencyResolutionException: An exception was thrown while activating λ:Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> λ:Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Auditing.AuditingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.EntityTracking.EntityTrackingDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Database.ProjectPathDecorators.ProjectPathDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Git.NameAsIdDocumentStoreDecorator`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> Octopus.Core.Persistence.Git.GitDocumentStore`2[[Octopus.Core.Model.Projects.DeploymentProcess, Octopus.Core, Version=2021.3.6708.0, Culture=neutral, PublicKeyToken=null],[System.String, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]] -> λ:Octopus.Core.Git.Transactions.IGitTransaction -> λ:Octopus.Core.Git.Transactions.IGitTransaction -> λ:Octopus.Core.Git.Transactions.IGitTransaction.
 ---> System.IO.FileNotFoundException: Could not find file '/opt/buildagent/temp/buildTmp/Octopus.Tests.Common.FileSystem/StubHomeConfiguration/.ctor/67456a7f/Git/WVKRUQ2OIOP2SOPEU6HBEVQXCVV4UD22/bGVDEJYLEIHXKW44ANRN5W6PBN2XJU7BZ/.git/objects/pack/pack_git2_W7bwQC'.
File name: '/opt/buildagent/temp/buildTmp/Octopus.Tests.Common.FileSystem/StubHomeConfiguration/.ctor/67456a7f/Git/WVKRUQ2OIOP2SOPEU6HBEVQXCVV4UD22/bGVDEJYLEIHXKW44ANRN5W6PBN2XJU7BZ/.git/objects/pack/pack_git2_W7bwQC'
   at System.IO.FileInfo.get_Length()
   at Octopus.Core.Extensions.DirectoryInfoExtensionMethods.GetSpaceUsed(DirectoryInfo d) in ./source/Octopus.Core/Extensions/DirectoryInfoExtensionMethods.cs:line 44
   at Octopus.Core.Extensions.DirectoryInfoExtensionMethods.GetSpaceUsed(DirectoryInfo d) in ./source/Octopus.Core/Extensions/DirectoryInfoExtensionMethods.cs:line 58
   at Octopus.Core.Extensions.DirectoryInfoExtensionMethods.GetSpaceUsed(DirectoryInfo d) in ./source/Octopus.Core/Extensions/DirectoryInfoExtensionMethods.cs:line 58
   at Octopus.Core.Extensions.DirectoryInfoExtensionMethods.GetSpaceUsed(DirectoryInfo d) in ./source/Octopus.Core/Extensions/DirectoryInfoExtensionMethods.cs:line 58
   at Octopus.Core.Extensions.DirectoryInfoExtensionMethods.GetSpaceUsed(DirectoryInfo d) in ./source/Octopus.Core/Extensions/DirectoryInfoExtensionMethods.cs:line 58
   at Octopus.Core.Extensions.DirectoryInfoExtensionMethods.GetSpaceUsed(DirectoryInfo d) in ./source/Octopus.Core/Extensions/DirectoryInfoExtensionMethods.cs:line 58
   at Octopus.Core.Git.GitSpaceLimit.CurrentUsageBytes() in ./source/Octopus.Core/Git/GitSpaceLimit.cs:line 80
   at Octopus.Core.Git.GitSpaceLimit.DoOperationWithProgressHandler(Action`1 operation, Func`1 onUserCancelledOperation) in ./source/Octopus.Core/Git/GitSpaceLimit.cs:line 57
   at Octopus.Core.Git.Repositories.RepositoryRoot.<EnsureRepositoryIsCurrent>b__13_0() in ./source/Octopus.Core/Git/Repositories/RepositoryRoot.cs:line 61
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Octopus.Core.Git.Repositories.RepositoryRoot.EnsureRepositoryIsCurrent(CancellationToken cancellationToken) in ./source/Octopus.Core/Git/Repositories/RepositoryRoot.cs:line 51
   at Octopus.Core.Git.Repositories.ErrorHandlingRepositoryRootDecorator.<EnsureRepositoryIsCurrent>b__9_0(CancellationToken ct) in ./source/Octopus.Core/Git/Repositories/ErrorHandlingRepositoryRootDecorator.cs:line 34
   at Octopus.Core.Git.Repositories.ErrorHandlingRepositoryRootDecorator.HandleLibGit2SharpExceptions(Func`2 action, String defaultErrorMessage, CancellationToken cancellationToken, List`1 statusErrorList) in ./source/Octopus.Core/Git/Repositories/ErrorHandlingRepositoryRootDecorator.cs:line 73
   at Octopus.Core.Git.Repositories.ErrorHandlingRepositoryRootDecorator.EnsureRepositoryIsCurrent(CancellationToken cancellationToken) in ./source/Octopus.Core/Git/Repositories/ErrorHandlingRepositoryRootDecorator.cs:line 33
   at Octopus.Core.Git.RepositoryFileSystemLoggingDecorator.EnsureRepositoryIsCurrent(CancellationToken cancellationToken) in ./source/Octopus.Core/Git/RepositoryFileSystemLoggingDecorator.cs:line 46
   at Octopus.Core.Git.Repositories.GitRepository.ValidateFullyQualifiedGitRef(String gitResourceName, CancellationToken cancellationToken) in ./source/Octopus.Core/Git/Repositories/GitRepository.cs:line 342
   at Octopus.Core.Git.Repositories.GitRepository.PrepareWorkTree(GitRef gitRef, CancellationToken cancellationToken) in ./source/Octopus.Core/Git/Repositories/GitRepository.cs:line 149
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException[TResult](Task`1 task)
   at Octopus.Core.Persistence.Git.GitTransactionFactory.BeginOrAdoptExisting() in ./source/Octopus.Core/Persistence/Git/GitTransactionFactory.cs:line 49
   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.Features.Indexed.KeyedServiceIndex`2.get_Item(TKey key)
   at Octopus.Core.Persistence.DocumentStore`2.GetOrNullAsync(TKey id, CancellationToken cancellationToken) in ./source/Octopus.Core/Persistence/DocumentStore.cs:line 87
   at Octopus.Core.Features.DeploymentProcesses.CurrentDeploymentProcessDocumentStoreExtensionMethods.GetOrNullByProjectAsync(IDocumentStore`2 documentStore, Project project, CancellationToken cancellationToken) in ./source/Octopus.Core/Features/DeploymentProcesses/CurrentDeploymentProcessDocumentStoreExtensionMethods.cs:line 24
   at Octopus.Server.Web.Api.Actions.Projects.ProjectToVersionControlConverter.ConfigureEntities(Project project, VersionControlSettings versionControlSettings, MessageCollection messages, DeploymentProcess databaseDeploymentProcess, DeploymentSettings databaseDeploymentSettings, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Api/Actions/Projects/ProjectToVersionControlConverter.cs:line 122
   at Octopus.Server.Web.Api.Actions.Projects.ProjectToVersionControlConverter.Convert(Project project, VersionControlSettings versionControlSettings, IOctopusQueryExecutor queryExecutor, MessageCollection messages, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Api/Actions/Projects/ProjectToVersionControlConverter.cs:line 84
   at Octopus.Server.Web.Api.Actions.Projects.ConvertProjectToVersionControlledCommandHandler.Handle(ConvertProjectToVersionControlledCommand command, CancellationToken cancellationToken) in ./source/Octopus.Server/Web/Api/Actions/Projects/ConvertProjectToVersionControlledCommandHandler.cs:line 61
   at Octopus.Core.Infrastructure.Mediator.AutofacMediator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/AutofacMediator.cs:line 28
   at Octopus.Core.Infrastructure.Mediator.Decorators.SystemComponentModelValidationDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/SystemComponentModelValidationDecorator.cs:line 31
   at Octopus.Core.Infrastructure.Mediator.Decorators.FluentValidationsDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/FluentValidationsDecorator.cs:line 44
   at Octopus.Core.Infrastructure.Mediator.Decorators.MessageBusSiphoningDecorator.Do[TCommand,TResponse](ICommand`2 command, CancellationToken cancellationToken) in ./source/Octopus.Core/Infrastructure/Mediator/Decorators/MessageBusSiphoningDecorator.cs:line 28
   at Nito.AsyncEx.Synchronous.TaskExtensions.WaitAndUnwrapException[TResult](Task`1 task)
   at Octopus.Server.Web.Api.Actions.Projects.ConvertProjectToVersionControlledResponder.ExecuteRegistered() in ./source/Octopus.Server/Web/Api/Actions/Projects/ConvertProjectToVersionControlledResponder.cs:line 37
   at Octopus.Server.Web.Infrastructure.Api.CustomResponder`1.Respond(TDescriptor options, NancyContext context) in ./source/Octopus.Server/Web/Infrastructure/Api/CustomResponder.cs:line 289
   at Octopus.Server.Web.Infrastructure.OctopusNancyModule.<>c__DisplayClass13_0.<get_Routes>b__1(Object _, CancellationToken _) in ./source/Octopus.Server/Web/Infrastructure/OctopusNancyModule.cs:line 75
   at Nancy.Routing.Route`1.Invoke(DynamicDictionary parameters, CancellationToken cancellationToken)
   at Nancy.Routing.DefaultRouteInvoker.Invoke(Route route, CancellationToken cancellationToken, DynamicDictionary parameters, NancyContext context)
   at Nancy.Routing.DefaultRequestDispatcher.Dispatch(NancyContext context, CancellationToken cancellationToken)
   at Nancy.NancyEngine.InvokeRequestLifeCycle(NancyContext context, CancellationToken cancellationToken, IPipelines pipelines)
   at Octopus.NancyOwin.NancyMiddleware.RequestComplete(NancyContext context, IDictionary`2 environment, CancellationToken cancellationToken) in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 171
   at Octopus.NancyOwin.NancyMiddleware.<>c__DisplayClass3_0.<<UseNancy>b__1>d.MoveNext() in ./source/Octopus.Server/Owin/NancyMiddleware.cs:line 111
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Owin.WebSocketAcceptAdapter.<>c__DisplayClass6_0.<<AdaptWebSockets>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.HttpOverrides.HttpMethodOverrideMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.UnitOfWorkMiddleware.InvokeAsync(HttpContext httpContext, IUnitOfWork unitOfWork) in ./source/Octopus.Server/Web/UnitOfWorkMiddleware.cs:line 47
   at Octopus.Server.Web.Middleware.OctopusClientOldVersionWarningMiddleware.InvokeAsync(HttpContext context, IAutomationContext automationContext) in ./source/Octopus.Server/Web/Middleware/OctopusClientOldVersionWarningMiddleware.cs:line 38
   at Octopus.Server.Web.Middleware.DynamicContentHeadersMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/DynamicContentHeadersMiddleware.cs:line 48
   at Octopus.Server.Web.Middleware.MaintenanceModeMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/MaintenanceModeMiddleware.cs:line 55
   at Octopus.Server.Web.Middleware.OctopusAuthenticationMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/OctopusAuthenticationMiddleware.cs:line 67
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Octopus.Server.Web.Middleware.RequestLoggerMiddleware.InvokeAsync(HttpContext context) in ./source/Octopus.Server/Web/Middleware/RequestLoggerMiddleware.cs:line 42
   at Octopus.Server.Web.Middleware.TelemetryMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/TelemetryMiddleware.cs:line 75
   at Octopus.Server.Web.Middleware.ErrorHandlingMiddleware.InvokeAsync(HttpContext context, CorrelationId correlationId) in ./source/Octopus.Server/Web/Middleware/ErrorHandlingMiddleware.cs:line 103

More Information

No response

Workaround

No response

octoreleasebot commented 2 years ago

Release Note: Modify VCS space used logic to account for file being removed while its getting calculated

Octobob commented 2 years ago

:tada: The fix for this issue has been released in:

Release stream Release
2021.3 2021.3.7040
2022.1+ all releases