gottscj / Hangfire.Mongo

Mongo DB support for Hangfire
MIT License
262 stars 84 forks source link

Error connection database #398

Closed kadirguloglu closed 5 months ago

kadirguloglu commented 5 months ago

Hello,

I keep seeing this life in my logs and my service keeps restarting. It's a project running on Kubernetes. My packages are currently on the latest version.

xxxx-deployment-df8c94d59-8scfw 1/1 Running 63 (3d12h ago) 3d17h `Hangfire.Mongo.MongoConnectException: Did not receive ping response from 'xxxx' within 5000ms assuming not able to connect to 'mongodb://:@xxxxx' you can disable database ping via the MongoStorageOption 'CheckConnection' field

---> System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.CancellationToken.ThrowIfCancellationRequested() at MongoDB.Driver.Core.Connections.BinaryConnection.ThrowIfCancelledOrDisposed(CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.ThrowIfCancelledOrDisposedOrNotOpen(CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.SendMessages(IEnumerable1 messages, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.ConnectionExtensions.SendMessage(IConnection connection, RequestMessage message, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken) at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol1.Execute(IConnection connection, CancellationToken cancellationToken) at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol1.Execute(IConnection connection, CancellationToken cancellationToken) at MongoDB.Driver.Core.Authentication.SaslAuthenticator.Authenticate(IConnection connection, ConnectionDescription description, CancellationToken cancellationToken) at MongoDB.Driver.Core.Authentication.DefaultAuthenticator.Authenticate(IConnection connection, ConnectionDescription description, CancellationToken cancellationToken) at MongoDB.Driver.Core.Authentication.AuthenticationHelper.Authenticate(IConnection connection, ConnectionDescription description, IReadOnlyList1 authenticators, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.ConnectionInitializer.Authenticate(IConnection connection, ConnectionInitializerContext connectionInitializerContext, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken) at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.PooledConnection.Open(CancellationToken cancellationToken) at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedInternal(CancellationToken cancellationToken) at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedOrReuse(CancellationToken cancellationToken) at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.AcquireConnection(CancellationToken cancellationToken) at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnection(CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.Server.GetChannel(CancellationToken cancellationToken) at MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken cancellationToken) at MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.RetryableReadContext.Initialize(CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.RetryableReadContext.Create(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.ReadCommandOperation1.Execute(IReadBinding binding, CancellationToken cancellationToken) at MongoDB.Driver.OperationExecutor.ExecuteReadOperation[TResult](IReadBinding binding, IReadOperation1 operation, CancellationToken cancellationToken) at MongoDB.Driver.MongoDatabaseImpl.ExecuteReadOperation[T](IClientSessionHandle session, IReadOperation1 operation, ReadPreference readPreference, CancellationToken cancellationToken) at MongoDB.Driver.MongoDatabaseImpl.RunCommand[TResult](IClientSessionHandle session, Command1 command, ReadPreference readPreference, CancellationToken cancellationToken) at MongoDB.Driver.MongoDatabaseImpl.<>cDisplayClass50_01.<RunCommand>b__0(IClientSessionHandle session) at MongoDB.Driver.MongoDatabaseImpl.UsingImplicitSession[TResult](Func2 func, CancellationToken cancellationToken) at MongoDB.Driver.MongoDatabaseImpl.RunCommand[TResult](Command`1 command, ReadPreference readPreference, CancellationToken cancellationToken) at Hangfire.Mongo.MongoStorage.CheckConnection() --- End of inner exception stack trace --- at Hangfire.Mongo.MongoStorage.CheckConnection() at Hangfire.Mongo.MongoStorage..ctor(IMongoClient mongoClient, String databaseName, MongoStorageOptions storageOptions) at Hangfire.Mongo.MongoBootstrapperConfigurationExtensions.UseMongoStorage(IGlobalConfiguration configuration, IMongoClient mongoClient, String databaseName, MongoStorageOptions storageOptions) at InitHangFire.<>c__DisplayClass0_0.b0(IGlobalConfiguration configuration) in /src/xxx/xxx/xxxx.cs:line 15 at Hangfire.HangfireServiceCollectionExtensions.<>cDisplayClass0_0.b0(IServiceProvider provider, IGlobalConfiguration config) in C:\projects\hangfire-525\src\Hangfire.NetCore\HangfireServiceCollectionExtensions.cs:line 40 at Hangfire.HangfireServiceCollectionExtensions.<>cDisplayClass1_0.b14(IServiceProvider serviceProvider) in C:\projects\hangfire-525\src\Hangfire.NetCore\HangfireServiceCollectionExtensions.cs:line 103 at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(ServiceIdentifier serviceIdentifier) at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider) at Hangfire.HangfireServiceCollectionExtensions.ThrowIfNotConfigured(IServiceProvider serviceProvider) in C:\projects\hangfire-525\src\Hangfire.NetCore\HangfireServiceCollectionExtensions.cs:line 307 at Hangfire.HangfireServiceCollectionExtensions.CreateBackgroundJobServerHostedService(IServiceProvider provider, JobStorage storage, IEnumerable1 additionalProcesses, BackgroundJobServerOptions options) in C:\projects\hangfire-525\src\Hangfire.NetCore\HangfireServiceCollectionExtensions.cs:line 241 at Hangfire.HangfireServiceCollectionExtensions.<>c__DisplayClass10_0.<AddHangfireServerInner>b__0(IServiceProvider provider) in C:\projects\hangfire-525\src\Hangfire.NetCore\HangfireServiceCollectionExtensions.cs:line 211 at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass2_0.<RealizeService>b__0(ServiceProviderEngineScope scope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(ServiceIdentifier serviceIdentifier, ServiceProviderEngineScope serviceProviderEngineScope) at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType) at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider) at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at xxxx.Api.Program.Main(String[] args) in /xxx/xxx.cs:line 27

gottscj commented 5 months ago

@kadirguloglu,

It's most likely due to your setup. I don't think this is a Mongo.Hangfire issue. Can you reproduce outside kubernetes?

gottscj commented 5 months ago

closing