HangfireIO / Hangfire

An easy way to perform background job processing in .NET and .NET Core applications. No Windows Service or separate process required
https://www.hangfire.io
Other
9.32k stars 1.69k forks source link

This may have occurred because all pooled connections were in use and max pool size was reached. #1835

Open zhongqingshen opened 3 years ago

zhongqingshen commented 3 years ago

ERROR Message:

System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.  This may have occurred because all pooled connections were in use and max pool size was reached.
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()
   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)
   at Hangfire.SqlServer.SqlServerStorage.UseConnection(DbConnection dedicatedConnection, Action`1 action)
   at Hangfire.SqlServer.SqlServerConnection.Heartbeat(String serverId)
   at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)

I have modified the Hangfire source code. The database password will be changed from time to time, so the project needs to dynamically obtain the database password. change code as below:

        internal DbConnection CreateAndOpenConnection()
        {
            using (_options.ImpersonationFunc?.Invoke())
            {
                DbConnection connection = null;

                try
                {
                    connection = new SqlConnection(ConnectionString); //_existingConnection ?? _connectionFactory();

                    if (connection.State == ConnectionState.Closed)
                    {
                        connection.Open();
                    }

                    return connection;
                }
                catch
                {
                    ReleaseConnection(connection);
                    throw;
                }
            }
        }

When adding doWork to a project, this is done every 2 minutes. A connection pooling error occurs after 5 hours. Do not know where is not releasing the connection, please help me

caruizse commented 2 years ago

HI, I have the same, not solved yet but I found interesting information.

According to internet info looks like it's necessary to increase the sql pool connections, but I hope HangFire team can share us their point of view @odinserj .

If you are on a dedicated server and simply need more SQL connections, you may update the "max pool size" entries in both connection strings by following these instructions:
Log into your server using Remote Desktop.
Open My Computer (Windows - E) and go to C:\inetpub\vhosts\[domain]\httpdocs
Double click on the web.config file.  This may just be listed as web if the file structure is set to hide extensions. This will open up Visual Basic or similar editor.
Find your Connection Strings, these will look similar to the examples below:
<add name="SiteSqlServer" connectionString="server=(local);database=dbname;uid=dbuser;pwd=dbpassword;pooling=true;connection lifetime=120;max pool size=25;" />
<add key="SiteSqlServer" value="server=(local);database=dbname;uid=dbuser;pwd=dbpassword;pooling=true;connection lifetime=120;max pool size=25;" />
Change the max pool size=X value to the required pool size.
Save and close your web.config file.
kgrosvenor commented 2 years ago

Same issue, we have a long running system and after about 7 days of automated testing it happens. How come the pools continually grow and never cleaned up?

Jeff-Tian commented 1 month ago
.AddHangfireServer(options =>
            {
                options.WorkerCount = 1;
            });

Could setting the worker count to 1 fix the "all pooled connections in use" error?