NullReferenceException when creating a LOT of connections

[avrahamcool]

I have a similar problem as described here: https://github.com/oracle/dotnet-db-samples/issues/278 but with a different setup - enough to open a new issue.

I'm running a WebAPI project .net framework 4.8 Oracle.EntityFrameworkCore 3.19.180 Oracle.ManagedDataAccess.Core 2.19.210

I'm using a proxy user in order to connect to the DB, where each request can be made by a different user. I have a basic connection string, and in each request - I only swap the USER ID for the current request. I've noticed its using the same pool for all requests as expected.

lets consider the following connection string: (notice the placeholder for the USER_ID): MAX POOL SIZE=20;LOAD BALANCING=True;DECR POOL SIZE=100;INCR POOL SIZE=1;DATA SOURCE=MY_DB;PROXY PASSWORD=BLABLA123456;PROXY USER ID=MY_PROXY_USER;USER ID={0};

I've created a simple app to stress-test this setup. I have a single POST action that let you choose the user to be used in the connection. the action creates a DBContext, starts a transaction [THIS IS AN MPORTANT PART FOR THE BUG] inserts an object to the DB (with save changes) waits for 500ms [to imitate "long" requests.] commit the transaction

I also have a simple client that sends 100 request using USER_A, and 100 requests using USER_B - all in parallel. the server caps at 20 connections, as expected, but most of the requests works fine. because we are able to satisfy the request within the 15 seconds default timeout of the connection pool.

some of the requests fails with the following error:

ExceptionMessage: "Object reference not set to an instance of an object." ExceptionType: "System.NullReferenceException" StackTrace:

   at OracleInternal.ConnectionPool.PoolManager`3.ProcessCriteriaCtxAndAlterSessionIfReqd(OracleConnection connRefForCriteria, PR pr)
   at OracleInternal.ConnectionPool.OraclePoolManager.Get(ConnectionString csWithNewPassword, Boolean bGetForApp, OracleConnection connRefForCriteria, String affinityInstanceName, Boolean bForceMatch)
   at OracleInternal.ConnectionPool.OracleConnectionDispenser`3.Get(ConnectionString cs, PM conPM, ConnectionString pmCS, SecureString securedPassword, SecureString securedProxyPassword, OracleConnection connRefForCriteria)
   at Oracle.ManagedDataAccess.Client.OracleConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransaction(IsolationLevel isolationLevel)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.BeginTransaction()
   at Microsoft.EntityFrameworkCore.Infrastructure.DatabaseFacade.BeginTransaction()
   at PoolWithProxyApi.Controllers.TestController.Post(String username) in <REDACTED>\\TestController.cs:line 22
   at lambda_method(Closure , Object , Object[] )
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments)
   at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()

If I don't use a transaction, it's seems to work fine with the 200 parallel request described earlier. (I know hat for this simple use-case I don't even need a transaction - but in our real app we do need a transaction context - and I was able to reproduce the problem with this minimal setup]

here is the server code that i'm using: [I can upload the full test project, including the client to Github if necessary]

    public class TestController : ApiController
    {
        public string Post(string username)
        {
            string connectionString = string.Format(ConfigurationManager.AppSettings["ProxyConnectionString"], username);

            DbContextOptionsBuilder<GenericContext> optionBuilder = new DbContextOptionsBuilder<GenericContext>()
                .UseOracle(connectionString);

            using GenericContext genericContext = new(optionBuilder.Options);

            genericContext.Database.BeginTransaction();

            genericContext.Dummy.Add(new DummyModel()
            {
                FirstName = $"First {username}",
                LastName = $"Last {username}"
            });

            genericContext.SaveChanges();

            Thread.Sleep(500);

            genericContext.Database.CommitTransaction();

            return "Done";
        }
    }

[alexkeh]

If you can publish a complete test case, Oracle can try to replicate your problem in house. It's hard to debug problems found through load testing without either a test case or artifacts from testing that provide enough data to identify the root cause.

[avrahamcool]

Hi @alexkeh,

I've publish the project under: https://github.com/avrahamcool/PoolWithProxyApi I made some small modification - so it's clearer.

in order to replicate the problem:

• fill the missing part in the connection string appropriately
• create the demo table [script embeded in the index.html]
• fix the db context schema name according to your environment

you need to run the client from 2 simultaneous browsers. in each client - use a different user name with 100 concurrent connections. the problem should appear after some runs. (sometime all the 100 connections works - in that case just try again)

[alexkeh]

Thanks, @avrahamcool. We'll try out your test case. In the meantime, I've created bug 35999664 to track this issue.

[alexkeh]

ODP.NET 21.13 is now available on NuGet Gallery.

[avrahamcool]

@alexkeh Thank you very much with all the help.