Open oshinyil opened 3 years ago
cincuranet
commented
3 years ago A call stack is not enough to understand this issue. Steps to reproduce would be best. But providing as much information as possible what was happening before and what is the environment is a good start.
NicFT
commented
2 years ago Hello, this error is happening for me too, it's been a few months, on several clients The Error occurs sporadically, as well as resolves itself after some time, This error stops occurring when the connection pool time is reached, after that it no longer occurs, Strange that the error seems to occur in the first hours of system use, afterward the error is no longer displayed.
oshinyil
commented
2 years ago Here is the environment.
Environment.Version: 4.0.30319.42000RuntimeInformation.FrameworkDescription: 4.7.3740.0RuntimeInformation.OSDescription: Microsoft Windows 10.0.17763I am not sure what was happening before. The error suddenly happened and we needed to restart the application.
oshinyil
commented
2 years ago I also reported different issues that might be related to this one. They happened at the same time, but with different exceptions. https://github.com/FirebirdSQL/NETProvider/issues/964 https://github.com/FirebirdSQL/NETProvider/issues/965
cincuranet
commented
2 years ago Sadly, without further information there's not much I can do to find what the issue is and work on correcting it. Hopefully somebody can bring some clues into the discussion.
NicFT
commented
2 years ago I'm using version 8.0.1.0 Maybe @cincuranet you can give me a light on how I can simulate this error? Because the error is happening now practically every day, but there is no pattern
StackTrace
at FirebirdSql.Data.Client.Managed.GdsConnection.<ProcessOperation>d__49.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.<ReadSingleResponse>d__52.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.<ReadSingleResponse>d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.<ReadResponse>d__49.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.Client.Managed.Version10.GdsTransaction.<Commit>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.FirebirdClient.FbTransaction.<CommitImpl>d__22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.FirebirdClient.FbCommand.<CommitImplicitTransaction>d__107.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at FirebirdSql.Data.FirebirdClient.FbCommand.<CommitImplicitTransaction>d__107.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.FirebirdClient.FbDataReader.<CloseImpl>d__33.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FirebirdSql.Data.FirebirdClient.FbDataReader.Close() at FirebirdSql.Data.FirebirdClient.FbDataReader.Dispose(Boolean disposing) at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior) at System.Data.Common.DbDataAdapter.Fill(DataTable[] dataTables, Int32 startRecord, Int32 maxRecords, IDbCommand command, CommandBehavior behavior) at System.Data.Common.DbDataAdapter.Fill(DataTable dataTable) at DtConsultaFechado(DbCommand comandoSQL) in
cincuranet
commented
2 years ago First try latest version, so we have up to date code.
NicFT
commented
2 years ago I did the update, and apparently it had some effect, it's been 2 hours since the error doesn't occur, but I'll wait another day to definitely confirm
NicFT
commented
2 years ago the problem persists, but with less intensity, since the update to version 8.5.2.0 the problem only happened once and during a period of heavy use,
djunges
commented
2 years ago I have the same problem, im using version 8.5.3.0.
any one has found a solution for it ?
regards Daniel
NicFT
commented
2 years ago Hello, Yes I managed to get around, Basically what I did was put a good part of the queries that are executed in REDIS Cache to avoid a query in Firebird, momentarily resolved
@cincuranet the only way to simulate the problem was the following, I created a web page that makes a query to the database, opened the SOAP UI and had an access test run several times 5k, in this case the error is presented in a way sporadic,
Another detail, decreasing the PACKET SIZE of the connection string makes the error not occur very often, as increasing it makes it appear faster
cincuranet
commented
2 years ago @NicFT Can you share that project?
NicFT
commented
2 years ago @cincuranet , when trying to create a separate project that simulates exactly what my software does, the problem is not repeated, indicating that there is some other factor that is impacting this error, I have not yet been able to identify which one, what I can only be sure is that it increases or decreasing the PacketSize directly impacts the more frequent presence of the error
cincuranet
commented
2 years ago As long as it happens at least once every 10 minutes, it's good-ish for me to look at it. Not a best case for hunting for bug, but better than nothing.
cincuranet
commented
2 years ago Just to let you all know, I'm currently working on some tests and hitting similar behavior and hitting it reliably(!). Still not sure what's what, but at least I can replicate it. Fingers crossed.
NicFT
commented
2 years ago Sorry @cincuranet I didn't have enough time to set up the example and dismember it from my system, end of year is complicated
cincuranet
commented
2 years ago The lead I was chasing eventually turned out to be bug in EF Core (https://github.com/dotnet/efcore/issues/26790). So either this is related or still some dangling issue in the provider.
Thus, if somebody has some decent way to simulate it, I'm all ears.
NicFT
commented
1 year ago OK I found a way to simulate the error, although it is in a forced situation I managed to catch the error when checking a programmer error, In this example sent by commenting on any of the parts below, the error stops being displayed,
In a production environment, although I don't use TASK.RUN, the same error is being displayed sporadically in the logs, the error starts to occur at times of peak use of the system
The example was created using blazor just run the app and click on the button the error will occur, sometimes the error will not be displayed on the first click another generic error is displayed from transaction, but clicking the error again (Error: System.ArgumentOutOfRangeException: operation =0 (Parameter 'operation')) will be displayed If you comment the task.run part, the error stops being displayed, as well as if you comment the part below the task run, the error is no longer displayed, it only occurs when the two are together, that is, having a task.run and just below another code
Below sending the files for simulation sorry for being late
NicFT
commented
1 year ago @cincuranet Please let me know if you were able to simulate with the project sent, thank you very much
FabricioAraujoRJ
commented
1 year ago OK I found a way to simulate the error, although it is in a forced situation I managed to catch the error when checking a programmer error, In this example sent by commenting on any of the parts below, the error stops being displayed,
In a production environment, although I don't use TASK.RUN, the same error is being displayed sporadically in the logs, the error starts to occur at times of peak use of the system
The example was created using blazor just run the app and click on the button the error will occur, sometimes the error will not be displayed on the first click another generic error is displayed from transaction, but clicking the error again (Error: System.ArgumentOutOfRangeException: operation =0 (Parameter 'operation')) will be displayed If you comment the task.run part, the error stops being displayed, as well as if you comment the part below the task run, the error is no longer displayed, it only occurs when the two are together, that is, having a task.run and just below another code
Below sending the files for simulation sorry for being late
The code of NicFT is very similar with mine. You put the code in Task.Run, the exception explodes. Remove the thread/async stuff, it updates the 8k rows in a flash without problems. My problem is with Dapper, however.
realic
commented
7 months ago Hey @cincuranet I have been having this ArgumentOutOfRangeException: operation=xxxx error for some time in our production environment, and I found this issue, and had a look at the "EmptyBlazorApp1" code to reproduce it, using the 10.0.0 version of the provider.
The sample code generates a lot of exceptions due to the fact that the same FbConnection is shared between two threads and one tread is opening/executing/closing while another is doing the same. This is due to the two Task.Run lines. As FbConnection and related classes is not threadsafe i regard this as expected behavior.
However after the two Task.Run statements there is the command2 variable where method RunCommand and RunCommand2 are called. I find it a bit worrying that calling those methods throws ArgumentOutOfRangeException: operation=xxxx exceptions. From what i can figure out, a "bad" FbConnectionInternal objects that somehow gets corrupted, is returned to the connection pool during or after the Task.Run methods, and it is returned when methods on command2 gets called.
Could this be a racecondition somewhere or is the client <> server protocol getting misaligned? This is is just pure speculation on my part.
Best regards Thomas
cincuranet
commented
7 months ago Yes, some incorrect use of threading and ADO.NET is my suspicion (also not something that's a bug in provider).
NicFT
commented
7 months ago A complement to my example sent, changing the connection string and informing "Pooling=false" the problem is no longer presented
lipchev
commented
6 months ago Hello guys,
Here's how I was able to consistently reproduce the issue (it's either the operation exception, an invalid BLOB handle or a deadlock..):
I've created a simple benchmark in order to test whether or not the Async methods on the provider are actually asynchronous (and whether that actually provides a benefit to my queries). After doing a test run with a simple query (selecting from a small table with just a few columns) I went on with the select * from MyLargeTable (..).
It's important to note that MyLargeTable contains several blob columns (long text), and that the issue does not occur unless I select one or more of them (it takes two when running on localhost, but just one if testing over the network).
I'm not going to bore you with the "Sync"/"Async" versions (that work), here is the relevant part from the one called "Parallel" (that fails):
public static async Task<int> ReadAllDataInParallel(DbDataReader reader)
{
var tasks = new List<Task<int>>();
for (var i = 0; i < reader.FieldCount; i++)
{
tasks.Add(ReadFieldAsync(reader, i));
}
var results = await Task.WhenAll(tasks);
return results.Sum();
}
private static async Task<int> ReadFieldAsync(DbDataReader reader, int i)
{
if (await reader.IsDBNullAsync(i)) return 0;
return (await reader.GetFieldValueAsync<object>(i)).ToString().Length;
}
These tests were run using
FirebirdSql.Data.FirebirdClient v10.0.0
against a server running
Firebird 4.0.4.3010
but I know I've seen this exception in the logs going back several years ago, precisely having to do with the asynchronous loading of those damn blobs..
"Pooling=false" did not work for me, however I know that the issue does not happen with the ODBC provider (which, as far as the benchmarks show, provides no Async benefit- even with the blobs).
Here are the exceptions I've gathered:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.ArgumentOutOfRangeException: operation=47 (Parameter 'operation')
at FirebirdSql.Data.Client.Managed.GdsConnection.ProcessOperation(Int32 operation)
at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadSingleResponse(Int32 operation)
at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadSingleResponse()
at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadResponse()
at FirebirdSql.Data.Client.Managed.Version10.GdsTransaction.Rollback()
at FirebirdSql.Data.Client.Managed.Version10.GdsTransaction.Dispose2()
at FirebirdSql.Data.FirebirdClient.FbTransaction.Dispose(Boolean disposing)
at FirebirdSql.Data.FirebirdClient.FbCommand.RollbackImplicitTransaction()
at FirebirdSql.Data.FirebirdClient.FbCommand.CommitImplicitTransaction()
at FirebirdSql.Data.FirebirdClient.FbDataReader.Close()
at FirebirdSql.Data.FirebirdClient.FbDataReader.Dispose(Boolean disposing)System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> FirebirdSql.Data.FirebirdClient.FbException (0x80004005): invalid BLOB handle
---> FirebirdSql.Data.Common.IscException: invalid BLOB handle
at FirebirdSql.Data.Client.Managed.IResponseExtensions.HandleResponseException(IResponse response)
at FirebirdSql.Data.Client.Managed.Version10.GdsDatabase.ReadResponse()
at FirebirdSql.Data.Client.Managed.Version10.GdsTransaction.Commit()
at FirebirdSql.Data.FirebirdClient.FbTransaction.Commit()
at FirebirdSql.Data.FirebirdClient.FbTransaction.Commit()
at FirebirdSql.Data.FirebirdClient.FbCommand.CommitImplicitTransaction()
at FirebirdSql.Data.FirebirdClient.FbDataReader.Close()
at FirebirdSql.Data.FirebirdClient.FbDataReader.Dispose(Boolean disposing)This is invalid case. You're mixing asynchronicity and parallelism.
lipchev
commented
6 months ago I wouldn't say it's an invalid use case- just not supported (note that I couldn't find any docs regarding the behavior of the Async methods). Given how much slower it is to read a single blob it's perhaps not surprising that people would resolve to things like Task.Run / WaitAll (a 300-400 characters long field takes ~50 times more than reading all the rest of the columns combined) - I assumed that another "streaming" task was being launched behind the scenes.. Please note that while this was an explicitly parallel query, there are likely more typical ORM usages that also result in such scenarios, and the output found in the logs does not immediately make it obvious what the issue is (I used to think that those were network related)- so unless you can protect against it, might be we worth throwing in a little .md somewhere about the dangers of accessing a reader in parallel..
Hi,
I got issue when using
FirebirdSql.EntityFrameworkCore.Firebirdversion7.10.1. This issue does not always happen and I don't know how the exact step reproduce it. This happened few times in production.Here is the translated query.
And here is the LINQ
Please help to fix this issue.
Thank you.