search

spaghettidba / WorkloadTools

A collection of tools to collect, analyze and replay SQL Server workloads, on premises and in the cloud
MIT License
232 stars 52 forks source link

ConvertWorkload cannot convert trace files. #96

Closed john-sterrett closed 1 year ago

john-sterrett commented 3 years ago

I am trying to convert an existing trace including 453 trace files each 512MB and it fails. I also don't understand why the tool requires localDB if SQL Server Database Engine is already installed.

The following is the command I issue. LocalDB is installed. ConvertWorkload.exe -I"D:\SQL\Backup\K\replay\sql_replay.trc" -O"D:\DReplay\WorkloadTools" -L"D:\DReplay\WorkloadTools\output"

The following is my output in command prompt. Nothing is logged with the log parameter either. Info - ConvertWorkload.Program : ConvertWorkload, Version=1.5.8.0, Culture=neutral, PublicKeyToken=null 1.5.8 Info - ConvertWorkload.Program : Checking LocalDB... Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Writing event data to D:\DReplay\WorkloadTools Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryCloseInternal(Boolean closeReader) at System.Data.SqlClient.SqlDataReader.Close() at System.Data.Common.DbDataReader.Dispose(Boolean disposing) at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTrace EventDataReader.cs:line 338 at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 64 Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : The wait operation timed out Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file Info - ConvertWorkload.Program : Converter stopped.

john-sterrett commented 3 years ago

PS C:\Program Files\WorkloadTools> .\ConvertWorkload.exe -I"D:\SQL\Backup\K\replay\sql_replay.trc" -O"D:\DReplay\WorkloadTools\SQLworkload.sqlite" -L"D:\DReplay\WorkloadTools\output.txt" Info - ConvertWorkload.Program : ConvertWorkload, Version=1.5.8.0, Culture=neutral, PublicKeyToken=null 1.5.8 Info - ConvertWorkload.Program : Checking LocalDB... Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Writing event data to D:\DReplay\WorkloadTools\SQLworkload.sqlite Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryCloseInternal(Boolean closeReader) at System.Data.SqlClient.SqlDataReader.Close() at System.Data.Common.DbDataReader.Dispose(Boolean disposing) at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTrace EventDataReader.cs:line 338 at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 64 Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : The wait operation timed out Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 1 events saved Info - ConvertWorkload.Program : Converter stopped.

spaghettidba commented 3 years ago

Hi John,

I had a look at the code and I see what is probably happening. The converter relies on localdb because it needs to be able to read the trace files from the local computer where ConvertWorkload is running and needs to impersonate the permissions of the user running the tool. This can only achieved by running sql server in process with the application, which is what localdb does.

That said, the input file needs to have some columns in order to be converted successfully:

Without some of these columns, the conversion may or may not be successful, but I can tell you for sure that without the EventSequence column in particular, there is no way the conversion will work. If you have a small sample of the trace file you're trying to convert, I could have a go at adapting the code to work even without some of the columns. Do you thing it would be possible for you to share it? In case there's sensitive data in the traces, I'm also ok with an obfuscated version (load trace file in profiler, save to trace table, update all commands to "SELECT 1" or something else, open table in profiler and save to trace file again).

john-sterrett commented 3 years ago

Gianluca,

Thank you for taking a look at this. I verified that all columns are included except duration, Read, Writes, CPU, and Duration. StartTime and EndTime are there so duration could be calculated from the difference.

With the error below it seems like there is a timeout reading the trace files which might make sense. There are 226 trace files and they are 512MB each so if it's trying to read the whole thing a timeout would be expected. I also wonder if the space required to handle this data might be too big for localdb?

Its been over a decade since I was a C# developer but I was hoping to spend a few cycles over the weekend trying to look and run the code in VS so I could do some live debugging to see where and why it's stopping.

Regards, John

On Thu, Oct 1, 2020 at 11:32 AM Gianluca Sartori notifications@github.com wrote:

Hi John,

I had a look at the code and I see what is probably happening. The converter relies on localdb because it needs to be able to read the trace files from the local computer where ConvertWorkload is running and needs to impersonate the permissions of the user running the tool. This can only achieved by running sql server in process with the application, which is what localdb does.

That said, the input file needs to have some columns in order to be converted successfully:

  • EventSequence
  • Error
  • TextData
  • BinaryData
  • DatabaseID
  • HostName
  • ApplicationName
  • LoginName
  • SPID
  • Duration
  • StartTime
  • EndTime
  • Reads
  • Writes
  • CPU
  • EventClass
  • DatabaseName

Without some of these columns, the conversion may or may not be successful, but I can tell you for sure that without the EventSequence column in particular, there is no way the conversion will work. If you have a small sample of the trace file you're trying to convert, I could have a go at adapting the code to work even without some of the columns. Do you thing it would be possible for you to share it? In case there's sensitive data in the traces, I'm also ok with an obfuscated version (load trace file in profiler, save to trace table, update all commands to "SELECT 1" or something else, open table in profiler and save to trace file again).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spaghettidba/WorkloadTools/issues/96#issuecomment-702254114, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLVWYV6W6L52UKVPCVRMSDSISVJRANCNFSM4R4Y3UBQ .

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

john-sterrett commented 3 years ago

Sorry forgot to put the command line and error in the previous email.

.\ConvertWorkload.exe -I"D:\SQL\Backup\K\replay\sql_replay.trc" -O"D:\DReplay\WorkloadTools\SQLworkload.sqlite" -L"D:\DReplay\WorkloadTools\output.txt"

Info - ConvertWorkload.Program : ConvertWorkload, Version=1.5.8.0, Culture=neutral, PublicKeyToken=null 1.5.8

Info - ConvertWorkload.Program : Checking LocalDB...

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Writing event data to D:\DReplay\WorkloadTools\SQLworkload.sqlite

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj,

Boolean& dataReady)

at System.Data.SqlClient.SqlDataReader.TryCloseInternal(Boolean closeReader)

at System.Data.SqlClient.SqlDataReader.Close()

at System.Data.Common.DbDataReader.Dispose(Boolean disposing)

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTrace

EventDataReader.cs:line 338

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 64

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : The wait operation timed out

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 1 events saved

Info - ConvertWorkload.Program : Converter stopped.

On Thu, Oct 1, 2020 at 1:06 PM John Sterrett jsterrett@gmail.com wrote:

Gianluca,

Thank you for taking a look at this. I verified that all columns are included except duration, Read, Writes, CPU, and Duration. StartTime and EndTime are there so duration could be calculated from the difference.

With the error below it seems like there is a timeout reading the trace files which might make sense. There are 226 trace files and they are 512MB each so if it's trying to read the whole thing a timeout would be expected. I also wonder if the space required to handle this data might be too big for localdb?

Its been over a decade since I was a C# developer but I was hoping to spend a few cycles over the weekend trying to look and run the code in VS so I could do some live debugging to see where and why it's stopping.

Regards, John

On Thu, Oct 1, 2020 at 11:32 AM Gianluca Sartori notifications@github.com wrote:

Hi John,

I had a look at the code and I see what is probably happening. The converter relies on localdb because it needs to be able to read the trace files from the local computer where ConvertWorkload is running and needs to impersonate the permissions of the user running the tool. This can only achieved by running sql server in process with the application, which is what localdb does.

That said, the input file needs to have some columns in order to be converted successfully:

  • EventSequence
  • Error
  • TextData
  • BinaryData
  • DatabaseID
  • HostName
  • ApplicationName
  • LoginName
  • SPID
  • Duration
  • StartTime
  • EndTime
  • Reads
  • Writes
  • CPU
  • EventClass
  • DatabaseName

Without some of these columns, the conversion may or may not be successful, but I can tell you for sure that without the EventSequence column in particular, there is no way the conversion will work. If you have a small sample of the trace file you're trying to convert, I could have a go at adapting the code to work even without some of the columns. Do you thing it would be possible for you to share it? In case there's sensitive data in the traces, I'm also ok with an obfuscated version (load trace file in profiler, save to trace table, update all commands to "SELECT 1" or something else, open table in profiler and save to trace file again).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spaghettidba/WorkloadTools/issues/96#issuecomment-702254114, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLVWYV6W6L52UKVPCVRMSDSISVJRANCNFSM4R4Y3UBQ .

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

spaghettidba commented 3 years ago

Hey John, Thanks for the info. I think the best option would be to try to load fewer files (like 10 or so) and see whether that makes a difference. In case it doesn't, you could send me a sample of your file and I could try to debug it. The logic for reading the trace files is one of the most complicated parts of the application and the chances for a bug are high.

john-sterrett commented 3 years ago

I copied the first two trace files and placed them in their own folder and tried to convert them again and got this error.

Info - ConvertWorkload.Program : ConvertWorkload, Version=1.5.8.0, Culture=neutral, PublicKeyToken=null 1.5.8

Info - ConvertWorkload.Program : Checking LocalDB...

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Writing event data to D:\DReplay\WorkloadTools\SQLworkload.sqlite

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Object reference not set to an instance of an object.

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at WorkloadTools.Listener.SqlTransformer.Skip(String command) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\SqlTransform

er.cs:line 54

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTrace

EventDataReader.cs:line 338

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 64

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 1 events saved

Info - ConvertWorkload.Program : Converter stopped.

On Fri, Oct 2, 2020 at 5:35 AM Gianluca Sartori notifications@github.com wrote:

Hey John, Thanks for the info. I think the best option would be to try to load fewer files (like 10 or so) and see whether that makes a difference. In case it doesn't, you could send me a sample of your file and I could try to debug it. The logic for reading the trace files is one of the most complicated parts of the application and the chances for a bug are high.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spaghettidba/WorkloadTools/issues/96#issuecomment-702656894, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLVWYR6VHGW45VBDV4UTR3SIWUG5ANCNFSM4R4Y3UBQ .

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

john-sterrett commented 3 years ago

Hi Gianluca,

I looked at the SQL Lite file as the end says 1 events saved. I do see 345 rows and all columns are populated except duration, CPU, reads, writes.

I am unsure why this is stopping. Thought this might be helpful.

On Fri, Oct 2, 2020 at 12:01 PM John Sterrett jsterrett@gmail.com wrote:

I copied the first two trace files and placed them in their own folder and tried to convert them again and got this error.

Info - ConvertWorkload.Program : ConvertWorkload, Version=1.5.8.0, Culture=neutral, PublicKeyToken=null 1.5.8

Info - ConvertWorkload.Program : Checking LocalDB...

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Writing event data to D:\DReplay\WorkloadTools\SQLworkload.sqlite

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Object reference not set to an instance of an object.

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at WorkloadTools.Listener.SqlTransformer.Skip(String command) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\SqlTransform

er.cs:line 54

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTrace

EventDataReader.cs:line 338

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 64

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 1 events saved

Info - ConvertWorkload.Program : Converter stopped.

On Fri, Oct 2, 2020 at 5:35 AM Gianluca Sartori notifications@github.com wrote:

Hey John, Thanks for the info. I think the best option would be to try to load fewer files (like 10 or so) and see whether that makes a difference. In case it doesn't, you could send me a sample of your file and I could try to debug it. The logic for reading the trace files is one of the most complicated parts of the application and the chances for a bug are high.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spaghettidba/WorkloadTools/issues/96#issuecomment-702656894, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLVWYR6VHGW45VBDV4UTR3SIWUG5ANCNFSM4R4Y3UBQ .

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

john-sterrett commented 3 years ago

Gianluca,

I think I got the code fixed. The skip method never checks to see if the string passed in is null. I made that the first check and to return false so it skips all the other checks. The small two trace files loaded with no problem. I am now working on the full 226 file trace. Once that is done if it works successfully I will check-in my changes to the project.

Regards, John

On Fri, Oct 2, 2020 at 12:55 PM John Sterrett jsterrett@gmail.com wrote:

Hi Gianluca,

I looked at the SQL Lite file as the end says 1 events saved. I do see 345 rows and all columns are populated except duration, CPU, reads, writes.

I am unsure why this is stopping. Thought this might be helpful.

On Fri, Oct 2, 2020 at 12:01 PM John Sterrett jsterrett@gmail.com wrote:

I copied the first two trace files and placed them in their own folder and tried to convert them again and got this error.

Info - ConvertWorkload.Program : ConvertWorkload, Version=1.5.8.0, Culture=neutral, PublicKeyToken=null 1.5.8

Info - ConvertWorkload.Program : Checking LocalDB...

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Writing event data to D:\DReplay\WorkloadTools\SQLworkload.sqlite

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Object reference not set to an instance of an object.

Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at WorkloadTools.Listener.SqlTransformer.Skip(String command) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\SqlTransform

er.cs:line 54

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTrace

EventDataReader.cs:line 338

at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 64

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 1 events saved

Info - ConvertWorkload.Program : Converter stopped.

On Fri, Oct 2, 2020 at 5:35 AM Gianluca Sartori notifications@github.com wrote:

Hey John, Thanks for the info. I think the best option would be to try to load fewer files (like 10 or so) and see whether that makes a difference. In case it doesn't, you could send me a sample of your file and I could try to debug it. The logic for reading the trace files is one of the most complicated parts of the application and the chances for a bug are high.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spaghettidba/WorkloadTools/issues/96#issuecomment-702656894, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLVWYR6VHGW45VBDV4UTR3SIWUG5ANCNFSM4R4Y3UBQ .

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

-- Regards, John Sterrett Blog: http://johnsterrett.com Twitter: @JohnSterrett http://twitter.com/johnsterrett

spaghettidba commented 3 years ago

That's awesome, thanks!

spaghettidba commented 3 years ago

Hey John, did you manage to check in the code? I'm happy to accept PRs here :)

TruKurl commented 2 years ago

Hi Gianluca , I am trying to convert large trace files ( 1x 10GB and 1 rollover file 1.2GB) and I get this error below after 6528000 event gets saved.

Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 6528000 events saved Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData() at System.Data.SqlClient.SqlDataReader.get_MetaData() at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted) at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry) at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) at System.Data.SqlClient.SqlCommand.ExecuteReader() at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadTraceData(SqlConnection conn, ReadIteration currentIteration) in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 353 at WorkloadTools.Listener.Trace.FileTraceEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\Trace\FileTraceEventDataReader.cs:line 65 Error - WorkloadTools.Listener.Trace.FileTraceEventDataReader : The wait operation timed out Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : Closing the connection to the output file Info - WorkloadTools.Consumer.WorkloadFile.WorkloadFileWriterConsumer : 6528001 events saved Info - ConvertWorkload.Program : Converter stopped.

spaghettidba commented 2 years ago

It looks like the connection to your localdb instance did not work at some point. I cannot tell why unfortunately.

TruKurl commented 2 years ago

Thanks. Retryed a few more time. Always errors out after recording the same number or events.

spaghettidba commented 2 years ago

How many events do you have in the trace?

TruKurl commented 2 years ago

7,436,001 events in total. Could this be about hitting the localdb 10gb limit ? The first trace file is 10GB in size

spaghettidba commented 2 years ago

I don't think so, the data is read from trace files and not from a database. Maybe it's the application taking too long to query the trace file. Let me check the code...

mcflyamorim commented 1 year ago

@8-KB I think this is an issue with the CommandTimeout used on SqlCommand on ReadXEData, since this was not specified, it is using the default of 30 seconds which may be not enough to read the xevent files. In the meantime, as a workaround you can try to read less files to be able to read it under the 30 seconds limit...

@spaghettidba I think you can easily fix it by adding a cmd.CommandTimeout = 0; after line 95 of file https://github.com/spaghettidba/WorkloadTools/blob/d6e6e02cbc9cc01e8af5a6c3c6170f82d1109c51/WorkloadTools/Listener/ExtendedEvents/FileTargetXEventDataReader.cs#L95

Let me know if you'd like me to open a new issue to make this change.