spaghettidba / WorkloadTools

A collection of tools to collect, analyze and replay SQL Server workloads, on premises and in the cloud
MIT License
234 stars 53 forks source link

Replay controller stops on error. #46

Closed marco-gallinari closed 5 years ago

marco-gallinari commented 5 years ago

Hi,

I'm trying to use SqlWorkload in order to benchmark and validate performance of a new production Sql Server instance before migrating from the current cluster.

Both servers are using Sql server 2014 Enterprise; baseline server is the master node of an AlwaysOn cluster, while replay server is a single server.

I'm able to run analysis + replay on baseline server, a few error pops up but the workload seems to be running just fine. If I run analysis on replay server, everything works fine with current test workload, but as soon as replay consumer starts to push workload from baseline, the analyzer throws an error, and stops working:

2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader : Error converting XE data from the stream: Unable to cast object of type 'System.Byte[]' to type 'System.String'.
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     client_app_name       : WorkloadTools-ReplayWorker
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     database_name         : msdb
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     client_hostname       : WSR02XJBW
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     server_principal_name : skylab
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     session_id            : 101
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener : Unable to cast object of type 'System.Byte[]' to type 'System.String'.
2019-04-10 11:39:43.1492 - Error - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener :    at WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\ExtendedEvents\StreamXEventDataReader.cs:line 165
   at WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\ExtendedEvents\ExtendedEventsWorkloadListener.cs:line 243
2019-04-10 11:39:43.1802 - Info - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener : Extended Events session [sqlworkload] stopped successfully.
2019-04-10 11:39:48.6488 - Info - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener : Extended Events session [sqlworkload] stopped successfully.
2019-04-10 11:39:48.8051 - Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : 178 rows aggregated
2019-04-10 11:39:48.8051 - Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : 77 rows written
2019-04-10 11:39:48.8213 - Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : Performance counters written
2019-04-10 11:39:48.8519 - Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : Wait stats written
2019-04-10 11:39:48.8675 - Info - SqlWorkload.Program : Controller stopped.

I'm running version 1.2.13 both for analysis+replay on baseline server, and for analysis on replay server.

I tried downgrading the replay analyser to version 1.2.11; error is still thrown but the analyzer continues working properly.

ghotz commented 5 years ago

Ciao, this has been fixed right after 1.2.13, @spaghettidba still has to publish the new release. If you are in a hurry I can tell you what to change in the code but you'll have to be able to compile it.

marco-gallinari commented 5 years ago

Ciao gianluca, grazie!

I'm not in a hurry, it's fine for me to wait for the next release.

Keep up the good work!

jpomfret commented 5 years ago

This is still happening, I have v1.2.14 downloaded and when replaying a capture if it encounters an error the process of SqlWorkload.exe I have running the AnalysisConsumer to recapture the performance data errors and stops.

This has happened with two different errors, both have caused the capture thread to end:

Warn - WorkloadTools.Consumer.Replay.ReplayWorker : Worker [102] - Error: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

or:

Worker [107] - Error: Could not find prepared statement with handle 1073766277.

Both above errors cause the following error to be thrown by the capture thread:

Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader : Error converting XE data from the stream: Unable to cast object of type 'System.Int32' to type 'System.String'.
Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     event type            : Error
Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     client_app_name       : WorkloadTools-ReplayWorker
Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     database_name         : database1
Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     client_hostname       : server1
Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     server_principal_name : DOMAIN\USERNAME
Error - WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader :     session_id            : 66
Error - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener : Unable to cast object of type 'System.Int32' to type 'System.String'.
Error - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener :    at WorkloadTools.Listener.ExtendedEvents.StreamXEventDataReader.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\ExtendedEvents\StreamXEventDataReader.cs:line 198
   at WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener.ReadEvents() in C:\GitHub\WorkloadTools\WorkloadTools\Listener\ExtendedEvents\ExtendedEventsWorkloadListener.cs:line 251
Info - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener : Extended Events session [sqlworkload] stopped successfully.
Info - WorkloadTools.Listener.ExtendedEvents.ExtendedEventsWorkloadListener : Extended Events session [sqlworkload] stopped successfully.
Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : 234 rows aggregated
Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : 63 rows written
Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : Performance counters written
Info - WorkloadTools.Consumer.Analysis.WorkloadAnalyzer : Wait stats written
Info - SqlWorkload.Program : Controller stopped.
spaghettidba commented 5 years ago

Thanks for the heads-up, Jess. I'll try to fix it ASAP.

jpomfret commented 5 years ago

Thanks @spaghettidba!! Will you make a new release with this fix in?

spaghettidba commented 5 years ago

It's published now! Please let me know if it works for you.