search

spaghettidba / WorkloadTools

A collection of tools to collect, analyze and replay SQL Server workloads, on premises and in the cloud
MIT License
234 stars 53 forks source link

Out of Memory when analyzing huge workloads #74

Closed spaghettidba closed 4 years ago

spaghettidba commented 4 years ago

The WorkloadAnalyzer uses a DataTable to write the data to the database in bulk. It also uses the DataTable to keep data in memory while it is being collected in the internal queue (a plain Queue of WorkloadEvents). The result is that events are held in memory briefly in the queue and moved to the DataTable, but that is a poor choice in terms of memory footprint.
It would be wiser to use a BinarySerializedBufferedEventQueue, write to a temporary table in batches and perform the aggregations on the database side. This would mean a significant rewrite of the aggregation logic client side but it would introduce significant benefits. Also, the DataTable is not used to write the data to the destination table directly, but it is queried from Linq to create the aggregated dataset, so any IEnumerable would do.

spaghettidba commented 4 years ago

Closed with commit 55171283a96f36b3f8094342e0443aa5c6c6b5bb