search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
9.02k stars 1.88k forks source link

GC pause caused by Tuple<int, int> on model load #6807

Open CodedBeard opened 1 year ago

CodedBeard commented 1 year ago

Is your feature request related to a problem? Please describe. My app is quite sensitive to GC pauses, so I've been doing allocation profiling. I noticed that a lot of GC pauses are being caused by the usage of Tuple<int, int> within the Attention class when the model is loaded.

https://github.com/dotnet/machinelearning/blob/aaf226c7e7c359edf27e663362e928e02c8b9d0f/src/Microsoft.ML.TorchSharp/AutoFormerV2/Attention.cs#L72-L79

Describe the solution you'd like In my own fork, I've switched to using ValueTuple, which has removed the allocations and thus GC pauses

Describe alternatives you've considered I don't think there's an easier/cleaner fix for this.

Additional context Using Microsoft.ML.TorchSharp version 0.21.0-preview.23266.6, I observed 240k allocations of Tuple<int, int> during model loading, coming in just after Tensor.

ml-allocations

After switching to ValueTuple on my own fork, these allocations were removed, resulting in a significant reduction in CG pauses during model loading.

GeorgeS2019 commented 1 year ago

@michaelgsharp Please add deep-learning label to all TorchSharp/HuggingFace issues image