search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.92k stars 1.86k forks source link

large models loading with mlcontext.model.load ends up OutOfMemoryException #6948

Open boneatjp opened 5 months ago

boneatjp commented 5 months ago

System Information (please complete the following information):

Describe the bug When loading models which size are over 100MB, mlcontext.model.load throws Exception.

Viewing stack tace, 'System.Reflection.TargetInvocationException' (Microsoft.ML.Core.dll)

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Additional context When loading models which size are less than 100MB, no problem.

michaelgsharp commented 5 months ago

Is it a tensorflow/torchsharp model or a normal ML.NET model?

Are you able to upload the model for us to have a repro?

ghost commented 5 months ago

This issue has been marked needs-author-action and may be missing some important information.

boneatjp commented 5 months ago

I just use Microsoft.ML.AutoML, mostly models are LightGBM, I guess.

Uploading models isn't any trouble for me. I'm not sure if I could upload it here with that kind of size.

boneatjp commented 5 months ago

I found out that I could use GitLFS to upload large files. Where is the good place to upload it?

boneatjp commented 5 months ago

model.zip.001.zip model.zip.002.zip model.zip.003.zip model.zip.004.zip model.zip.005.zip model.zip.006.zip model.zip.007.zip model.zip.008.zip

Using 7-zip, model.zip is splitted to model.zip.001 to model.zip.008. Here files must have extension like .zip so renamed like model.zip.001 to model.zip.001.zip. They should be renamed refore combining.