Model Training stuck at Finalizing model...

[JesseP7]

System Information (please complete the following information):

• Model Builder or CLI Version: 16.14.3.2307502
• Visual Studio Version (if applicable): 17.4.4

Describe the bug

• On which page of the tutorial did you run into an issue (URL): https://dotnet.microsoft.com/en-us/learn/ml-dotnet/get-started-tutorial/train
• Clear description of the problem: After training finishes, it gets stuck at 'Finalizing model...'. Memory usage by ServiceHub.Host.netfx.x64 grows until it uses up all the available memory on the computer. CPU usage also holds constant around 50%. I have let it run for over a day, which just makes the computer unusable, and it never finishes. I have tried various different configuration options, all with the same outcome.

Screenshots

[zewditu]

@JesseP7 Sorry for hitting the issue! Did you the issue for specific scenario or for all of them?

[JesseP7]

I am having the issue with any scenario. JesseOn Jan 31, 2023, at 2:05 PM, zewditu Hailemariam @.***> wrote: @JesseP7 Sorry for hitting the issue! Did you the issue for specific scenario or for all of them?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[JakeRadMSFT]

@JesseP7 -

Can you share more details about your dataset:

• How many rows?
• How many columns?

Screenshots of the Data page would be helpful too.

[JesseP7]

Training Dataset has 30,699 rows, with 15 columns. Pulling from a csv file.

Data column layout below. dRank is the label. [image: image.png]

[image: image.png]

On Wed, Feb 1, 2023 at 11:57 AM Jake @.***> wrote:

@JesseP7 https://github.com/JesseP7 -

Can you share more details about your dataset:

• How many rows?
• How many columns?

Screenshots of the Data page would be helpful too.

— Reply to this email directly, view it on GitHub https://github.com/dotnet/machinelearning-modelbuilder/issues/2469#issuecomment-1412391302, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASQWRZ2NDOONDIPSNHBV3S3WVKIWPANCNFSM6AAAAAAUMXPSXQ . You are receiving this because you were mentioned.Message ID: @.***>

[JesseP7]

After letting it run for several days, it does finally stop running, and returns a failure message:

at Microsoft.ML.AutoML.AutoMLExperiment.d26.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.MultiClassificationExperiment.d14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/MultiClassificationExperiment.cs:line 123 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.ML.ModelBuilder.AutoMLEngine.d_21.MoveNext() in //src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 144

On Wed, Feb 1, 2023 at 12:30 PM Jesse Pierce @.***> wrote:

Training Dataset has 30,699 rows, with 15 columns. Pulling from a csv file.

Data column layout below. dRank is the label. [image: image.png]

[image: image.png]

On Wed, Feb 1, 2023 at 11:57 AM Jake @.***> wrote:

@JesseP7 https://github.com/JesseP7 -

Can you share more details about your dataset:

• How many rows?
• How many columns?

Screenshots of the Data page would be helpful too.

— Reply to this email directly, view it on GitHub https://github.com/dotnet/machinelearning-modelbuilder/issues/2469#issuecomment-1412391302, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASQWRZ2NDOONDIPSNHBV3S3WVKIWPANCNFSM6AAAAAAUMXPSXQ . You are receiving this because you were mentioned.Message ID: @.***>

[gbanuk2]

I have a problem with mlnet hanging in various locations. I've noticed that when it hangs and I go into Task Manager/Analyze wait chain, I can see the mlnet.exe process waiting on another thread of the same mlnet process. Not sure if that's related to the hang.

Example from Analyze Wait Chain window: mlnet.exe (PID: 17132) Thread: 768 --mlnet.exe (PID: 17132) Thread: 2324

[aliajboy]

I have the same problem but sometimes Stucks not always. I just start and stop to make it out of that stuck position.