SDCA seems to contain a livelock

actopozipc commented 4 years ago

System information

OS version/distro: Windows 10
.NET Version (eg., dotnet --info): 4.6.1

Issue

What did you do? I tried to call the SDCA method and fit the model.
What happened? When I call the SDCA method in the pipeline, the code wont get past the Fit method. Noticeable is the fact that the CPU percentage wont go down, so its still calculating

However there is no exception. The program just freezes

Source code / logs

grafik

CPU usage: grafik

antoniovs1029 commented 4 years ago

Can you please provide your complete code and dataset to reproduce this error? And also, how much time did you let it train? and how big is your data? Thanks

actopozipc commented 4 years ago

I posted the issue after ~30 minutes. After more than a hour (so ~30 minutes after posting) there was some result and the code passed on. It still seemed weird to me due to the fact that the dataset isnt very large. The whole code is on my github, altthrough I changed it from SDCA to FastTree. The dataset is around 50-60 co2-emissions with the population (both floats, althrough the population are whole numbers) and a year (int). I copied it to pastebin. Its noteable that the values get normalized, so the biggest values are 1 and the samllest 0.
Hope this helps, althrough Im not sure anymore if this really is a livelock.

antoniovs1029 commented 4 years ago

Hi @actopozipc . So I wasn't able to reproduce your issue, but I would recommend trying out updating your ML.NET references to version 1.5.0-preview2. Issue #3830 was very similar to your issue, and I've recently found that updating to 1.5.0preview1 or preview2 actually fixed it. Please, let us know if this fixes your issue.

To try to reproduce this issue, I cloned your repo and tried to ran it as it is, but got several error messages so I couldn't test that.

I also tried out writing the code below, and train the SDCA model with the dataset you provided. I guess this is how you would have done it, but after looking on different commits of your repo, I couldn't find the exact code you would have used to load this particular dataset (closest is this, which has more columns). In any case, running the below code in a brand new Console App with .NET Framework 4.6.1 ran without any problem (I tested this both with ML.NET 1.4.0 which is the one you have on your repo, and with ML.NET 1.5.0-preview2).

using Microsoft.ML;
using Microsoft.ML.Data;

namespace ConsoleApp461_2
{
    class Input
    {
        [LoadColumn(0)]
        public float year;
        [LoadColumn(1)]
        public float value;
    }

    class Program
    {
        private static readonly string DataPath = @"C:\Users\anvelazq\Desktop\issue4767\input.tsv";

        static void Main(string[] args)
        {
            var mLContext = new MLContext();
            IDataView dataView = mLContext.Data.LoadFromTextFile<Input>(DataPath, hasHeader: true);
            IEstimator<ITransformer> pipeline = mLContext.Transforms.CopyColumns(outputColumnName: "Label", inputColumnName: "value")
                                                .Append(mLContext.Transforms.Concatenate("Features", "year"))
                                                .Append(mLContext.Regression.Trainers.Sdca(maximumNumberOfIterations: 2000)); // It works for me even if I remove the maximumNumberOfIterations parameter

            ITransformer model = pipeline.Fit(dataView);
            var x = model.Transform(dataView).Preview();
        }
    }
}

antoniovs1029 commented 4 years ago

Hi, @actopozipc . Since I couldn't reproduce this issue I will close it. Please feel free to reopen it if you're still having problem with this, and are able to share a repro. Thanks.

dotnet / machinelearning