fslaborg / Deedle

Easy to use .NET library for data and time series manipulation and for scientific programming
http://fslab.org/Deedle/
BSD 2-Clause "Simplified" License
924 stars 196 forks source link

C#: Slow row indexing makes merging frames slow #367

Closed CarlKCarlK closed 6 years ago

CarlKCarlK commented 6 years ago

Greetings,

From C#, I want to merge frames. I couldn't find a method for that, so I created the utility function below. Sadly, when the # of rows is 10K or more, it takes minutes to work. I think the problem is that integer index access to rows is slow. If this can be fixed (or if there is a better way to merge frames), that would be great.

    public static Frame<int, string> MergeFramesByRow(IList<Frame<int, string>> judgementFrameList)
    {
        var judgementAllFrame =
            Frame.FromRows(
                from judgementFrame in judgementFrameList
                from rowKey in judgementFrame.RowKeys
                select judgementFrame.Rows[rowKey]);
        return judgementAllFrame;
    }
CarlKCarlK commented 6 years ago

This fixes it:

    public static Frame<int, string> MergeFramesByRow(IList<Frame<int, string>> frameList)
    {
        int start = 0;
        var list2 = new List<Frame<int, string>>();
        foreach(var frame in frameList)
        {
            list2.Add(frame.IndexRowsWith(Enumerable.Range(start, frame.RowCount)));
            start += frame.RowCount;
        }
        var result = FrameModule.MergeAll(list2);
        return result;
    }