Open vijoc opened 2 years ago
For what it's worth, I did some more testing and found that the bad performance can also be avoided by using Frame.getNumericCols
or even simply Frame.getCols
.
// About the same performance as approach number 3 from above
frame
|> Frame.getNumericCols
|> Series.mapValues (Series.chunkWhileInto inlineComparison Stats.mean)
|> Frame.ofColumns
// Slightly worse behavior, but still around 50 milliseconds versus ~10+ milliseconds for the above
// or ~1+ seconds for Frame.mapColValues
frame
|> Frame.getCols
|> Series.mapValues (Series.chunkWhileInto inlineComparison Stats.mean)
|> Frame.ofColumns
I ran into an issue where a frame with tens of thousands of rows and a handful (<10) of columns is very slow to apply
Frame.mapColValues
over. On the other hand, when first mapping the series "manually" (see below) and then joining withFrame.ofColumns
, the difference in speed is of orders of magnitude.What I'm looking to do is a naive hourly averaging of time series data. I implemented it with essentially the following:
It may well be that I'm overlooking something here, I'm not super confident with either the Deedle codebase nor performance diagnosis in F#. I do have a setup with BenchmarkDotNet, which I could extract and share if that would be helpful.
Is this kind of performance expected? I believe I can avoid the issue in my use case by using method 3 from above, but I'm struggling to understand what could cause this kind of performance difference in this case.