Closed buybackoff closed 9 years ago
I think that adding nicer aliases for shift -1
and shift +1
would be useful.
Although, I think it is hard to come up with good names for those! I think finance people will probably know lag, so perhaps that would be good addition? As for shiftPrevious
and shiftNext
, I think those are very similar to shift -1
and shift +1
(and I'm not sure if they help much?). At least, I don't really have a good intuition whether these are shifting forward/backward the values or the keys of the series. So something like shiftValuesForward
might be more descriptive, but then - it's ugly as hell!
Perhaps @hmansell or @adamklein have some opinions on this?
May be very explicit comment in the intellisense doc could be enough, e.g. "for historical values, use positive numbers". The word lag
implies history, and using uint
for parameter will remove ambiguity completely.
I think those two would be great
shift
(using something along the lines of what you said above)Series.lag
and Frame.lag
makes sense too (in the comment, it would be good to give example showing the uint
literal, e.g. Series.lag 1u ts
- in case people don't know the syntax)So, if you can send PR adding those two, that would be amazing!
Will do next week
A week was too optimistic after I defined lag
as a simple function and solved the issue locally, sorry...
Some more issues: I miss overlapped windows (like here), stats for product, JSON.NET serialization and want to try Yeppp for boosting speed.
Overlapped windows are a general case for windows/chunks with a size and a step. When step = 1 - it is a window, when step=size - it is a chunk. Now I do this with size as an integer multiple of steps, then calculate chunks of step size, and then a process windows of the multiple of chunks. It works, but ugly.
Product operations are surprisingly missing from moving/expanding in Stats.fs. I do it now as let index = log (return + 1.0) |> Stats.expandingSum |> Series.Exp
or (fun s -> Math.Exp <| (s.Data + 1.0).Log().Sum() )
and could live without built-in product for a while, but they are often needed in a fast simple form.
JSON.NET (v.6), despite being so versatile, cannot serialize series out of the box. I haven't looked deeper why and use an intermediary DTO with key/values array to feed to JSON.NET, but serialization other than .csv is useful.
There is a nice package https://www.nuget.org/packages/Yeppp.CLR.Bundle/ that works on all platforms and gives 5-10x performance boost for double[]
operations with 100+ elements (in isolated tests). Now I call many operations such as log, sum, add, product on series much longer than 100 in a tight loop, and using a SIMD-optimized library seems like a free lunch. Especially when I what to minimize design time and get quick and dirty, but working within reasonable time, code.
Finally, documentation for chunkSizeInto
says:
The key is the last key of the chunk, unless boundary behavior is
Boundary.AtEnding
(in which case it is the first key).
But the line always takes the first one and should be instead:
[<CompiledName("ChunkSizeInto")>]
let inline chunkSizeInto bounds f (series:Series<'K, 'T>) : Series<'K, 'R> =
series.Aggregate(
ChunkSize(bounds),
(fun d -> d.Data.Keys |> if (snd bounds) = Boundary.AtEnding then Seq.head else Seq.last),
fun ds -> OptionalValue(f ds))
I think that the documented behavior is the correct one.
I won't tell any date now, but I want to do this in random priority at some point when I finish a project at work. I will keep posting issues here if I find more. What do you think about these issues?
Thanks for the fantastic feedback and suggestions :-).
Overlapped windows
This sounds very useful - and the fact that it generalizes windowing & chunking is nice too. It would be nice to have this! (Though there are some more tricks around chunking - you can tell it to create chunk from the end and have incomplete chunks at the beginning - but I guess that can be done with overlapping windows too...)
product operations
This looks like a nice addition too!
JSON.NET 6
Supporting JSON.NET serialization would be nice. I wouldn't want to have a dependency on it, but if we can figure out what is it that's breaking it (and fix that), that would be great!
Yeppp
This looks interesting. Probably something that we should investigate! There are some performance tools in the tests
folder that can be used to compare the speed of various operations before & after..
So, any PRs for any of these would be more then welcome :-)
What about chunkSizeInto? I have already changed the behavior locally according to documentation and hope that master/nuget will be the same rather than documentation changed.
However, all other window...
and chunk...
functions are consistent in returning the first key. So changing the comment in chunkSizeInto
is probably better.
Series silently accepts empty arrays in constructor
var dates = new DateTime[100];
var values = new double[100];
var series = new Deedle.Series<DateTime, double>(dates, values); // expect to throw here
var enumerable = series.Keys;
var newArr = enumerable.ToArray(); // but throws here
With the spirit of "Don't make me think", I think it will be useful to have Series.shiftPrevious and Series.shiftNext that accept only positive numbers. I had to read the description more than twice and still was not comfortable without a test if I am looking into the future or the past... Another option is to use a name
lag
that only accepts positive values for history, to protect people from looking into future during backtests. It is too easy to make a mistake now with +1/-1 and then hard to find it. One could argue thatsomething[-1]
is more natural so theshift
should use -1 for history. What do you think?