Open tbreloff opened 9 years ago
Hi Tom,
Nice meeting you at JuliaCon. I think the main things we'd need to merge are the HyperLogLog implementation, the online bootstrap and the AdaGrad-based SGD OLS/logit code. I'd love help porting all of that.
I'd also like to move OnlineStats.jl into JuliaStats so that we can all collaborate more effectively.
Nice meeting you too! (and I'm impressed by the immediate response). When I have a few minutes to dive into your code again, I can try to make a first pass merge. @joshday may be able to help as well, since he'll probably have better knowledge about the implementation details.
I'd love for the package to get more visibility in the hope that others can help add online variations to lots of algorithms. Also, as a side note, Josh and I discussed your desire to allow various user-defined weightings, and that should be pretty straightforward given the design... equal, exponential, and stochastic weightings are already supported.
I think the really awesome thing here would be showing how to use this stuff with the aggregation UDF's in SQLite that Jacob Quinn's been working on.
Agreed, and on that note, I was brainstorming yesterday (thanks to comments from @quinnj) on how to make an interface to this stuff that made it easier to compose/chain various online stats and data flows without explicitly updating each object. This is very similar to Reactive, which might be the right package to use. It would be nice to write something like:
demeaned = x - Mean(x)
Here x
is some stream in a Reactive pipeline, and Mean(x)
implicitly produces an OnlineStat
which lifts from the stream. Now every time there's a new data point, Mean(x) is updated, which is then lifted to update demeaned
, all without calling update!
. This may be trickier than expected considering all the different ways data can come in, but it could drive some really nice data analytics.
I'm happy to help to get things rolling. Sounds like I'll need to look a bit more into SQLite and Reactive as well. I like where this is headed.
@johnmyleswhite After seeing some of the comments on the PRs here, would you be receptive to a PR that adds a note in the README which directs people to OnlineStats.jl?
Yes, I definitely would be open to that.
Excellent. Thanks!
I am trying to move my code from StreamStats to OnlineStats. StreamStats does overload Base.merge to merge two StreamStats, which becomes very useful for parrallel applications. Is there a similar functionality in OnlineStats?
There used to be merge methods for most OnlineStat types. They've been temporarily lost in some rewrites. I'll make it a priority to get them back in.
Great, thanks for your work.
It's my understanding that this repo has been inactive for a while and that the functionality could be merged into OnlineStats.jl, which has lots of awesome online algorithms thanks to Josh Day's PhD work (and it should probably be included in JuliaStats).
John (or anyone else willing to help): assuming this makes sense to you, what are the steps required for a merge? What exists here that doesn't exist in OnlineStats? Do you need/want help with any part of it?