search

rjagerman / glint

Glint: High performance scala parameter server
MIT License
168 stars 67 forks source link

Random init #60

Open MLnick opened 7 years ago

MLnick commented 7 years ago

Would there be interest in some random initialisation mechanism for vectors / matrices? Often it is ok to init with 0s (the implicit case with new Glint data structures). But sometimes you want to have random normal or random uniform init.

It's possible to do it yourself via a forEachPartition and using a push, but it is a bit of a hack really. It seems much simpler & cleaner to provide an interface for this, like client.random.vector.randn[Double](dim, mean = 0, std = 1.0).

What do you think?

rjagerman commented 7 years ago

This seems quite useful and relatively easy to do. I feel like Glint is starting to look more like a distributed linear algebra library and less of a parameter server, which is quite an interesting direction.

I would need to refactor some code that deals with initialization though. The easiest approach to me seems to design an abstract Initializer class, which can be implemented as a ZeroInitializer or a RandomNormalInitializer. The class should be serializable and shipped to the parameter servers when we construct the actors (e.g. by providing it in the Akka Props, such as at Client.scala#L175).

This would allow us to easily write some high-level wrapper functions in Client such as client.randn(...). I'm still quite busy in the next few weeks, but after Spark Summit EU I should have some more time to look into this. Nevertheless, I like the suggestion!