search

rjagerman / glint

Glint: High performance scala parameter server
MIT License
168 stars 67 forks source link

Can glint support BigInt type? #63

Open craighagerman opened 7 years ago

craighagerman commented 7 years ago

I have been playing around with glint for a few weeks and am excited about it's potential. However the counts I want to eventually push to the parameter server are larger than the max value of type Long. I see there isn't any support for BigInts. Wondering if there is any reason for this?

rjagerman commented 7 years ago

Thanks for your question!

The main reason for this limitation is performance. The vector/matrix data is stored as dense arrays of primitives which the JVM stores as a single object. When using BigInts (which is an object) we need to store an object for each entry, which has significant garbage-collection overhead because we are going from O(1) for a primitive array to O(n) for an object array.

We use breeze.semiring to check the type supplied to the client.vector[_] and client.matrix[_] methods. It seems BigInt is supported by breeze.semiring, so it should not be extremely difficult to add this to glint!