search

Bihaqo / t3f

Tensor Train decomposition on TensorFlow
https://t3f.readthedocs.io/en/latest/index.html
MIT License
219 stars 55 forks source link

Provide example of converting massive tensor to TT-format at scale. #140

Open cjnolet opened 6 years ago

cjnolet commented 6 years ago

I just finished reading Chicolki's monograph on tensor networks for dimensionality reduction and I'm excited to start using t3f for some problems I'm working on.

I noticed in the last chapter of the monograph that I should be able to represent my input tensor directly as a set of TT cores but I'm having trouble finding a good example of this being done at scale in t3f and the examples in the codebase.

The problem I want to start with is using t3f to factorize a PMI matrix at scale using HOSVD. My data is going to contain several 10's of millions (to possibly billions) of items that I need to map to this PMI matrix, meaning that I'll end up with a square matrix of size N*N to factorize. I might be able to sparsify this matrix a tad using the normalized PMI but I have not checked exact metrics as of yet.

I'm working with several SMP systems with >=3tb ram that are equipped with several older GPUs (K80's) on each node.

I'm also not seeing an easy way to turn even a large-scale sparse matrix into a TT tensor, since it doesn't appear that the t3f API supports going directly from a common sparse library into the TT format without having to convert it to its dense form.

Forgive me if I've missed details which are documented publicly. I have looked through the published documentation and example notebooks and have not been able to figure out where to start.

Bihaqo commented 6 years ago

Unfortunately, it's really hard to parallelize HOSVD, not even for multiple nodes (servers), but even just for exploiting GPU power.

At the moment tf.svd is slower on high-end GPUs than on CPUs (which is frustrating, see this bug: https://github.com/tensorflow/tensorflow/issues/13603#issuecomment-383318506). Likewise, it would be really hard to implement a multi-server version of HOSVD.

To make things even grimmer for you, t3f doesn't support sparse stuff well at the moment (the same applies to TensorFlow itself).

I guess your best bet would be to use ttpy if your staff is sparse and t3f if not, and run it on the single most powerful machine you can find.

Or somehow avoid directly decomposing your matrix into TT format (i.e. came up with a smarter algorithm) :)

cjnolet commented 6 years ago

That's a bummer.

I was under the impression that the whole purpose of these tensor networks (and, more specifically, Tensor Trains) was to find ways to make large-scale tensor factorizations more tractable by being able to

1) break the operations up over many machines and, 2) provide a more compressed "core" representation that scales better with order size than other formats

Am I wrong in this interpretation or is the technology just not yet aligning fully with the theory?

I know it's not possible to do a distributed version of SVD without approximation but the monograph seemed to imply that there was an incremental approach to performing the SVD, which I took to mean that it could be broken into a series of distributed or even message-passing algorithms for tractability.

Am I way off on that one? Thanks again for your reply!