search

openopt / chop

CHOP: An optimization library based on PyTorch, with applications to adversarial examples and structured neural network training.
http://openo.pt/chop
Other
76 stars 14 forks source link

Rank 1 tensor decomposition #63

Open GeoffNN opened 3 years ago

GeoffNN commented 3 years ago

We'd like to have a tensor LMO for tensor nuclear norm constraints. Therefore, an implementation of https://hal.archives-ouvertes.fr/hal-01385538/file/SeROAP_final.pdf (or some other rank 1 tensor decomposition algorithm) would be nice.

Our NuclearNormBall.lmo should do this by default for tensor inputs of shape (batch_size, *shape).

fabianp commented 3 years ago

random thought: can we get away with an (unbiased estimator of) the largest eigenvalue? We could get this for example from the power method + russian roulette

GeoffNN commented 3 years ago

Good question -- how many iterations would you perform with RR? I think I'm currently doing 10 and it works well enough for the matrix LMO.

It might be more interesting for higher order tensors.

fabianp commented 3 years ago

I don't know. Now that I think about it its not even clear that for the LMO an unbiased estimator is better than an inexact one (i.e., deterministic but with a bounded error as would be the case in truncated power method)