rostyboost
commented
7 years ago Thanks for your question @yvdriess. If I understand correctly, you still want a dense forward-prop for your loss, but your gradient that needs to be backpropagated is sparse, because it only involves a subset of the output neurons (presumably on the coordinates matching what's in your SparseF[] labels?). This is definitely possible in vectorflow. There is an overload of the callback mechanism resolved at compile-time where you can populate a sparse gradient for your loss. It's not so well documented, but you can find a dummy example here. So you can just append the non-zero SparseF coordinates of the gradient into this buffer. This would be the most efficient way for your problem, as this would avoid any unnecessary "dense to sparse" operation.
Typically vectorflow works by reference and tries to avoid any memcopy, so there is no "gather" as it is assumed that the memcopy cost outweights any potential SIMD-gain from "densifying" a sparse operation, which tends to be the case for most IO-bound / shallow problems.
yvdriess
I have a multi-target workload with very sparse labels. In other words, a single observable/sample is not a (bool label, SparseF[] features) tuple, but rather a (SparseF[] labels, SparseF[] features). During the learn/optimize steps, the predict run currently appears to just produce a dense vector of predictions (e.g. a 65k vector), but only a few elements (e.g. 200) of that feature vector will have a label and contribute to the gradient.
a) Is there a way in vectorflow to deal with the multi-target learning scenario. I suppose I should make a loss function that scatters the non-zero gradients into a zero-vector of the predict-vector size? b) Where I can see me doing a), it is kind of wasteful as it means producing a large vector of predictions where only a handful are used to produce the gradient.
I implemented b) In Tensorflow by gathering columns of the last-layer weight matrix, where the gathered columns match the label's non-zero coordinates (viz. target index). For Vectorflow, it probably makes more sense to have a SparseData() leaf/output layer. It appears that float[] dense output is currently hardcoded in the learning interface.. The loss function interface could probably be kept the same, as it should not care about the actual target index, but there needs to be some logic that produces a float[] from the SparseF[] output before passing it to the loss function, and then transforms the dense gradient vector back into a sparse gradient for back-prop. SparseLinear layer? (dense prev-vector + SparseData input, SparseData output)