search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.44k stars 3.49k forks source link

Requesting Sparse Input Support - MOE layer #665

Open pgyrya opened 6 years ago

pgyrya commented 6 years ago

feature request

Description

I would like to suggest to prominently add support for sparse input tensors - specifically for mixture-of-expert gating functionality (e.g. local_moe and noisy_top_k_gating).

Sparse input support is prominently important when installing layers in the beginning of the network that relies on sparsely encoded data - a common phenomenon in large-scale feature problems.

rsepassi commented 6 years ago

So why not first embed the sparse inputs then use an MOE layer on those dense representations? What did you have in mind?

pgyrya commented 6 years ago

The example I previously worked on was a bag-of-words representation of text, where embedding doesn’t seem like a natural choice.

From: Ryan Sepassi notifications@github.com Sent: Thursday, May 3, 2018 10:42 PM To: tensorflow/tensor2tensor tensor2tensor@noreply.github.com Cc: Pavel Gyrya pavel.gyrya@cornell.edu; Author author@noreply.github.com Subject: Re: [tensorflow/tensor2tensor] Requesting Sparse Input Support - MOE layer (#665)

So why not first embed the sparse inputs then use an MOE layer on those dense representations? What did you have in mind?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/tensorflow/tensor2tensor/issues/665#issuecomment-386492854 , or mute the thread https://github.com/notifications/unsubscribe-auth/AeFaQe5NzwzuH02SUe4ycfCixxjSLTB7ks5tu7_wgaJpZM4S5QYA . https://github.com/notifications/beacon/AeFaQd1Pbb3HD-Riz0K2RFh90_6rXaiDks5tu7_wgaJpZM4S5QYA.gif