Closed matthew-jurewicz closed 4 years ago
@matthew-jurewicz Hi Matthew! Yup, you just to pass the token ids and make sure you instantiate the language model with the num_tokens
set to the maximum of the ids! The token embedding are fetched from the embedding table here https://github.com/lucidrains/routing-transformer/blob/master/routing_transformer/routing_transformer.py#L523
Excellent! I'm a big fan of your work!
This was an implementation of someone else's research https://openreview.net/forum?id=B1gjs6EtDr Hope you find it useful!
What I mean is, as far as I know, no one's written the code for a fully-functional sparse transformer, much less this.
I'm looking through the code, and I'm not seeing the token IDs being converted to one-hot encoded vectors. Is the input to the language model with autoregressive wrapper the token IDs?