search

ManifoldRG / NEKO

In Progress Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks
https://discord.gg/brsPnzNd8h
GNU General Public License v3.0
46 stars 10 forks source link

need to review next_token in predict_text function (code changes due to add_text_modality) #25

Closed henryj18 closed 1 year ago

henryj18 commented 1 year ago

Our NEKO vocabulary includes non-text tokens such as discrete action tokens, continuous action tokens etc., along with text tokens. When generating text tokens for tasks such as image-caption and pure text (perhaps more), we need to investigate whether we should restrict the generated tokens fall in the range of the text tokens of the vocabulary.

I am leaning toward needing this restriction and will put that restriction in the image-caption task and test it out

bhavul commented 1 year ago

This was sorted out already, as discussed in chat with you @henryj18. So closing this one.