0 is a valid token (a note to potentially improve masking method?)

lucidrains / DALLE2-pytorch

Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch

MIT License

11.03k stars 1.07k forks source link

0 is a valid token (a note to potentially improve masking method?) #207

Closed nousr closed 2 years ago

nousr commented 2 years ago

So, after yesterday I was reminded of some experiments I had done with kat a few months ago.

Basically, the 0 token is ! with no word boundary, There are a few common cases where this crops up I gave one common example above.

I believe the current method for masking looks at all 0 values and marks them, however, the ideal method would be to find the 0 tokens after the <eos> token and only mask those.

lucidrains commented 2 years ago

@nousr this should address it! https://github.com/lucidrains/DALLE2-pytorch/commit/f9882077188e9846ebf67bd78b2f8621754232ab