search

turtlesoupy / this-word-does-not-exist

This Word Does Not Exist
https://www.thisworddoesnotexist.com
MIT License
1.02k stars 82 forks source link

Questions regarding a "GPT-3" update #37

Closed DonaldTsang closed 2 years ago

DonaldTsang commented 4 years ago

Would it be useful to update this project with a better language model, or is that redundant?

xynydev commented 3 years ago

I don't think this would need a better one, you can't retrain GPT-3 either. But if there would be a need or want for another model I'd suggest https://github.com/EleutherAI/gpt-neo/

DonaldTsang commented 3 years ago

Fairs, but can this system accept GPT-NEO tho

xynydev commented 3 years ago

Fairs, but can this system accept GPT-NEO tho

What do you mean by that?

Also apparently eleuther has a 6B model too https://github.com/kingoflolz/mesh-transformer-jax/.

BradKML commented 2 years ago

GPT-NEO and other FOSS model exists, but how does one "mill" them into a definition generator?

turtlesoupy commented 2 years ago

You'll almost certainly need to fine tune them on a dataset... my dataset was a normalized form of the Apple dictionaries shipped on mac -- the code is here: https://github.com/turtlesoupy/this-word-does-not-exist/blob/master/title_maker_pro/dictionary_definition.py

Basically I fed a definition at a time with sequences like <word_def>word<definition>an interesting word<example>the word is right<end>

turtlesoupy commented 2 years ago

Going to close for now!

BradKML commented 2 years ago

@turtlesoupy thanks for the conversation! The potential is there to revamp this with multiple dictionaries.