Open pure-rgb opened 1 year ago
Official pytorch: https://github.com/EleutherAI/gpt-neox Unofficial jax: https://github.com/kingoflolz/mesh-transformer-jax
+1
This is the open source version of GPT-3, right?
yes, as far as I know. @abheesht17 wdyt? Is keras nlp ready to support this type of model? It's a very big model and unlikely to train for all.
Minimal code pytorch and flax
Some reading.
People use jax/flax to leverage TPU:
GPT-NeoX
Official pytorch: https://github.com/EleutherAI/gpt-neox Unofficial jax: https://github.com/kingoflolz/mesh-transformer-jax