search

cfoster0 / CLAP

Contrastive Language-Audio Pretraining
BSD 3-Clause "New" or "Revised" License
87 stars 4 forks source link

Add Pod Orchestration #7

Closed cfoster0 closed 3 years ago

cfoster0 commented 3 years ago

https://github.com/kingoflolz/mesh-transformer-jax

Use the pod orchestation code from here. Effectively, we should borrow everything and modify the transformer_shard and tfrecord_loader files.

cfoster0 commented 3 years ago

For small (<< 1B parameter) models, it was decided that model parallelism isn't needed. Closing for now.