search

akash-network / awesome-akash

Awesome List of Akash Deployment Examples
298 stars 217 forks source link

Implemented image for grok w/ weights and deployer. #506

Closed idiom-bytes closed 3 months ago

idiom-bytes commented 3 months ago

Please be aware of potential costs when deploying this.

This deployment sets up all dependencies for grok-1, downloads the weights, and runs the model.

Weights are ~296GB in size and take ~10h+ to download via torrent (aria2c). Due to DockerHub limitations in dockerfile layer size, we cannot build the weights into the image and must do this at entrypoint.sh.

Model is 314B int8 params, so 314GB of mem is required to load the model.

Approximately 630GB of VRAM is required to run the model, this is equivalent to ~8 H100s (80GB).

gosuri commented 3 months ago

Reviewing. Thank you.

idiom-bytes commented 3 months ago

Cordially closing this PR. Congrats @yusufpraditya