Model parallelism to use a huge huge model in colab

[YeonwooSung]

Would it be possible to use DeBERTa v2 xlarge on colab notebook?

Model parallelism tutorial notebook Huggingface Transformers issue - Model parallelism and big models Huggingface Accelerate Huggingface Deepspeed integration

단일 머신을 사용한 모델 병렬화 모범사례 단일 머신을 사용한 모델 병렬화 모범사례 - colab notebook

[YeonwooSung]

Megatron also used Deepspeed for distributed training

link 1 link 2

[YeonwooSung]

Using TPU on Colab

PyTorch XLA

[YeonwooSung]

블로그 - Deepspeed with Huggingface for training BERT

[YeonwooSung]

Multi GPU training on Colab is only available by using either connecting Google Cloud Platform or local machine with multiple GPUs