Dose Context Parallel support Packing Inputs Without Cross-Contamination Attention?

NVIDIA / Megatron-LM

Ongoing research training transformer models at scale

Other

10.69k stars 2.39k forks source link

Open Lzhang-hub opened 2 months ago

Lzhang-hub commented 2 months ago

For long seq model train,I want ues both Context Parallel and packing Inputs without cross-contamination attention link , Dose is support?

cross-contamination attention like:

github-actions[bot] commented 2 weeks ago

Marking as stale. No activity in 60 days.