NVIDIA / Megatron-LM

Ongoing research training transformer models at scale
https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/index.html#quick-start
Other
10.13k stars 2.28k forks source link

[bugfix] Fix the incorrect with-statement #972

Open aaa123git opened 1 month ago

aaa123git commented 1 month ago

You may want to enter multiple context managers in one with-statement. However, with rng_context and fp8_context is equivalent to with fp8_context, because rng_context doesn't rewrite __and__ method. The correct way is with rng_context, fp8_context: .

Ref: https://docs.python.org/3/reference/compound_stmts.html#the-with-statement