search

NVIDIA / apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
BSD 3-Clause "New" or "Revised" License
8.16k stars 1.35k forks source link

memory format option is only supported by strided tensors #1776

Closed Cheny1m closed 4 months ago

Cheny1m commented 4 months ago

Describe the Bug When I run the program to apex/normalization/fused_layer_norm.py I get the error memory format option is only supported by strided tensors. Minimal Steps/Code to Reproduce the Bug

I then checked the properties of the incoming tensor： ![image](https://github.com/NVIDIA/apex/assets/65207305/4f04579d-031d-4829-897c-cf940625933a) ![image](https://github.com/NVIDIA/apex/assets/65207305/8d850f83-aa3d-4dca-9ea7-6806619fafd8) All tensors are strided compliant, I don't understand why this error is reported. **Expected Behavior**

This is the stack error message that appears when an error is reported:

Traceback (most recent call last):
Traceback (most recent call last):
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 356, in <module>
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 356, in <module>
    pretrain(train_valid_test_datasets_provider,
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/training.py", line 226, in pretrain
    pretrain(train_valid_test_datasets_provider,
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/training.py", line 226, in pretrain
    iteration = train(forward_step_func,
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/training.py", line 1203, in train
    iteration = train(forward_step_func,
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/training.py", line 1203, in train
    train_step(forward_step_func,
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/training.py", line 694, in train_step
        train_step(forward_step_func,losses_reduced = forward_backward_func(

  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/training.py", line 694, in train_step
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/core/pipeline_parallel/schedules.py", line 358, in forward_backward_no_pipelining
    losses_reduced = forward_backward_func(
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/core/pipeline_parallel/schedules.py", line 358, in forward_backward_no_pipelining
        output_tensor = forward_step(forward_step_func, data_iterator, model, num_microbatches,output_tensor = forward_step(forward_step_func, data_iterator, model, num_microbatches,

  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/core/pipeline_parallel/schedules.py", line 199, in forward_step
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/core/pipeline_parallel/schedules.py", line 199, in forward_step
    output_tensor, loss_func = forward_step_func(data_iterator, model)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 282, in forward_step
    output_tensor, loss_func = forward_step_func(data_iterator, model)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 282, in forward_step
    output_tensor, other_losses = model(tokens, position_ids, attention_mask,
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    output_tensor, other_losses = model(tokens, position_ids, attention_mask,
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1842, in forward
    return forward_call(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1842, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    loss = self.module(*inputs, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/gpt_model.py", line 125, in forward
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/gpt_model.py", line 125, in forward
    lm_output, moe_losses = self.language_model(    
lm_output, moe_losses = self.language_model(  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl

  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
      File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/language_model.py", line 552, in forward
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/language_model.py", line 552, in forward
        encoder_output, *encoder_moe_losses = self.encoder(encoder_output, *encoder_moe_losses = self.encoder(

  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)return self._call_impl(*args, **kwargs)

  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
        return forward_call(*args, **kwargs)return forward_call(*args, **kwargs)

  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1907, in forward
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1907, in forward
        hidden_states, moe_losses = self._checkpointed_forward(hidden_states,hidden_states, moe_losses = self._checkpointed_forward(hidden_states,

  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1740, in _checkpointed_forward
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1740, in _checkpointed_forward
    hidden_states, *local_moe_losses = tensor_parallel.checkpoint(
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/core/tensor_parallel/random.py", line 327, in checkpoint
    hidden_states, *local_moe_losses = tensor_parallel.checkpoint(
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/core/tensor_parallel/random.py", line 327, in checkpoint
    return deepspeed.checkpointing.checkpoint(function, *args)    
return deepspeed.checkpointing.checkpoint(function, *args)  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 994, in checkpoint

  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 994, in checkpoint
    CheckpointFunction.apply(function, all_outputs, *args)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/autograd/function.py", line 553, in apply
    CheckpointFunction.apply(function, all_outputs, *args)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/autograd/function.py", line 553, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 565, in forward
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 565, in forward
    outputs = run_function(*inputs_cuda)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1724, in custom_forward
    outputs = run_function(*inputs_cuda)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1724, in custom_forward
    output = layer(x_, *args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    output = layer(x_, *args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1213, in forward
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/transformer.py", line 1213, in forward
    layernorm_output = self.input_layernorm(hidden_states)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    layernorm_output = self.input_layernorm(hidden_states)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/fused_layer_norm.py", line 92, in forward
    return forward_call(*args, **kwargs)
  File "/data01/cym/Project/MoE/Megatron-DeepSpeed/Megatron-DeepSpeed/megatron/model/fused_layer_norm.py", line 92, in forward
        return FusedLayerNormAffineFunction.apply(input, weight, self.bias, self.normalized_shape, self.eps, self.mem_efficient_ln)return FusedLayerNormAffineFunction.apply(input, weight, self.bias, self.normalized_shape, self.eps, self.mem_efficient_ln)

  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/autograd/function.py", line 553, in apply
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/torch/autograd/function.py", line 553, in apply
        return super().apply(*args, **kwargs)  # type: ignore[misc]return super().apply(*args, **kwargs)  # type: ignore[misc]

  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/apex/normalization/fused_layer_norm.py", line 61, in forward
  File "/home/upa1/.conda/envs/dsmoe/lib/python3.9/site-packages/apex/normalization/fused_layer_norm.py", line 61, in forward
        output, mean, invvar = fused_layer_norm_cuda.forward_affine(output, mean, invvar = fused_layer_norm_cuda.forward_affine(

RuntimeErrorRuntimeError: : memory format option is only supported by strided tensorsmemory format option is only supported by strided tensors

How should I solve this problem?

Environment

OS:Linux Ubuntu 18.04.5 LTS GPU:2 x NVIDIA V100 Python3.9 (conda) CUDA11.8 Pytorch2.2.0 APEX: via C++/CUDA

Cheny1m commented 4 months ago

This issue has been resolved by me, and I suspect it was caused by a series of compatibility issues with the system, python, and various expansion packages. Initially I used pytorch, Megatron-DeepSpeed and Transformer primarily for MOE model training and inference. The initial installation of APEX was a source install, which eventually led to the problem. After that I used the container installation method recommended by APEX and successfully skipped the issue.

More complete details you can find below（If you're familiar with docker, you can skip this part）：

[pull docker]

docker pull nvcr.io/nvidia/pytorch:24.01-py3

[run container]

docker run --gpus all --name xxx -itd -v /dev/shm:/dev/shm -v /xxx/your/Project/:/workspace nvcr.io/nvidia/pytorch:24.01-py3 /bin/bash

After that you can install your own dependencies and run the program!