[BUG]A tremendous amount of errors when trying to install deepspeed with DS_BUILD_OPS=1

[flandore]

hai

If I do pip install deepspeed, it installs. But when I try to run it, the code itself works, but the deepspeed efficiencies, etc. don't seem to work. I also get this message at the end

/home/ub_flan/bert/bert/bin/python3: Error while finding module specification for 'deepspeed.launcher.launch' (ModuleNotFoundError: No module named 'deepspeed.launcher'; 'deepspeed' is not a package) I assume this is happening because the "installed" field is set to NO

I added DS_BUILD_OPS=1 to make the installed field [YES]. DS_BUILD_OPS=1 gives a lot of errors

this is (DS_BUILD_OPS=1 pip install deepspeed) error

  Using cached deepspeed-0.7.7.tar.gz (712 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: hjson in ./bert/lib/python3.10/site-packages (from deepspeed) (3.1.0)
Requirement already satisfied: ninja in ./bert/lib/python3.10/site-packages (from deepspeed) (1.11.1)
Requirement already satisfied: numpy in ./bert/lib/python3.10/site-packages (from deepspeed) (1.24.1)
Requirement already satisfied: packaging in ./bert/lib/python3.10/site-packages (from deepspeed) (23.0)
Requirement already satisfied: psutil in ./bert/lib/python3.10/site-packages (from deepspeed) (5.9.4)
Requirement already satisfied: py-cpuinfo in ./bert/lib/python3.10/site-packages (from deepspeed) (9.0.0)
Requirement already satisfied: pydantic in ./bert/lib/python3.10/site-packages (from deepspeed) (1.10.4)
Requirement already satisfied: torch in ./bert/lib/python3.10/site-packages (from deepspeed) (1.13.1+cu116)
Requirement already satisfied: tqdm in ./bert/lib/python3.10/site-packages (from deepspeed) (4.64.1)
Requirement already satisfied: typing-extensions>=4.2.0 in ./bert/lib/python3.10/site-packages (from pydantic->deepspeed) (4.4.0)
Using legacy 'setup.py install' for deepspeed, since package 'wheel' is not installed.
Installing collected packages: deepspeed
  Running setup.py install for deepspeed ... error
  error: subprocess-exited-with-error

  × Running setup.py install for deepspeed did not run successfully.
  │ exit code: 1
  ╰─> [487 lines of output]
      DS_BUILD_OPS=1
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination
      Install Ops={'cpu_adam': 1, 'cpu_adagrad': 1, 'fused_adam': 1, 'fused_lamb': 1, 'sparse_attn': 1, 'transformer': 1, 'stochastic_transformer': 1, 'async_io': 1, 'utils': 1, 'quantizer': 1, 'transformer_inference': 1, 'spatial_inference': 1}
      fatal: not a git repository (or any of the parent directories): .git
      version=0.7.7, git_hash=unknown, git_branch=unknown
      install_requires=['hjson', 'ninja', 'numpy', 'packaging', 'psutil', 'py-cpuinfo', 'pydantic', 'torch', 'tqdm']
      compatible_ops={'cpu_adam': True, 'cpu_adagrad': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': True, 'transformer': True, 'stochastic_transformer': True, 'async_io': True, 'utils': True, 'quantizer': True, 'transformer_inference': True, 'spatial_inference': True}
      ext_modules=[<setuptools.extension.Extension('deepspeed.ops.adam.cpu_adam_op') at 0x7ff4dc607b80>, <setuptools.extension.Extension('deepspeed.ops.adagrad.cpu_adagrad_op') at 0x7ff590c91e10>, <setuptools.extension.Extension('deepspeed.ops.adam.fused_adam_op') at 0x7ff4dbb4abc0>, <setuptools.extension.Extension('deepspeed.ops.lamb.fused_lamb_op') at 0x7ff4dbb4abf0>, <setuptools.extension.Extension('deepspeed.ops.sparse_attention.sparse_attn_op') at 0x7ff4dbb4ab90>, <setuptools.extension.Extension('deepspeed.ops.transformer.transformer_op') at 0x7ff4dbb4ac50>, <setuptools.extension.Extension('deepspeed.ops.transformer.stochastic_transformer_op') at 0x7ff4dbb4ab60>, <setuptools.extension.Extension('deepspeed.ops.aio.async_io_op') at 0x7ff4dbb4b7f0>, <setuptools.extension.Extension('deepspeed.ops.utils_op') at 0x7ff4dbb024d0>, <setuptools.extension.Extension('deepspeed.ops.quantizer.quantizer_op') at 0x7ff4dbb031c0>, <setuptools.extension.Extension('deepspeed.ops.transformer.inference.transformer_inference_op') at 0x7ff4dbb03430>, <setuptools.extension.Extension('deepspeed.ops.spatial.spatial_inference_op') at 0x7ff4dbb033a0>]
      running install
      /home/ub_flan/bert/bert/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.10
      creating build/lib.linux-x86_64-3.10/deepspeed
      copying deepspeed/env_report.py -> build/lib.linux-x86_64-3.10/deepspeed
      copying deepspeed/git_version_info_installed.py -> build/lib.linux-x86_64-3.10/deepspeed
      copying deepspeed/constants.py -> build/lib.linux-x86_64-3.10/deepspeed
      copying deepspeed/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed
      copying deepspeed/git_version_info.py -> build/lib.linux-x86_64-3.10/deepspeed
      creating build/lib.linux-x86_64-3.10/benchmarks
      copying benchmarks/__init__.py -> build/lib.linux-x86_64-3.10/benchmarks
      creating build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/tensorboard.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/monitor.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/csv_monitor.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/config.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/wandb.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      copying deepspeed/monitor/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/monitor
      creating build/lib.linux-x86_64-3.10/deepspeed/inference
      copying deepspeed/inference/engine.py -> build/lib.linux-x86_64-3.10/deepspeed/inference
      copying deepspeed/inference/config.py -> build/lib.linux-x86_64-3.10/deepspeed/inference
      copying deepspeed/inference/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/inference
      creating build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/reshape_meg_2d.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/universal_checkpoint.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/reshape_utils.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/deepspeed_checkpoint.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/reshape_3d_utils.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/zero_checkpoint.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      copying deepspeed/checkpoint/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/checkpoint
      creating build/lib.linux-x86_64-3.10/deepspeed/launcher
      copying deepspeed/launcher/multinode_runner.py -> build/lib.linux-x86_64-3.10/deepspeed/launcher
      copying deepspeed/launcher/launch.py -> build/lib.linux-x86_64-3.10/deepspeed/launcher
      copying deepspeed/launcher/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/launcher
      copying deepspeed/launcher/runner.py -> build/lib.linux-x86_64-3.10/deepspeed/launcher
      copying deepspeed/launcher/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/launcher
      creating build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/inject.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/module_quantize.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/layers.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/replace_policy.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/load_checkpoint.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      copying deepspeed/module_inject/replace_module.py -> build/lib.linux-x86_64-3.10/deepspeed/module_inject
      creating build/lib.linux-x86_64-3.10/deepspeed/moe
      copying deepspeed/moe/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/moe
      copying deepspeed/moe/sharded_moe.py -> build/lib.linux-x86_64-3.10/deepspeed/moe
      copying deepspeed/moe/experts.py -> build/lib.linux-x86_64-3.10/deepspeed/moe
      copying deepspeed/moe/layer.py -> build/lib.linux-x86_64-3.10/deepspeed/moe
      copying deepspeed/moe/mappings.py -> build/lib.linux-x86_64-3.10/deepspeed/moe
      copying deepspeed/moe/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/moe
      creating build/lib.linux-x86_64-3.10/deepspeed/model_implementations
      copying deepspeed/model_implementations/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations
      creating build/lib.linux-x86_64-3.10/deepspeed/elasticity
      copying deepspeed/elasticity/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/elasticity
      copying deepspeed/elasticity/elastic_agent.py -> build/lib.linux-x86_64-3.10/deepspeed/elasticity
      copying deepspeed/elasticity/elasticity.py -> build/lib.linux-x86_64-3.10/deepspeed/elasticity
      copying deepspeed/elasticity/config.py -> build/lib.linux-x86_64-3.10/deepspeed/elasticity
      copying deepspeed/elasticity/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/elasticity
      copying deepspeed/elasticity/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/elasticity
      creating build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/timer.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/tensor_fragment.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/mixed_precision_linkage.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/comms_logging.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/debug.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/exceptions.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/zero_to_fp32.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/nvtx.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/init_on_device.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/types.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/logging.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/groups.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      copying deepspeed/utils/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/utils
      creating build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/config.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/comm.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/backend.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/torch.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      copying deepspeed/comm/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/comm
      creating build/lib.linux-x86_64-3.10/deepspeed/ops
      copying deepspeed/ops/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops
      creating build/lib.linux-x86_64-3.10/deepspeed/profiling
      copying deepspeed/profiling/config.py -> build/lib.linux-x86_64-3.10/deepspeed/profiling
      copying deepspeed/profiling/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/profiling
      copying deepspeed/profiling/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/profiling
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/lr_schedules.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/engine.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/config.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/state_dict_factory.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/config_utils.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/eigenvalue.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/sparse_tensor.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/quantize.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/progressive_layer_drop.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/dataloader.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/weight_quantizer.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      copying deepspeed/runtime/bf16_optimizer.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime
      creating build/lib.linux-x86_64-3.10/deepspeed/nebula
      copying deepspeed/nebula/config.py -> build/lib.linux-x86_64-3.10/deepspeed/nebula
      copying deepspeed/nebula/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/nebula
      copying deepspeed/nebula/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/nebula
      creating build/lib.linux-x86_64-3.10/deepspeed/pipe
      copying deepspeed/pipe/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/pipe
      creating build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/basic_layer.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/config.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/scheduler.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/helper.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      copying deepspeed/compression/compress.py -> build/lib.linux-x86_64-3.10/deepspeed/compression
      creating build/lib.linux-x86_64-3.10/deepspeed/accelerator
      copying deepspeed/accelerator/cuda_accelerator.py -> build/lib.linux-x86_64-3.10/deepspeed/accelerator
      copying deepspeed/accelerator/real_accelerator.py -> build/lib.linux-x86_64-3.10/deepspeed/accelerator
      copying deepspeed/accelerator/abstract_accelerator.py -> build/lib.linux-x86_64-3.10/deepspeed/accelerator
      copying deepspeed/accelerator/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/accelerator
      creating build/lib.linux-x86_64-3.10/deepspeed/autotuning
      copying deepspeed/autotuning/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning
      copying deepspeed/autotuning/config.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning
      copying deepspeed/autotuning/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning
      copying deepspeed/autotuning/autotuner.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning
      copying deepspeed/autotuning/scheduler.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning
      copying deepspeed/autotuning/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning
      creating build/lib.linux-x86_64-3.10/deepspeed/model_implementations/transformers
      copying deepspeed/model_implementations/transformers/clip_encoder.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations/transformers
      copying deepspeed/model_implementations/transformers/ds_transformer.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations/transformers
      copying deepspeed/model_implementations/transformers/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations/transformers
      creating build/lib.linux-x86_64-3.10/deepspeed/model_implementations/diffusers
      copying deepspeed/model_implementations/diffusers/vae.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations/diffusers
      copying deepspeed/model_implementations/diffusers/unet.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations/diffusers
      copying deepspeed/model_implementations/diffusers/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/model_implementations/diffusers
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/softmax.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/sparse_attention_utils.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/sparse_self_attention.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/sparsity_config.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/bert_sparse_self_attention.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      copying deepspeed/ops/sparse_attention/matmul.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/quantizer.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/fused_adam.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/fused_lamb.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/builder_names.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/all_ops.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/async_io.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/cpu_adam.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/spatial_inference.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/transformer.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/transformer_inference.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/cpu_adagrad.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/builder.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/stochastic_transformer.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/sparse_attn.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      copying deepspeed/ops/op_builder/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/op_builder
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/adam
      copying deepspeed/ops/adam/fused_adam.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/adam
      copying deepspeed/ops/adam/cpu_adam.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/adam
      copying deepspeed/ops/adam/multi_tensor_apply.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/adam
      copying deepspeed/ops/adam/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/adam
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/transformer
      copying deepspeed/ops/transformer/transformer.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer
      copying deepspeed/ops/transformer/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/adagrad
      copying deepspeed/ops/adagrad/cpu_adagrad.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/adagrad
      copying deepspeed/ops/adagrad/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/adagrad
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/aio
      copying deepspeed/ops/aio/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/aio
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/quantizer
      copying deepspeed/ops/quantizer/quantizer.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/quantizer
      copying deepspeed/ops/quantizer/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/quantizer
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/lamb
      copying deepspeed/ops/lamb/fused_lamb.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/lamb
      copying deepspeed/ops/lamb/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/lamb
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention/trsrc
      copying deepspeed/ops/sparse_attention/trsrc/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention/trsrc
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/ds_attention.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/diffusers_transformer_block.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/ds_mlp.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/triton_ops.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/config.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/bias_add.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/diffusers_attention.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/moe_inference.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/diffusers_2d_transformer.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      copying deepspeed/ops/transformer/inference/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/ops/transformer/inference
      creating build/lib.linux-x86_64-3.10/deepspeed/profiling/flops_profiler
      copying deepspeed/profiling/flops_profiler/profiler.py -> build/lib.linux-x86_64-3.10/deepspeed/profiling/flops_profiler
      copying deepspeed/profiling/flops_profiler/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/profiling/flops_profiler
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/unfused_optimizer.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/loss_scaler.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16
      copying deepspeed/runtime/fp16/fused_optimizer.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/activation_checkpointing
      copying deepspeed/runtime/activation_checkpointing/config.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/activation_checkpointing
      copying deepspeed/runtime/activation_checkpointing/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/activation_checkpointing
      copying deepspeed/runtime/activation_checkpointing/checkpointing.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/activation_checkpointing
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/offload_config.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/config.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/partition_parameters.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/contiguous_memory_allocator.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/stage_1_and_2.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/parameter_offload.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/partitioned_param_coordinator.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/test.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/linear.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/tiling.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      copying deepspeed/runtime/zero/stage3.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/zero
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/data_pipeline
      copying deepspeed/runtime/data_pipeline/curriculum_scheduler.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/data_pipeline
      copying deepspeed/runtime/data_pipeline/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/data_pipeline
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/nccl.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/coalesced_collectives.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/mpi.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/comm
      copying deepspeed/runtime/comm/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/comm
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/partitioned_optimizer_swapper.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/aio_config.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/async_swapper.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/optimizer_utils.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/partitioned_param_swapper.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/constants.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/pipelined_optimizer_swapper.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      copying deepspeed/runtime/swap_tensor/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/swap_tensor
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/module.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/engine.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/p2p.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/schedule.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      copying deepspeed/runtime/pipe/topology.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/pipe
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/compression
      copying deepspeed/runtime/compression/cupy.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/compression
      copying deepspeed/runtime/compression/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/compression
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/checkpoint_engine
      copying deepspeed/runtime/checkpoint_engine/nebula_checkpoint_engine.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/checkpoint_engine
      copying deepspeed/runtime/checkpoint_engine/torch_checkpoint_engine.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/checkpoint_engine
      copying deepspeed/runtime/checkpoint_engine/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/checkpoint_engine
      copying deepspeed/runtime/checkpoint_engine/checkpoint_engine.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/checkpoint_engine
      creating build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/zoadam.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/lamb.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16/onebit
      copying deepspeed/runtime/fp16/onebit/adam.py -> build/lib.linux-x86_64-3.10/deepspeed/runtime/fp16/onebit
      creating build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/utils.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/cost_model.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/index_based_tuner.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/base_tuner.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/model_based_tuner.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      copying deepspeed/autotuning/tuner/__init__.py -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/tuner
      creating build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/all_reduce.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/utils.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/pt2pt.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/all_gather.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/run_all.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/all_to_all.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/constants.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/broadcast.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      copying benchmarks/communication/__init__.py -> build/lib.linux-x86_64-3.10/benchmarks/communication
      running egg_info
      writing deepspeed.egg-info/PKG-INFO
      writing dependency_links to deepspeed.egg-info/dependency_links.txt
      writing entry points to deepspeed.egg-info/entry_points.txt
      writing requirements to deepspeed.egg-info/requires.txt
      writing top-level names to deepspeed.egg-info/top_level.txt
      reading manifest file 'deepspeed.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.hip' under directory 'deepspeed'
      warning: no files found matching '*.cc' under directory 'deepspeed'
      warning: no files found matching '*.tr' under directory 'csrc'
      warning: no files found matching '*.cc' under directory 'csrc'
      adding license file 'LICENSE'
      writing manifest file 'deepspeed.egg-info/SOURCES.txt'
      creating build/lib.linux-x86_64-3.10/benchmarks/inference
      copying benchmarks/inference/bert-bench.py -> build/lib.linux-x86_64-3.10/benchmarks/inference
      copying benchmarks/inference/collect_results.py -> build/lib.linux-x86_64-3.10/benchmarks/inference
      copying benchmarks/inference/gpt-bench.py -> build/lib.linux-x86_64-3.10/benchmarks/inference
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adagrad
      copying deepspeed/ops/csrc/adagrad/cpu_adagrad.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adagrad
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adam
      copying deepspeed/ops/csrc/adam/cpu_adam.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adam
      copying deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adam
      copying deepspeed/ops/csrc/adam/multi_tensor_adam.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adam
      copying deepspeed/ops/csrc/adam/multi_tensor_apply.cuh -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/adam
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      copying deepspeed/ops/csrc/aio/common/deepspeed_aio_common.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      copying deepspeed/ops/csrc/aio/common/deepspeed_aio_common.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      copying deepspeed/ops/csrc/aio/common/deepspeed_aio_types.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      copying deepspeed/ops/csrc/aio/common/deepspeed_aio_types.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      copying deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      copying deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/common
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      copying deepspeed/ops/csrc/aio/py_lib/py_ds_aio.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_lib
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_test
      copying deepspeed/ops/csrc/aio/py_test/single_process_config.json -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/aio/py_test
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/common
      copying deepspeed/ops/csrc/common/custom_cuda_kernel.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/common
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/StopWatch.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/Timer.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/compat.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/context.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/conversion_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/cpu_adagrad.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/cpu_adam.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/cublas_wrappers.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/custom_cuda_layers.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/dequantization_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/dropout.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/ds_kernel_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/ds_transformer_cuda.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/feed_forward.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/gelu.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/gemm_test.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/general_kernels.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/memory_access_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/normalize_layer.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/quantization.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/quantization_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/quantizer.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/reduction_utils.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/simd.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/softmax.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/strided_batch_gemm.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      copying deepspeed/ops/csrc/includes/type_shim.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/includes
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/lamb
      copying deepspeed/ops/csrc/lamb/fused_lamb_cuda.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/lamb
      copying deepspeed/ops/csrc/lamb/fused_lamb_cuda_kernel.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/lamb
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/quantization
      copying deepspeed/ops/csrc/quantization/dequantize.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/quantization
      copying deepspeed/ops/csrc/quantization/fake_quantizer.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/quantization
      copying deepspeed/ops/csrc/quantization/pt_binding.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/quantization
      copying deepspeed/ops/csrc/quantization/quantize.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/quantization
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/sparse_attention
      copying deepspeed/ops/csrc/sparse_attention/utils.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/sparse_attention
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/spatial
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/spatial/csrc
      copying deepspeed/ops/csrc/spatial/csrc/opt_bias_add.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/spatial/csrc
      copying deepspeed/ops/csrc/spatial/csrc/pt_binding.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/spatial/csrc
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/spatial/includes
      copying deepspeed/ops/csrc/spatial/includes/spatial_cuda_layers.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/spatial/includes
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/cublas_wrappers.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/dropout_kernels.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/ds_transformer_cuda.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/gelu_kernels.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/general_kernels.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/normalize_kernels.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/softmax_kernels.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      copying deepspeed/ops/csrc/transformer/transform_kernels.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/apply_rotary_pos_emb.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/dequantize.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/layer_norm.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/relu.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/softmax.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      copying deepspeed/ops/csrc/transformer/inference/csrc/transform.cu -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/csrc
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/includes
      copying deepspeed/ops/csrc/transformer/inference/includes/inference_context.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/includes
      copying deepspeed/ops/csrc/transformer/inference/includes/inference_cublas_wrappers.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/includes
      copying deepspeed/ops/csrc/transformer/inference/includes/inference_cuda_layers.h -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/transformer/inference/includes
      creating build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/utils
      copying deepspeed/ops/csrc/utils/flatten_unflatten.cpp -> build/lib.linux-x86_64-3.10/deepspeed/ops/csrc/utils
      creating build/lib.linux-x86_64-3.10/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero0.json -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero1.json -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero2.json -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/config_templates
      copying deepspeed/autotuning/config_templates/template_zero3.json -> build/lib.linux-x86_64-3.10/deepspeed/autotuning/config_templates
      copying deepspeed/ops/sparse_attention/trsrc/matmul.tr -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention/trsrc
      copying deepspeed/ops/sparse_attention/trsrc/softmax_bwd.tr -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention/trsrc
      copying deepspeed/ops/sparse_attention/trsrc/softmax_fwd.tr -> build/lib.linux-x86_64-3.10/deepspeed/ops/sparse_attention/trsrc
      running build_ext
      /home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/utils/cpp_extension.py:387: UserWarning: The detected CUDA version (11.5) has a minor version mismatch with the version that was used to compile PyTorch (11.6). Most likely this shouldn't be a problem.
        warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
      building 'deepspeed.ops.adam.cpu_adam_op' extension
      creating build/temp.linux-x86_64-3.10
      creating build/temp.linux-x86_64-3.10/csrc
      creating build/temp.linux-x86_64-3.10/csrc/adam
      creating build/temp.linux-x86_64-3.10/csrc/common
      x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Icsrc/includes -I/include -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include/TH -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include/THC -I/include -I/home/ub_flan/bert/bert/include -I/usr/include/python3.10 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-3.10/csrc/adam/cpu_adam.o -O3 -std=c++14 -g -Wno-reorder -L/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
      In file included from csrc/includes/cpu_adam.h:12,
                       from csrc/adam/cpu_adam.cpp:1:
      csrc/includes/simd.h:63: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
         63 | #pragma unroll
            |
      csrc/includes/simd.h:70: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
         70 | #pragma unroll
            |
      csrc/includes/simd.h:76: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
         76 | #pragma unroll
            |
      csrc/includes/simd.h:84: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
         84 | #pragma unroll
            |
      csrc/includes/simd.h:92: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
         92 | #pragma unroll
            |
      csrc/includes/simd.h:100: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
        100 | #pragma unroll
            |
      csrc/includes/simd.h:106: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
        106 | #pragma unroll
            |
      csrc/includes/simd.h:112: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
        112 | #pragma unroll
            |
      csrc/includes/simd.h:118: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
        118 | #pragma unroll
            |
      csrc/includes/simd.h:124: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
        124 | #pragma unroll
            |
      csrc/includes/simd.h:130: warning: ignoring ‘#pragma unroll ’ [-Wunknown-pragmas]
        130 | #pragma unroll
            |
      csrc/adam/cpu_adam.cpp: In member function ‘void Adam_Optimizer::Step_1(float*, float*, float*, float*, size_t, __half*, bool)’:
      csrc/adam/cpu_adam.cpp:45:17: warning: ‘params_cast_h’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         45 |         __half* params_cast_h;
            |                 ^~~~~~~~~~~~~
      csrc/adam/cpu_adam.cpp:44:17: warning: ‘grads_cast_h’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         44 |         __half* grads_cast_h;
            |                 ^~~~~~~~~~~~
      /bin/nvcc -Icsrc/includes -I/include -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include/TH -I/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch/include/THC -I/include -I/home/ub_flan/bert/bert/include -I/usr/include/python3.10 -c csrc/common/custom_cuda_kernel.cu -o build/temp.linux-x86_64-3.10/csrc/common/custom_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
      /usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
        435 |         function(_Functor&& __f)
            |                                                                                                                                                 ^
      /usr/include/c++/11/bits/std_function.h:435:145: note:         ‘_ArgTypes’
      /usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
        530 |         operator=(_Functor&& __f)
            |                                                                                                                                                  ^
      /usr/include/c++/11/bits/std_function.h:530:146: note:         ‘_ArgTypes’
      error: command '/bin/nvcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> deepspeed

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

this is my env

$ ds_report(that I typed in the case of pip install deepspeed)

DeepSpeed C++/CUDA extension op report

NOTE: Ops not installed will be just-in-time (JIT) compiled at runtime if needed. Op compatibility means that your system meet the required dependencies to JIT install the op.

JIT compiled ops requires ninja ninja .................. [OKAY]

op name ................ installed .. compatible

cpu_adam ............... [NO] ....... [OKAY] cpu_adagrad ............ [NO] ....... [OKAY] fused_adam ............. [NO] ....... [OKAY] fused_lamb ............. [NO] ....... [OKAY] sparse_attn ............ [NO] ....... [OKAY] transformer ............ [NO] ....... [OKAY] stochastic_transformer . [NO] ....... [OKAY] async_io ............... [NO] ....... [OKAY] utils .................. [NO] ....... [OKAY] quantizer .............. [NO] ....... [OKAY] transformer_inference .. [NO] ....... [OKAY] spatial_inference ...... [NO] ....... [OKAY]

DeepSpeed general environment info: torch install path ............... ['/home/ub_flan/bert/bert/lib/python3.10/site-packages/torch'] torch version .................... 1.13.1+cu116 torch cuda version ............... 11.6 torch hip version ................ None nvcc version ..................... 11.5 deepspeed install path ........... ['/home/ub_flan/bert/bert/lib/python3.10/site-packages/deepspeed'] deepspeed info ................... 0.7.7, unknown, unknown deepspeed wheel compiled w. ...... torch 1.13, cuda 11.6

os ubuntu

$nvidia-smi Mon Jan 16 12:21:56 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.01 Driver Version: 516.94 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:08:00.0 On | N/A | | 40% 29C P8 24W / 250W | 586MiB / 11264MiB | 1% Default | | | | N/A | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+

[ghost]

Suggestion: use triple ticks rather than single tick for multi line monospaced formatting.

[mrwyattii]

@flandore I'm unable to reproduce this issue on my local system. Based on posts about similar nvcc compilation errors (i.e., error: parameter packs not expanded with ‘...’), it may be resolved by upgrading nvcc/cuda. The versions I'm using are:

CUDA Version: 11.8
Driver Version: 520.56.06
Cuda compilation tools, release 11.6, V11.6.124

Additionally, I see a mismatch between torch compiled version and installed CUDA version: Installed CUDA version 11.5 does not match the version torch was compiled with 11.6 but since the APIs are compatible, accepting this combination

This usually isn't a problem, but the output doesn't give any other indication of what the issue could be.

Can you try updating to a more recent version and try again?

[ghost]

@mrwyattii:

• which LLVM and clang versions do you have installed?
• which GCC version do you have installed?
• what are your default compilers? (e.g.: /usr/bin/cc, /usr/bin/c++. clang or GCC?)

I had similar problems when trying to install DeepSpeed.

[mrwyattii]

@augmented-fog using gcc 9.4.0

[ghost]

@augmented-fog using gcc 9.4.0

So, gcc 9.4.0 is your default compiler for C and C++, and you don't have LLVM or clang installed?

[loadams]

@flandore and @augmented-fog - are we safe to close this issue now that it is fairly old and issues appear to be resolved?

[hhhwwwpoi]

可能是文件名称的原因，换个文件名试试，不要使用deepspeed.py