search

NVIDIA / apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch
BSD 3-Clause "New" or "Revised" License
8.2k stars 1.36k forks source link

`pyproject.toml` missing `packaging` dependency #1679

Open calebho opened 1 year ago

calebho commented 1 year ago

Describe the Bug

1669 adds a pyproject.toml file, but the build dependencies are underspecified. The setup.py file depends on packaging but this dependency isn't declared in the build dependencies.

Minimal Steps/Code to Reproduce the Bug

#!/bin/bash

set -euo pipefail
set -x

tmpdir=$(mktemp -d)
conda create -p $tmpdir/env -c pytorch -c nvidia --yes python=3.10 pytorch pytorch-cuda=11.8
conda init bash
conda activate $tmpdir/env
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" git+https://github.com/NVIDIA/apex.git@2d8302a6c12e202f7b40b13a43daa95f326fd0ea

yields 

Traceback (most recent call last):
    File "/tmp/tmp.DhZuKHLF3V/env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/tmp/tmp.DhZuKHLF3V/env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/tmp/tmp.DhZuKHLF3V/env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-x0zi0j0a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/tmp/pip-build-env-x0zi0j0a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-x0zi0j0a/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 4, in <module>
  ModuleNotFoundError: No module named 'packaging'
  error: subprocess-exited-with-error

full log: https://gist.github.com/calebho/35fa3bf2fdc4e818bc5bded4456988c3

Expected Behavior

It should install without errors

Environment

Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.10.11 (main, May 16 2023, 00:28:57) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 470.141.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          80
On-line CPU(s) list:             0-79
Thread(s) per core:              2
Core(s) per socket:              20
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           85
Model name:                      Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz
Stepping:                        7
CPU MHz:                         1708.821
CPU max MHz:                     3900.0000
CPU min MHz:                     800.0000
BogoMIPS:                        4200.00
Virtualization:                  VT-x
L1d cache:                       1.3 MiB
L1i cache:                       1.3 MiB
L2 cache:                        40 MiB
L3 cache:                        55 MiB
NUMA node0 CPU(s):               0-19,40-59
NUMA node1 CPU(s):               20-39,60-79
Vulnerability Itlb multihit:     KVM: Mitigation: Split huge pages
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Mitigation; TSX disabled
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] torch==2.0.1
[conda] No relevant packages
crcrpar commented 1 year ago

that sounds right, would you mind opening a pull request?

loadams commented 1 year ago

@crcrpar - I made one here if you could take a look.

Colezwhy commented 1 year ago

I encountered the same error, when i added packaging and torch to pyproject.toml, a new error occurred:

ERROR: Could not find a version that satisfies the requirement torch (from versions: none) ERROR: No matching distribution found for torch error: subprocess-exited-with-error

my conda env is torch1.7.1, cuda 11.0, and using the 22.04branch for installation.

crcrpar commented 1 year ago

Personally I recommend using --no-build-isolation as even packaging is installed, I guess it would be a bit tricky to install the same torch as in the environment into a build environment.

Colezwhy commented 1 year ago

Personally I recommend using --no-build-isolation as even packaging is installed, I guess it would be a bit tricky to install the same torch as in the environment into a build environment.

Is the latest version of apex necessary?

loadams commented 1 year ago

@Colezwhy and @crcrpar - when I build with --no-build-isolation I don't get the packaging error, but neither the cpp nor cuda_ext are installed. And no warnings for reasons why they weren't either.

Sample build here that installs apex with --no-build-isolation and there are no nvcc lines in the apex install, and later on amp_C cannot be found

crcrpar commented 1 year ago

hmm, I haven't found myself in the same situation. what if the latest pip and multiple --config-settigns?

xman1991 commented 1 year ago

is it firgured out?

ydshieh commented 1 year ago

same issue in transformers CI (during docker image build):

cd apex && MAX_JOBS=1 python3 -m pip install --global-option="--cpp_ext" --global-option="--cuda_ext" --no-cache -v --disable-pip-version-check .
loadams commented 1 year ago

@crcrpar - even with the newer pip the issue doesn't seem to be multiple config settings either. When you tested this did you hit any issues with this?

xwang233 commented 1 year ago

Something like this should work for multiple extensions

pip install -v --no-build-isolation --config-settings --build-option="--cpp_ext --cuda_ext --bnp --xentropy ..." .
loadams commented 1 year ago

Adding torch to the dependencies does help, but the bigger issue seems to be that no matter who I specify the build options, they're not being picked up as being in argv here

loadams commented 1 year ago

@xwang233 - were you able to build the cpp or cuda exts with these commands at all? I've been able to build, but I'm not seeing the arguments passed to setup.py?

xwang233 commented 1 year ago

Yes, I was able to build cpp, cuda, and other extensions with this command. https://github.com/NVIDIA/apex/issues/1679#issuecomment-1602117816

We were using python 3.10.11 and pip 23.0.1

loadams commented 1 year ago

@xwang233 - Are you using the most up to date master branch from the repo? Since you should also need to add packaging and torch as dependencies to the pyproject.toml first, right?

Here is my output:

pip install -v --no-build-isolation --config-settings --build-option="--cpp_ext --cuda_ext" ./
Using pip 23.1.2 from /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex-loadams/unit-test-venv/lib/python3.8/site-packages/pip (python 3.8)
Processing /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex
  Preparing metadata (pyproject.toml) ...   Running command Preparing metadata (pyproject.toml)

  torch.__version__  = 1.13.1+cu116

  running dist_info
  creating /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info
  writing /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/SOURCES.txt'
  adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
  reading manifest file '/tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/SOURCES.txt'
  writing manifest file '/tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-i6iyi61n/apex.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
done
Requirement already satisfied: packaging>20.6 in /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex-loadams/unit-test-venv/lib/python3.8/site-packages (from apex==0.1) (23.1)
Building wheels for collected packages: apex
  Building wheel for apex (pyproject.toml) ...   Running command Building wheel for apex (pyproject.toml)

  torch.__version__  = 1.13.1+cu116

  running bdist_wheel
  running build
  running build_py
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/amp
  creating build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/torch_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/__init__.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/functional_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/tensor_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/_process_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/wrap.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/utils.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/compat.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/_initialize.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/__version__.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/amp.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/frontend.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/scaler.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/_amp_state.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/handle.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/rnn_compat.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/opt.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/__init__.py -> build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/normalization
  copying build/lib/apex/normalization/__init__.py -> build/bdist.linux-x86_64/wheel/apex/normalization
  copying build/lib/apex/normalization/fused_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/normalization
  creating build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/multiproc.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/LARC.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/optimized_sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/optimized_sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/distributed.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  creating build/bdist.linux-x86_64/wheel/apex/contrib
  creating build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  copying build/lib/apex/contrib/cudnn_gbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  copying build/lib/apex/contrib/cudnn_gbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  copying build/lib/apex/contrib/index_mul_2d/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  copying build/lib/apex/contrib/index_mul_2d/index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  creating build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/focal_loss/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/focal_loss/focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib
  creating build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  copying build/lib/apex/contrib/xentropy/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  copying build/lib/apex/contrib/xentropy/softmax_xentropy.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  creating build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/peer_halo_exchanger_1d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/peer_memory.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  creating build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  copying build/lib/apex/contrib/conv_bias_relu/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  copying build/lib/apex/contrib/conv_bias_relu/conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  creating build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  copying build/lib/apex/contrib/fmha/fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  copying build/lib/apex/contrib/fmha/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  creating build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/_transducer_ref.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/transducer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  creating build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/test.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/bottleneck.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/halo_exchangers.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  creating build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  copying build/lib/apex/contrib/layer_norm/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  copying build/lib/apex/contrib/layer_norm/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  creating build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  copying build/lib/apex/contrib/groupbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  copying build/lib/apex/contrib/groupbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/channel_swap.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/permutation_lib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/asp.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/sparse_masklib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  copying build/lib/apex/contrib/test/cudnn_gbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  copying build/lib/apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  copying build/lib/apex/contrib/test/index_mul_2d/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  copying build/lib/apex/contrib/test/index_mul_2d/test_index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/focal_loss/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/focal_loss/test_focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  copying build/lib/apex/contrib/test/xentropy/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  copying build/lib/apex/contrib/test/xentropy/test_label_smoothing.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  copying build/lib/apex/contrib/test/peer_memory/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  copying build/lib/apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  copying build/lib/apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  copying build/lib/apex/contrib/test/conv_bias_relu/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  copying build/lib/apex/contrib/test/fmha/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  copying build/lib/apex/contrib/test/fmha/test_fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/test_transducer_joint.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/test_transducer_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  copying build/lib/apex/contrib/test/bottleneck/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  copying build/lib/apex/contrib/test/bottleneck/test_bottleneck_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  copying build/lib/apex/contrib/test/layer_norm/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  copying build/lib/apex/contrib/test/layer_norm/test_fast_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/test_distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/test_dist_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  copying build/lib/apex/contrib/test/clip_grad/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  copying build/lib/apex/contrib/test/clip_grad/test_clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_mha_fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/distributed_fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  copying build/lib/apex/contrib/clip_grad/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  copying build/lib/apex/contrib/clip_grad/clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  creating build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/mask_softmax_dropout_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  creating build/bdist.linux-x86_64/wheel/apex/fused_dense
  copying build/lib/apex/fused_dense/__init__.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense
  copying build/lib/apex/fused_dense/fused_dense.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense
  creating build/bdist.linux-x86_64/wheel/apex/mlp
  copying build/lib/apex/mlp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/mlp
  copying build/lib/apex/mlp/mlp.py -> build/bdist.linux-x86_64/wheel/apex/mlp
  creating build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/loss_scaler.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/__init__.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/fp16util.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  creating build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_mixed_precision_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_adagrad.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_novograd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/amp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/amp/grad_scaler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/_ucc_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/parallel_state.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/common.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/p2p_communication.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/_timers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/layers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/cross_entropy.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/mappings.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/data.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/memory.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/random.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  creating build/bdist.linux-x86_64/wheel/apex/transformer/_data
  copying build/lib/apex/transformer/_data/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data
  copying build/lib/apex/transformer/_data/_batchsampler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data
  creating build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/layers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/layers/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/enums.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/microbatches.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/functional/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/functional/fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/log_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_transformer_lm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/commons.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/global_vars.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_bert.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/distributed_test_base.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/arguments.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_gpt.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  creating build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/RNNBackend.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/cells.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/__init__.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/models.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/_autocast_utils.py -> build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  copying build/lib/apex/multi_tensor_apply/__init__.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  copying build/lib/apex/multi_tensor_apply/multi_tensor_apply.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  running install_egg_info
  running egg_info
  writing apex.egg-info/PKG-INFO
  writing dependency_links to apex.egg-info/dependency_links.txt
  writing requirements to apex.egg-info/requires.txt
  writing top-level names to apex.egg-info/top_level.txt
  adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
  reading manifest file 'apex.egg-info/SOURCES.txt'
  writing manifest file 'apex.egg-info/SOURCES.txt'
  Copying apex.egg-info to build/bdist.linux-x86_64/wheel/apex-0.1-py3.8.egg-info
  running install_scripts
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  creating build/bdist.linux-x86_64/wheel/apex-0.1.dist-info/WHEEL
  creating '/tmp/pip-wheel-6bsifi7_/tmp6gur0gc5/apex-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
  adding 'apex/__init__.py'
  adding 'apex/_autocast_utils.py'
  adding 'apex/RNN/RNNBackend.py'
  adding 'apex/RNN/__init__.py'
  adding 'apex/RNN/cells.py'
  adding 'apex/RNN/models.py'
  adding 'apex/amp/__init__.py'
  adding 'apex/amp/__version__.py'
  adding 'apex/amp/_amp_state.py'
  adding 'apex/amp/_initialize.py'
  adding 'apex/amp/_process_optimizer.py'
  adding 'apex/amp/amp.py'
  adding 'apex/amp/compat.py'
  adding 'apex/amp/frontend.py'
  adding 'apex/amp/handle.py'
  adding 'apex/amp/opt.py'
  adding 'apex/amp/rnn_compat.py'
  adding 'apex/amp/scaler.py'
  adding 'apex/amp/utils.py'
  adding 'apex/amp/wrap.py'
  adding 'apex/amp/lists/__init__.py'
  adding 'apex/amp/lists/functional_overrides.py'
  adding 'apex/amp/lists/tensor_overrides.py'
  adding 'apex/amp/lists/torch_overrides.py'
  adding 'apex/contrib/__init__.py'
  adding 'apex/contrib/bottleneck/__init__.py'
  adding 'apex/contrib/bottleneck/bottleneck.py'
  adding 'apex/contrib/bottleneck/halo_exchangers.py'
  adding 'apex/contrib/bottleneck/test.py'
  adding 'apex/contrib/clip_grad/__init__.py'
  adding 'apex/contrib/clip_grad/clip_grad.py'
  adding 'apex/contrib/conv_bias_relu/__init__.py'
  adding 'apex/contrib/conv_bias_relu/conv_bias_relu.py'
  adding 'apex/contrib/cudnn_gbn/__init__.py'
  adding 'apex/contrib/cudnn_gbn/batch_norm.py'
  adding 'apex/contrib/fmha/__init__.py'
  adding 'apex/contrib/fmha/fmha.py'
  adding 'apex/contrib/focal_loss/__init__.py'
  adding 'apex/contrib/focal_loss/focal_loss.py'
  adding 'apex/contrib/groupbn/__init__.py'
  adding 'apex/contrib/groupbn/batch_norm.py'
  adding 'apex/contrib/index_mul_2d/__init__.py'
  adding 'apex/contrib/index_mul_2d/index_mul_2d.py'
  adding 'apex/contrib/layer_norm/__init__.py'
  adding 'apex/contrib/layer_norm/layer_norm.py'
  adding 'apex/contrib/multihead_attn/__init__.py'
  adding 'apex/contrib/multihead_attn/encdec_multihead_attn.py'
  adding 'apex/contrib/multihead_attn/encdec_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py'
  adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py'
  adding 'apex/contrib/multihead_attn/mask_softmax_dropout_func.py'
  adding 'apex/contrib/multihead_attn/self_multihead_attn.py'
  adding 'apex/contrib/multihead_attn/self_multihead_attn_func.py'
  adding 'apex/contrib/optimizers/__init__.py'
  adding 'apex/contrib/optimizers/distributed_fused_adam.py'
  adding 'apex/contrib/optimizers/distributed_fused_lamb.py'
  adding 'apex/contrib/optimizers/fp16_optimizer.py'
  adding 'apex/contrib/optimizers/fused_adam.py'
  adding 'apex/contrib/optimizers/fused_lamb.py'
  adding 'apex/contrib/optimizers/fused_sgd.py'
  adding 'apex/contrib/peer_memory/__init__.py'
  adding 'apex/contrib/peer_memory/peer_halo_exchanger_1d.py'
  adding 'apex/contrib/peer_memory/peer_memory.py'
  adding 'apex/contrib/sparsity/__init__.py'
  adding 'apex/contrib/sparsity/asp.py'
  adding 'apex/contrib/sparsity/permutation_lib.py'
  adding 'apex/contrib/sparsity/sparse_masklib.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/__init__.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/channel_swap.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py'
  adding 'apex/contrib/test/__init__.py'
  adding 'apex/contrib/test/bottleneck/__init__.py'
  adding 'apex/contrib/test/bottleneck/test_bottleneck_module.py'
  adding 'apex/contrib/test/clip_grad/__init__.py'
  adding 'apex/contrib/test/clip_grad/test_clip_grad.py'
  adding 'apex/contrib/test/conv_bias_relu/__init__.py'
  adding 'apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py'
  adding 'apex/contrib/test/cudnn_gbn/__init__.py'
  adding 'apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py'
  adding 'apex/contrib/test/fmha/__init__.py'
  adding 'apex/contrib/test/fmha/test_fmha.py'
  adding 'apex/contrib/test/focal_loss/__init__.py'
  adding 'apex/contrib/test/focal_loss/test_focal_loss.py'
  adding 'apex/contrib/test/index_mul_2d/__init__.py'
  adding 'apex/contrib/test/index_mul_2d/test_index_mul_2d.py'
  adding 'apex/contrib/test/layer_norm/__init__.py'
  adding 'apex/contrib/test/layer_norm/test_fast_layer_norm.py'
  adding 'apex/contrib/test/multihead_attn/__init__.py'
  adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py'
  adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py'
  adding 'apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py'
  adding 'apex/contrib/test/multihead_attn/test_mha_fused_softmax.py'
  adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn.py'
  adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py'
  adding 'apex/contrib/test/optimizers/__init__.py'
  adding 'apex/contrib/test/optimizers/test_dist_adam.py'
  adding 'apex/contrib/test/optimizers/test_distributed_fused_lamb.py'
  adding 'apex/contrib/test/peer_memory/__init__.py'
  adding 'apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py'
  adding 'apex/contrib/test/transducer/__init__.py'
  adding 'apex/contrib/test/transducer/test_transducer_joint.py'
  adding 'apex/contrib/test/transducer/test_transducer_loss.py'
  adding 'apex/contrib/test/xentropy/__init__.py'
  adding 'apex/contrib/test/xentropy/test_label_smoothing.py'
  adding 'apex/contrib/transducer/__init__.py'
  adding 'apex/contrib/transducer/_transducer_ref.py'
  adding 'apex/contrib/transducer/transducer.py'
  adding 'apex/contrib/xentropy/__init__.py'
  adding 'apex/contrib/xentropy/softmax_xentropy.py'
  adding 'apex/fp16_utils/__init__.py'
  adding 'apex/fp16_utils/fp16_optimizer.py'
  adding 'apex/fp16_utils/fp16util.py'
  adding 'apex/fp16_utils/loss_scaler.py'
  adding 'apex/fused_dense/__init__.py'
  adding 'apex/fused_dense/fused_dense.py'
  adding 'apex/mlp/__init__.py'
  adding 'apex/mlp/mlp.py'
  adding 'apex/multi_tensor_apply/__init__.py'
  adding 'apex/multi_tensor_apply/multi_tensor_apply.py'
  adding 'apex/normalization/__init__.py'
  adding 'apex/normalization/fused_layer_norm.py'
  adding 'apex/optimizers/__init__.py'
  adding 'apex/optimizers/fused_adagrad.py'
  adding 'apex/optimizers/fused_adam.py'
  adding 'apex/optimizers/fused_lamb.py'
  adding 'apex/optimizers/fused_mixed_precision_lamb.py'
  adding 'apex/optimizers/fused_novograd.py'
  adding 'apex/optimizers/fused_sgd.py'
  adding 'apex/parallel/LARC.py'
  adding 'apex/parallel/__init__.py'
  adding 'apex/parallel/distributed.py'
  adding 'apex/parallel/multiproc.py'
  adding 'apex/parallel/optimized_sync_batchnorm.py'
  adding 'apex/parallel/optimized_sync_batchnorm_kernel.py'
  adding 'apex/parallel/sync_batchnorm.py'
  adding 'apex/parallel/sync_batchnorm_kernel.py'
  adding 'apex/transformer/__init__.py'
  adding 'apex/transformer/_ucc_util.py'
  adding 'apex/transformer/enums.py'
  adding 'apex/transformer/log_util.py'
  adding 'apex/transformer/microbatches.py'
  adding 'apex/transformer/parallel_state.py'
  adding 'apex/transformer/utils.py'
  adding 'apex/transformer/_data/__init__.py'
  adding 'apex/transformer/_data/_batchsampler.py'
  adding 'apex/transformer/amp/__init__.py'
  adding 'apex/transformer/amp/grad_scaler.py'
  adding 'apex/transformer/functional/__init__.py'
  adding 'apex/transformer/functional/fused_softmax.py'
  adding 'apex/transformer/layers/__init__.py'
  adding 'apex/transformer/layers/layer_norm.py'
  adding 'apex/transformer/pipeline_parallel/__init__.py'
  adding 'apex/transformer/pipeline_parallel/_timers.py'
  adding 'apex/transformer/pipeline_parallel/p2p_communication.py'
  adding 'apex/transformer/pipeline_parallel/utils.py'
  adding 'apex/transformer/pipeline_parallel/schedules/__init__.py'
  adding 'apex/transformer/pipeline_parallel/schedules/common.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py'
  adding 'apex/transformer/tensor_parallel/__init__.py'
  adding 'apex/transformer/tensor_parallel/cross_entropy.py'
  adding 'apex/transformer/tensor_parallel/data.py'
  adding 'apex/transformer/tensor_parallel/layers.py'
  adding 'apex/transformer/tensor_parallel/mappings.py'
  adding 'apex/transformer/tensor_parallel/memory.py'
  adding 'apex/transformer/tensor_parallel/random.py'
  adding 'apex/transformer/tensor_parallel/utils.py'
  adding 'apex/transformer/testing/__init__.py'
  adding 'apex/transformer/testing/arguments.py'
  adding 'apex/transformer/testing/commons.py'
  adding 'apex/transformer/testing/distributed_test_base.py'
  adding 'apex/transformer/testing/global_vars.py'
  adding 'apex/transformer/testing/standalone_bert.py'
  adding 'apex/transformer/testing/standalone_gpt.py'
  adding 'apex/transformer/testing/standalone_transformer_lm.py'
  adding 'apex-0.1.dist-info/LICENSE'
  adding 'apex-0.1.dist-info/METADATA'
  adding 'apex-0.1.dist-info/WHEEL'
  adding 'apex-0.1.dist-info/top_level.txt'
  adding 'apex-0.1.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
done
  Created wheel for apex: filename=apex-0.1-py3-none-any.whl size=361346 sha256=bcb7f1a307c282a024de732f6c2ebbcff4c00807a5c27503b53f6f6ab70205b4
  Stored in directory: /tmp/pip-ephem-wheel-cache-3gyf6bgs/wheels/0e/9e/61/20762dce2ad2deec46512626627c2fe5487fc7306c71d6cfc8
Successfully built apex
Installing collected packages: apex
Successfully installed apex-0.1

And this hasn't built amp_C or apex_C.

xwang233 commented 1 year ago

We're using the latest commit, which includes #1669. Our pytorch is from a source build, but I'm not sure if that's the issue.

I also tried pip 23.1.2 and it worked as expected. Can you try if pip install with --no-cache-dir --disable-pip-version-check helps? Also make sure to delete previous build caches and uninstall previous installations.

loadams commented 1 year ago

Interesting, mine is torch 1.13, but I doubt the torch version makes a difference. Also in a venv if that matters. I'm just not able to see it ever get to the part where it parses any sys.argv values?

apex$ pip list
Package            Version
------------------ ------------
certifi            2023.5.7
charset-normalizer 3.1.0
idna               3.4
numpy              1.24.3
packaging          23.1
Pillow             9.5.0
pip                23.1.2
requests           2.31.0
setuptools         56.0.0
torch              1.13.1+cu116
torchvision        0.14.1+cu116
typing_extensions  4.6.3
urllib3            2.0.3
wheel              0.40.0

apex$ pip install -v --no-build-isolation --no-cache-dir --disable-pip-version-check --config-settings --build-option="--cpp_ext --cuda_ext" ./
Using pip 23.1.2 from /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex-test/lib/python3.8/site-packages/pip (python 3.8)
Processing /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex
  Preparing metadata (pyproject.toml) ...   Running command Preparing metadata (pyproject.toml)

  torch.__version__  = 1.13.1+cu116

  running dist_info
  creating /tmp/pip-modern-metadata-z3_0v813/apex.egg-info
  writing /tmp/pip-modern-metadata-z3_0v813/apex.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-z3_0v813/apex.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-z3_0v813/apex.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-z3_0v813/apex.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-z3_0v813/apex.egg-info/SOURCES.txt'
  adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
  reading manifest file '/tmp/pip-modern-metadata-z3_0v813/apex.egg-info/SOURCES.txt'
  writing manifest file '/tmp/pip-modern-metadata-z3_0v813/apex.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-z3_0v813/apex.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
done
Requirement already satisfied: packaging>20.6 in /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex-test/lib/python3.8/site-packages (from apex==0.1) (23.1)
Building wheels for collected packages: apex
  Building wheel for apex (pyproject.toml) ...   Running command Building wheel for apex (pyproject.toml)

  torch.__version__  = 1.13.1+cu116

  running bdist_wheel
  running build
  running build_py
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/amp
  creating build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/torch_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/__init__.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/functional_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/tensor_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/_process_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/wrap.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/utils.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/compat.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/_initialize.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/__version__.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/amp.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/frontend.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/scaler.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/_amp_state.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/handle.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/rnn_compat.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/opt.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/__init__.py -> build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/normalization
  copying build/lib/apex/normalization/__init__.py -> build/bdist.linux-x86_64/wheel/apex/normalization
  copying build/lib/apex/normalization/fused_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/normalization
  creating build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/multiproc.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/LARC.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/optimized_sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/optimized_sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/distributed.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  creating build/bdist.linux-x86_64/wheel/apex/contrib
  creating build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  copying build/lib/apex/contrib/cudnn_gbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  copying build/lib/apex/contrib/cudnn_gbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  copying build/lib/apex/contrib/index_mul_2d/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  copying build/lib/apex/contrib/index_mul_2d/index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  creating build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/focal_loss/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/focal_loss/focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib
  creating build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  copying build/lib/apex/contrib/xentropy/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  copying build/lib/apex/contrib/xentropy/softmax_xentropy.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  creating build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/peer_halo_exchanger_1d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/peer_memory.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  creating build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  copying build/lib/apex/contrib/conv_bias_relu/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  copying build/lib/apex/contrib/conv_bias_relu/conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  creating build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  copying build/lib/apex/contrib/fmha/fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  copying build/lib/apex/contrib/fmha/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  creating build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/_transducer_ref.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/transducer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  creating build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/test.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/bottleneck.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/halo_exchangers.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  creating build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  copying build/lib/apex/contrib/layer_norm/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  copying build/lib/apex/contrib/layer_norm/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  creating build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  copying build/lib/apex/contrib/groupbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  copying build/lib/apex/contrib/groupbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/channel_swap.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/permutation_lib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/asp.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/sparse_masklib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  copying build/lib/apex/contrib/test/cudnn_gbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  copying build/lib/apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  copying build/lib/apex/contrib/test/index_mul_2d/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  copying build/lib/apex/contrib/test/index_mul_2d/test_index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/focal_loss/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/focal_loss/test_focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  copying build/lib/apex/contrib/test/xentropy/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  copying build/lib/apex/contrib/test/xentropy/test_label_smoothing.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  copying build/lib/apex/contrib/test/peer_memory/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  copying build/lib/apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  copying build/lib/apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  copying build/lib/apex/contrib/test/conv_bias_relu/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  copying build/lib/apex/contrib/test/fmha/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  copying build/lib/apex/contrib/test/fmha/test_fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/test_transducer_joint.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/test_transducer_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  copying build/lib/apex/contrib/test/bottleneck/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  copying build/lib/apex/contrib/test/bottleneck/test_bottleneck_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  copying build/lib/apex/contrib/test/layer_norm/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  copying build/lib/apex/contrib/test/layer_norm/test_fast_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/test_distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/test_dist_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  copying build/lib/apex/contrib/test/clip_grad/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  copying build/lib/apex/contrib/test/clip_grad/test_clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_mha_fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/distributed_fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  copying build/lib/apex/contrib/clip_grad/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  copying build/lib/apex/contrib/clip_grad/clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  creating build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/mask_softmax_dropout_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  creating build/bdist.linux-x86_64/wheel/apex/fused_dense
  copying build/lib/apex/fused_dense/__init__.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense
  copying build/lib/apex/fused_dense/fused_dense.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense
  creating build/bdist.linux-x86_64/wheel/apex/mlp
  copying build/lib/apex/mlp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/mlp
  copying build/lib/apex/mlp/mlp.py -> build/bdist.linux-x86_64/wheel/apex/mlp
  creating build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/loss_scaler.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/__init__.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/fp16util.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  creating build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_mixed_precision_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_adagrad.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_novograd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/amp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/amp/grad_scaler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/_ucc_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/parallel_state.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/common.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/p2p_communication.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/_timers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/layers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/cross_entropy.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/mappings.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/data.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/memory.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/random.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  creating build/bdist.linux-x86_64/wheel/apex/transformer/_data
  copying build/lib/apex/transformer/_data/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data
  copying build/lib/apex/transformer/_data/_batchsampler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data
  creating build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/layers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/layers/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/enums.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/microbatches.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/functional/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/functional/fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/log_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_transformer_lm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/commons.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/global_vars.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_bert.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/distributed_test_base.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/arguments.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_gpt.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  creating build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/RNNBackend.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/cells.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/__init__.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/models.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/_autocast_utils.py -> build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  copying build/lib/apex/multi_tensor_apply/__init__.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  copying build/lib/apex/multi_tensor_apply/multi_tensor_apply.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  running install_egg_info
  running egg_info
  writing apex.egg-info/PKG-INFO
  writing dependency_links to apex.egg-info/dependency_links.txt
  writing requirements to apex.egg-info/requires.txt
  writing top-level names to apex.egg-info/top_level.txt
  adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
  reading manifest file 'apex.egg-info/SOURCES.txt'
  writing manifest file 'apex.egg-info/SOURCES.txt'
  Copying apex.egg-info to build/bdist.linux-x86_64/wheel/apex-0.1-py3.8.egg-info
  running install_scripts
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  creating build/bdist.linux-x86_64/wheel/apex-0.1.dist-info/WHEEL
  creating '/tmp/pip-wheel-ftuic7hb/tmpd6e9dhnd/apex-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
  adding 'apex/__init__.py'
  adding 'apex/_autocast_utils.py'
  adding 'apex/RNN/RNNBackend.py'
  adding 'apex/RNN/__init__.py'
  adding 'apex/RNN/cells.py'
  adding 'apex/RNN/models.py'
  adding 'apex/amp/__init__.py'
  adding 'apex/amp/__version__.py'
  adding 'apex/amp/_amp_state.py'
  adding 'apex/amp/_initialize.py'
  adding 'apex/amp/_process_optimizer.py'
  adding 'apex/amp/amp.py'
  adding 'apex/amp/compat.py'
  adding 'apex/amp/frontend.py'
  adding 'apex/amp/handle.py'
  adding 'apex/amp/opt.py'
  adding 'apex/amp/rnn_compat.py'
  adding 'apex/amp/scaler.py'
  adding 'apex/amp/utils.py'
  adding 'apex/amp/wrap.py'
  adding 'apex/amp/lists/__init__.py'
  adding 'apex/amp/lists/functional_overrides.py'
  adding 'apex/amp/lists/tensor_overrides.py'
  adding 'apex/amp/lists/torch_overrides.py'
  adding 'apex/contrib/__init__.py'
  adding 'apex/contrib/bottleneck/__init__.py'
  adding 'apex/contrib/bottleneck/bottleneck.py'
  adding 'apex/contrib/bottleneck/halo_exchangers.py'
  adding 'apex/contrib/bottleneck/test.py'
  adding 'apex/contrib/clip_grad/__init__.py'
  adding 'apex/contrib/clip_grad/clip_grad.py'
  adding 'apex/contrib/conv_bias_relu/__init__.py'
  adding 'apex/contrib/conv_bias_relu/conv_bias_relu.py'
  adding 'apex/contrib/cudnn_gbn/__init__.py'
  adding 'apex/contrib/cudnn_gbn/batch_norm.py'
  adding 'apex/contrib/fmha/__init__.py'
  adding 'apex/contrib/fmha/fmha.py'
  adding 'apex/contrib/focal_loss/__init__.py'
  adding 'apex/contrib/focal_loss/focal_loss.py'
  adding 'apex/contrib/groupbn/__init__.py'
  adding 'apex/contrib/groupbn/batch_norm.py'
  adding 'apex/contrib/index_mul_2d/__init__.py'
  adding 'apex/contrib/index_mul_2d/index_mul_2d.py'
  adding 'apex/contrib/layer_norm/__init__.py'
  adding 'apex/contrib/layer_norm/layer_norm.py'
  adding 'apex/contrib/multihead_attn/__init__.py'
  adding 'apex/contrib/multihead_attn/encdec_multihead_attn.py'
  adding 'apex/contrib/multihead_attn/encdec_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py'
  adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py'
  adding 'apex/contrib/multihead_attn/mask_softmax_dropout_func.py'
  adding 'apex/contrib/multihead_attn/self_multihead_attn.py'
  adding 'apex/contrib/multihead_attn/self_multihead_attn_func.py'
  adding 'apex/contrib/optimizers/__init__.py'
  adding 'apex/contrib/optimizers/distributed_fused_adam.py'
  adding 'apex/contrib/optimizers/distributed_fused_lamb.py'
  adding 'apex/contrib/optimizers/fp16_optimizer.py'
  adding 'apex/contrib/optimizers/fused_adam.py'
  adding 'apex/contrib/optimizers/fused_lamb.py'
  adding 'apex/contrib/optimizers/fused_sgd.py'
  adding 'apex/contrib/peer_memory/__init__.py'
  adding 'apex/contrib/peer_memory/peer_halo_exchanger_1d.py'
  adding 'apex/contrib/peer_memory/peer_memory.py'
  adding 'apex/contrib/sparsity/__init__.py'
  adding 'apex/contrib/sparsity/asp.py'
  adding 'apex/contrib/sparsity/permutation_lib.py'
  adding 'apex/contrib/sparsity/sparse_masklib.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/__init__.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/channel_swap.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py'
  adding 'apex/contrib/test/__init__.py'
  adding 'apex/contrib/test/bottleneck/__init__.py'
  adding 'apex/contrib/test/bottleneck/test_bottleneck_module.py'
  adding 'apex/contrib/test/clip_grad/__init__.py'
  adding 'apex/contrib/test/clip_grad/test_clip_grad.py'
  adding 'apex/contrib/test/conv_bias_relu/__init__.py'
  adding 'apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py'
  adding 'apex/contrib/test/cudnn_gbn/__init__.py'
  adding 'apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py'
  adding 'apex/contrib/test/fmha/__init__.py'
  adding 'apex/contrib/test/fmha/test_fmha.py'
  adding 'apex/contrib/test/focal_loss/__init__.py'
  adding 'apex/contrib/test/focal_loss/test_focal_loss.py'
  adding 'apex/contrib/test/index_mul_2d/__init__.py'
  adding 'apex/contrib/test/index_mul_2d/test_index_mul_2d.py'
  adding 'apex/contrib/test/layer_norm/__init__.py'
  adding 'apex/contrib/test/layer_norm/test_fast_layer_norm.py'
  adding 'apex/contrib/test/multihead_attn/__init__.py'
  adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py'
  adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py'
  adding 'apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py'
  adding 'apex/contrib/test/multihead_attn/test_mha_fused_softmax.py'
  adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn.py'
  adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py'
  adding 'apex/contrib/test/optimizers/__init__.py'
  adding 'apex/contrib/test/optimizers/test_dist_adam.py'
  adding 'apex/contrib/test/optimizers/test_distributed_fused_lamb.py'
  adding 'apex/contrib/test/peer_memory/__init__.py'
  adding 'apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py'
  adding 'apex/contrib/test/transducer/__init__.py'
  adding 'apex/contrib/test/transducer/test_transducer_joint.py'
  adding 'apex/contrib/test/transducer/test_transducer_loss.py'
  adding 'apex/contrib/test/xentropy/__init__.py'
  adding 'apex/contrib/test/xentropy/test_label_smoothing.py'
  adding 'apex/contrib/transducer/__init__.py'
  adding 'apex/contrib/transducer/_transducer_ref.py'
  adding 'apex/contrib/transducer/transducer.py'
  adding 'apex/contrib/xentropy/__init__.py'
  adding 'apex/contrib/xentropy/softmax_xentropy.py'
  adding 'apex/fp16_utils/__init__.py'
  adding 'apex/fp16_utils/fp16_optimizer.py'
  adding 'apex/fp16_utils/fp16util.py'
  adding 'apex/fp16_utils/loss_scaler.py'
  adding 'apex/fused_dense/__init__.py'
  adding 'apex/fused_dense/fused_dense.py'
  adding 'apex/mlp/__init__.py'
  adding 'apex/mlp/mlp.py'
  adding 'apex/multi_tensor_apply/__init__.py'
  adding 'apex/multi_tensor_apply/multi_tensor_apply.py'
  adding 'apex/normalization/__init__.py'
  adding 'apex/normalization/fused_layer_norm.py'
  adding 'apex/optimizers/__init__.py'
  adding 'apex/optimizers/fused_adagrad.py'
  adding 'apex/optimizers/fused_adam.py'
  adding 'apex/optimizers/fused_lamb.py'
  adding 'apex/optimizers/fused_mixed_precision_lamb.py'
  adding 'apex/optimizers/fused_novograd.py'
  adding 'apex/optimizers/fused_sgd.py'
  adding 'apex/parallel/LARC.py'
  adding 'apex/parallel/__init__.py'
  adding 'apex/parallel/distributed.py'
  adding 'apex/parallel/multiproc.py'
  adding 'apex/parallel/optimized_sync_batchnorm.py'
  adding 'apex/parallel/optimized_sync_batchnorm_kernel.py'
  adding 'apex/parallel/sync_batchnorm.py'
  adding 'apex/parallel/sync_batchnorm_kernel.py'
  adding 'apex/transformer/__init__.py'
  adding 'apex/transformer/_ucc_util.py'
  adding 'apex/transformer/enums.py'
  adding 'apex/transformer/log_util.py'
  adding 'apex/transformer/microbatches.py'
  adding 'apex/transformer/parallel_state.py'
  adding 'apex/transformer/utils.py'
  adding 'apex/transformer/_data/__init__.py'
  adding 'apex/transformer/_data/_batchsampler.py'
  adding 'apex/transformer/amp/__init__.py'
  adding 'apex/transformer/amp/grad_scaler.py'
  adding 'apex/transformer/functional/__init__.py'
  adding 'apex/transformer/functional/fused_softmax.py'
  adding 'apex/transformer/layers/__init__.py'
  adding 'apex/transformer/layers/layer_norm.py'
  adding 'apex/transformer/pipeline_parallel/__init__.py'
  adding 'apex/transformer/pipeline_parallel/_timers.py'
  adding 'apex/transformer/pipeline_parallel/p2p_communication.py'
  adding 'apex/transformer/pipeline_parallel/utils.py'
  adding 'apex/transformer/pipeline_parallel/schedules/__init__.py'
  adding 'apex/transformer/pipeline_parallel/schedules/common.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py'
  adding 'apex/transformer/tensor_parallel/__init__.py'
  adding 'apex/transformer/tensor_parallel/cross_entropy.py'
  adding 'apex/transformer/tensor_parallel/data.py'
  adding 'apex/transformer/tensor_parallel/layers.py'
  adding 'apex/transformer/tensor_parallel/mappings.py'
  adding 'apex/transformer/tensor_parallel/memory.py'
  adding 'apex/transformer/tensor_parallel/random.py'
  adding 'apex/transformer/tensor_parallel/utils.py'
  adding 'apex/transformer/testing/__init__.py'
  adding 'apex/transformer/testing/arguments.py'
  adding 'apex/transformer/testing/commons.py'
  adding 'apex/transformer/testing/distributed_test_base.py'
  adding 'apex/transformer/testing/global_vars.py'
  adding 'apex/transformer/testing/standalone_bert.py'
  adding 'apex/transformer/testing/standalone_gpt.py'
  adding 'apex/transformer/testing/standalone_transformer_lm.py'
  adding 'apex-0.1.dist-info/LICENSE'
  adding 'apex-0.1.dist-info/METADATA'
  adding 'apex-0.1.dist-info/WHEEL'
  adding 'apex-0.1.dist-info/top_level.txt'
  adding 'apex-0.1.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
done
  Created wheel for apex: filename=apex-0.1-py3-none-any.whl size=361346 sha256=cb727c79515b1287e5794387bf1b981a7ac896e0048fc649177c6b1189f63242
  Stored in directory: /tmp/pip-ephem-wheel-cache-suoww_xw/wheels/0e/9e/61/20762dce2ad2deec46512626627c2fe5487fc7306c71d6cfc8
Successfully built apex
Installing collected packages: apex
Successfully installed apex-0.1
RuABraun commented 1 year ago

also have install problem that only appeared after this change. if I fix the missing "packaging" i get an error about torch despite it being installed.

for everyone with this problem I suggest checking out 82ee367, that worked for me

loadams commented 1 year ago

@RuABraun - that's what many of us are doing, but that will prevent future changes from being taken.

If you add pytorch and packaging, that will resolve those issues but the overall installation (at least for me) is failing with another issue of not installing the cpp_ext or cuda_ext.

dashstander commented 1 year ago

I've been experiencing the precise same issues as @loadams

crcrpar commented 1 year ago

one way (I wouldn't recommend though) to dodge pyproject.toml dependency management could be to use python setup.py install ---cpp_ext --cuda_ext ... to avoid pip being called

dayL-W commented 1 year ago

Did you solve it? I have the same error

janEbert commented 1 year ago

I think the README is simply wrong. You need to use --global-option instead of --build-option. Here's the fixed installation command, adapted from the README:

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--global-option=--cpp_ext" --config-settings "--global-option=--cuda_ext" ./
loadams commented 1 year ago

@janEbert - thanks, that does work for me.

@crcrpar - could we get a review on and merge that PR? Also I don't need the changes to adding packaging and torch to the pyproject.toml, but I do think those are still needed here.

RuABraun commented 1 year ago

what about this message (from pip I guess)?

          ********************************************************************************
          The arguments ['--cpp_ext', '--cuda_ext'] were given via `--global-option`.
          Please use `--build-option` instead,
          `--global-option` is reserved for flags like `--verbose` or `--quiet`.

          By 2023-Sep-26, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.
          ********************************************************************************

Doesn't seem like --global-option is a sustainable solution

janEbert commented 1 year ago

Thank @RuABraun, I had not seen that warning! So this is an incorrect solution after all, but at least it's a workaround until pip 23.3. Maybe the problem of accessing --build-option is then in setup.py?

janEbert commented 1 year ago

I don't get the same warning for some reason, but I created a new PR with an alternative solution based on what Pillow uses to support custom arguments.

--build-options aren't actually passed to the build_wheel command for reasons unknown to me. So the new PR patches setuptools.build_meta to take the flags manually.

VarunGumma commented 1 year ago

I am using Torch 2.0.1, python 3.9, pip 23.1.2 and the latest version of apex. I still run into the same issue which gives the error ModuleNotFoundError: No module named 'packaging', error: subprocess-exited-with-error. I want to use this with fairseq, and I am using the following command to install it:

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" \
  --global-option="--deprecated_fused_adam" --global-option="--xentropy" \
  --global-option="--fast_multihead_attn" ./

Any workarounds?

janEbert commented 1 year ago

Try with --no-build-isolation.

slent310 commented 1 year ago

pip install -v --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" --global-option="--xentropy" --global-option="--fast_multihead_attn" ./ it's works for me!

VarunGumma commented 1 year ago

But then I get this error: ModuleNotFoundError: No module named 'fused_layer_norm_cuda'

SolenoidWGT commented 1 year ago

But then I get this error: ModuleNotFoundError: No module named 'fused_layer_norm_cuda'

@VarunGumma try git checkout 23.05 before installation and then run pip installation command, this is work fine for me.

ChaosPengs commented 11 months ago

I am using Torch 2.0.1, python 3.9, pip 23.1.2 and the latest version of apex. I still run into the same issue which gives the error ModuleNotFoundError: No module named 'packaging', error: subprocess-exited-with-error. I want to use this with fairseq, and I am using the following command to install it:

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" \
  --global-option="--deprecated_fused_adam" --global-option="--xentropy" \
  --global-option="--fast_multihead_attn" ./

Any workarounds?

we have the same environment, did you solve the problem?

VarunGumma commented 11 months ago

So, what I did was clone the repo and checkout to an older commit (something around April), and install it the regular way. Once it is successfully installed, I just git pull the latest changes.

loadams commented 11 months ago

@ChaosPengs - you'd need to ensure you have packaging installed in your environment and add it to the pyproject.toml as a dependency.

yuhuai4554 commented 10 months ago

@xwang233 - Are you using the most up to date master branch from the repo? Since you should also need to add packaging and torch as dependencies to the pyproject.toml first, right?

Here is my output:

pip install -v --no-build-isolation --config-settings --build-option="--cpp_ext --cuda_ext" ./
Using pip 23.1.2 from /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex-loadams/unit-test-venv/lib/python3.8/site-packages/pip (python 3.8)
Processing /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex
  Preparing metadata (pyproject.toml) ...   Running command Preparing metadata (pyproject.toml)

  torch.__version__  = 1.13.1+cu116

  running dist_info
  creating /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info
  writing /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/SOURCES.txt'
  adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
  reading manifest file '/tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/SOURCES.txt'
  writing manifest file '/tmp/pip-modern-metadata-i6iyi61n/apex.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-i6iyi61n/apex.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
done
Requirement already satisfied: packaging>20.6 in /tmp/azureml/cr/j/4e2a604ed7974feaac58e0dfb7e2f4d3/exe/wd/apex-loadams/unit-test-venv/lib/python3.8/site-packages (from apex==0.1) (23.1)
Building wheels for collected packages: apex
  Building wheel for apex (pyproject.toml) ...   Running command Building wheel for apex (pyproject.toml)

  torch.__version__  = 1.13.1+cu116

  running bdist_wheel
  running build
  running build_py
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/amp
  creating build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/torch_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/__init__.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/functional_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/lists/tensor_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists
  copying build/lib/apex/amp/_process_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/wrap.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/utils.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/compat.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/_initialize.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/__version__.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/amp.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/frontend.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/scaler.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/_amp_state.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/handle.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/rnn_compat.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/amp/opt.py -> build/bdist.linux-x86_64/wheel/apex/amp
  copying build/lib/apex/__init__.py -> build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/normalization
  copying build/lib/apex/normalization/__init__.py -> build/bdist.linux-x86_64/wheel/apex/normalization
  copying build/lib/apex/normalization/fused_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/normalization
  creating build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/multiproc.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/LARC.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/optimized_sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/optimized_sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  copying build/lib/apex/parallel/distributed.py -> build/bdist.linux-x86_64/wheel/apex/parallel
  creating build/bdist.linux-x86_64/wheel/apex/contrib
  creating build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  copying build/lib/apex/contrib/cudnn_gbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  copying build/lib/apex/contrib/cudnn_gbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  copying build/lib/apex/contrib/index_mul_2d/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  copying build/lib/apex/contrib/index_mul_2d/index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d
  creating build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/focal_loss/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/focal_loss/focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss
  copying build/lib/apex/contrib/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib
  creating build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  copying build/lib/apex/contrib/xentropy/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  copying build/lib/apex/contrib/xentropy/softmax_xentropy.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy
  creating build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/peer_halo_exchanger_1d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  copying build/lib/apex/contrib/peer_memory/peer_memory.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory
  creating build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  copying build/lib/apex/contrib/conv_bias_relu/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  copying build/lib/apex/contrib/conv_bias_relu/conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu
  creating build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  copying build/lib/apex/contrib/fmha/fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  copying build/lib/apex/contrib/fmha/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha
  creating build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/_transducer_ref.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  copying build/lib/apex/contrib/transducer/transducer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer
  creating build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/test.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/bottleneck.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  copying build/lib/apex/contrib/bottleneck/halo_exchangers.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck
  creating build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  copying build/lib/apex/contrib/layer_norm/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  copying build/lib/apex/contrib/layer_norm/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm
  creating build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  copying build/lib/apex/contrib/groupbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  copying build/lib/apex/contrib/groupbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/channel_swap.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels
  copying build/lib/apex/contrib/sparsity/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/permutation_lib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/asp.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  copying build/lib/apex/contrib/sparsity/sparse_masklib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  copying build/lib/apex/contrib/test/cudnn_gbn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  copying build/lib/apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  copying build/lib/apex/contrib/test/index_mul_2d/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  copying build/lib/apex/contrib/test/index_mul_2d/test_index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/focal_loss/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/focal_loss/test_focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss
  copying build/lib/apex/contrib/test/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  copying build/lib/apex/contrib/test/xentropy/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  copying build/lib/apex/contrib/test/xentropy/test_label_smoothing.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  copying build/lib/apex/contrib/test/peer_memory/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  copying build/lib/apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  copying build/lib/apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  copying build/lib/apex/contrib/test/conv_bias_relu/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  copying build/lib/apex/contrib/test/fmha/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  copying build/lib/apex/contrib/test/fmha/test_fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/test_transducer_joint.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  copying build/lib/apex/contrib/test/transducer/test_transducer_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  copying build/lib/apex/contrib/test/bottleneck/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  copying build/lib/apex/contrib/test/bottleneck/test_bottleneck_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  copying build/lib/apex/contrib/test/layer_norm/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  copying build/lib/apex/contrib/test/layer_norm/test_fast_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/test_distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  copying build/lib/apex/contrib/test/optimizers/test_dist_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  copying build/lib/apex/contrib/test/clip_grad/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  copying build/lib/apex/contrib/test/clip_grad/test_clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad
  creating build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_mha_fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  copying build/lib/apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn
  creating build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/distributed_fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  copying build/lib/apex/contrib/optimizers/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  copying build/lib/apex/contrib/clip_grad/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  copying build/lib/apex/contrib/clip_grad/clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad
  creating build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/__init__.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/mask_softmax_dropout_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn
  creating build/bdist.linux-x86_64/wheel/apex/fused_dense
  copying build/lib/apex/fused_dense/__init__.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense
  copying build/lib/apex/fused_dense/fused_dense.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense
  creating build/bdist.linux-x86_64/wheel/apex/mlp
  copying build/lib/apex/mlp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/mlp
  copying build/lib/apex/mlp/mlp.py -> build/bdist.linux-x86_64/wheel/apex/mlp
  creating build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/loss_scaler.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/__init__.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  copying build/lib/apex/fp16_utils/fp16util.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils
  creating build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_mixed_precision_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_adagrad.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  copying build/lib/apex/optimizers/fused_novograd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers
  creating build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/amp/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/amp/grad_scaler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp
  copying build/lib/apex/transformer/_ucc_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/parallel_state.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/common.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules
  copying build/lib/apex/transformer/pipeline_parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/p2p_communication.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/pipeline_parallel/_timers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel
  copying build/lib/apex/transformer/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/layers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/cross_entropy.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/mappings.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/data.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/memory.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  copying build/lib/apex/transformer/tensor_parallel/random.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel
  creating build/bdist.linux-x86_64/wheel/apex/transformer/_data
  copying build/lib/apex/transformer/_data/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data
  copying build/lib/apex/transformer/_data/_batchsampler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data
  creating build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/layers/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/layers/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers
  copying build/lib/apex/transformer/enums.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  copying build/lib/apex/transformer/microbatches.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/functional/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/functional/fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional
  copying build/lib/apex/transformer/log_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer
  creating build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/__init__.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_transformer_lm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/commons.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/global_vars.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_bert.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/distributed_test_base.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/arguments.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  copying build/lib/apex/transformer/testing/standalone_gpt.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing
  creating build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/RNNBackend.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/cells.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/__init__.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/RNN/models.py -> build/bdist.linux-x86_64/wheel/apex/RNN
  copying build/lib/apex/_autocast_utils.py -> build/bdist.linux-x86_64/wheel/apex
  creating build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  copying build/lib/apex/multi_tensor_apply/__init__.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  copying build/lib/apex/multi_tensor_apply/multi_tensor_apply.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply
  running install_egg_info
  running egg_info
  writing apex.egg-info/PKG-INFO
  writing dependency_links to apex.egg-info/dependency_links.txt
  writing requirements to apex.egg-info/requires.txt
  writing top-level names to apex.egg-info/top_level.txt
  adding license file 'LICENSE' (matched pattern 'LICEN[CS]E*')
  reading manifest file 'apex.egg-info/SOURCES.txt'
  writing manifest file 'apex.egg-info/SOURCES.txt'
  Copying apex.egg-info to build/bdist.linux-x86_64/wheel/apex-0.1-py3.8.egg-info
  running install_scripts
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  creating build/bdist.linux-x86_64/wheel/apex-0.1.dist-info/WHEEL
  creating '/tmp/pip-wheel-6bsifi7_/tmp6gur0gc5/apex-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
  adding 'apex/__init__.py'
  adding 'apex/_autocast_utils.py'
  adding 'apex/RNN/RNNBackend.py'
  adding 'apex/RNN/__init__.py'
  adding 'apex/RNN/cells.py'
  adding 'apex/RNN/models.py'
  adding 'apex/amp/__init__.py'
  adding 'apex/amp/__version__.py'
  adding 'apex/amp/_amp_state.py'
  adding 'apex/amp/_initialize.py'
  adding 'apex/amp/_process_optimizer.py'
  adding 'apex/amp/amp.py'
  adding 'apex/amp/compat.py'
  adding 'apex/amp/frontend.py'
  adding 'apex/amp/handle.py'
  adding 'apex/amp/opt.py'
  adding 'apex/amp/rnn_compat.py'
  adding 'apex/amp/scaler.py'
  adding 'apex/amp/utils.py'
  adding 'apex/amp/wrap.py'
  adding 'apex/amp/lists/__init__.py'
  adding 'apex/amp/lists/functional_overrides.py'
  adding 'apex/amp/lists/tensor_overrides.py'
  adding 'apex/amp/lists/torch_overrides.py'
  adding 'apex/contrib/__init__.py'
  adding 'apex/contrib/bottleneck/__init__.py'
  adding 'apex/contrib/bottleneck/bottleneck.py'
  adding 'apex/contrib/bottleneck/halo_exchangers.py'
  adding 'apex/contrib/bottleneck/test.py'
  adding 'apex/contrib/clip_grad/__init__.py'
  adding 'apex/contrib/clip_grad/clip_grad.py'
  adding 'apex/contrib/conv_bias_relu/__init__.py'
  adding 'apex/contrib/conv_bias_relu/conv_bias_relu.py'
  adding 'apex/contrib/cudnn_gbn/__init__.py'
  adding 'apex/contrib/cudnn_gbn/batch_norm.py'
  adding 'apex/contrib/fmha/__init__.py'
  adding 'apex/contrib/fmha/fmha.py'
  adding 'apex/contrib/focal_loss/__init__.py'
  adding 'apex/contrib/focal_loss/focal_loss.py'
  adding 'apex/contrib/groupbn/__init__.py'
  adding 'apex/contrib/groupbn/batch_norm.py'
  adding 'apex/contrib/index_mul_2d/__init__.py'
  adding 'apex/contrib/index_mul_2d/index_mul_2d.py'
  adding 'apex/contrib/layer_norm/__init__.py'
  adding 'apex/contrib/layer_norm/layer_norm.py'
  adding 'apex/contrib/multihead_attn/__init__.py'
  adding 'apex/contrib/multihead_attn/encdec_multihead_attn.py'
  adding 'apex/contrib/multihead_attn/encdec_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py'
  adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_func.py'
  adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py'
  adding 'apex/contrib/multihead_attn/mask_softmax_dropout_func.py'
  adding 'apex/contrib/multihead_attn/self_multihead_attn.py'
  adding 'apex/contrib/multihead_attn/self_multihead_attn_func.py'
  adding 'apex/contrib/optimizers/__init__.py'
  adding 'apex/contrib/optimizers/distributed_fused_adam.py'
  adding 'apex/contrib/optimizers/distributed_fused_lamb.py'
  adding 'apex/contrib/optimizers/fp16_optimizer.py'
  adding 'apex/contrib/optimizers/fused_adam.py'
  adding 'apex/contrib/optimizers/fused_lamb.py'
  adding 'apex/contrib/optimizers/fused_sgd.py'
  adding 'apex/contrib/peer_memory/__init__.py'
  adding 'apex/contrib/peer_memory/peer_halo_exchanger_1d.py'
  adding 'apex/contrib/peer_memory/peer_memory.py'
  adding 'apex/contrib/sparsity/__init__.py'
  adding 'apex/contrib/sparsity/asp.py'
  adding 'apex/contrib/sparsity/permutation_lib.py'
  adding 'apex/contrib/sparsity/sparse_masklib.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/__init__.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/channel_swap.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py'
  adding 'apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py'
  adding 'apex/contrib/test/__init__.py'
  adding 'apex/contrib/test/bottleneck/__init__.py'
  adding 'apex/contrib/test/bottleneck/test_bottleneck_module.py'
  adding 'apex/contrib/test/clip_grad/__init__.py'
  adding 'apex/contrib/test/clip_grad/test_clip_grad.py'
  adding 'apex/contrib/test/conv_bias_relu/__init__.py'
  adding 'apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py'
  adding 'apex/contrib/test/cudnn_gbn/__init__.py'
  adding 'apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py'
  adding 'apex/contrib/test/fmha/__init__.py'
  adding 'apex/contrib/test/fmha/test_fmha.py'
  adding 'apex/contrib/test/focal_loss/__init__.py'
  adding 'apex/contrib/test/focal_loss/test_focal_loss.py'
  adding 'apex/contrib/test/index_mul_2d/__init__.py'
  adding 'apex/contrib/test/index_mul_2d/test_index_mul_2d.py'
  adding 'apex/contrib/test/layer_norm/__init__.py'
  adding 'apex/contrib/test/layer_norm/test_fast_layer_norm.py'
  adding 'apex/contrib/test/multihead_attn/__init__.py'
  adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py'
  adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py'
  adding 'apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py'
  adding 'apex/contrib/test/multihead_attn/test_mha_fused_softmax.py'
  adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn.py'
  adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py'
  adding 'apex/contrib/test/optimizers/__init__.py'
  adding 'apex/contrib/test/optimizers/test_dist_adam.py'
  adding 'apex/contrib/test/optimizers/test_distributed_fused_lamb.py'
  adding 'apex/contrib/test/peer_memory/__init__.py'
  adding 'apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py'
  adding 'apex/contrib/test/transducer/__init__.py'
  adding 'apex/contrib/test/transducer/test_transducer_joint.py'
  adding 'apex/contrib/test/transducer/test_transducer_loss.py'
  adding 'apex/contrib/test/xentropy/__init__.py'
  adding 'apex/contrib/test/xentropy/test_label_smoothing.py'
  adding 'apex/contrib/transducer/__init__.py'
  adding 'apex/contrib/transducer/_transducer_ref.py'
  adding 'apex/contrib/transducer/transducer.py'
  adding 'apex/contrib/xentropy/__init__.py'
  adding 'apex/contrib/xentropy/softmax_xentropy.py'
  adding 'apex/fp16_utils/__init__.py'
  adding 'apex/fp16_utils/fp16_optimizer.py'
  adding 'apex/fp16_utils/fp16util.py'
  adding 'apex/fp16_utils/loss_scaler.py'
  adding 'apex/fused_dense/__init__.py'
  adding 'apex/fused_dense/fused_dense.py'
  adding 'apex/mlp/__init__.py'
  adding 'apex/mlp/mlp.py'
  adding 'apex/multi_tensor_apply/__init__.py'
  adding 'apex/multi_tensor_apply/multi_tensor_apply.py'
  adding 'apex/normalization/__init__.py'
  adding 'apex/normalization/fused_layer_norm.py'
  adding 'apex/optimizers/__init__.py'
  adding 'apex/optimizers/fused_adagrad.py'
  adding 'apex/optimizers/fused_adam.py'
  adding 'apex/optimizers/fused_lamb.py'
  adding 'apex/optimizers/fused_mixed_precision_lamb.py'
  adding 'apex/optimizers/fused_novograd.py'
  adding 'apex/optimizers/fused_sgd.py'
  adding 'apex/parallel/LARC.py'
  adding 'apex/parallel/__init__.py'
  adding 'apex/parallel/distributed.py'
  adding 'apex/parallel/multiproc.py'
  adding 'apex/parallel/optimized_sync_batchnorm.py'
  adding 'apex/parallel/optimized_sync_batchnorm_kernel.py'
  adding 'apex/parallel/sync_batchnorm.py'
  adding 'apex/parallel/sync_batchnorm_kernel.py'
  adding 'apex/transformer/__init__.py'
  adding 'apex/transformer/_ucc_util.py'
  adding 'apex/transformer/enums.py'
  adding 'apex/transformer/log_util.py'
  adding 'apex/transformer/microbatches.py'
  adding 'apex/transformer/parallel_state.py'
  adding 'apex/transformer/utils.py'
  adding 'apex/transformer/_data/__init__.py'
  adding 'apex/transformer/_data/_batchsampler.py'
  adding 'apex/transformer/amp/__init__.py'
  adding 'apex/transformer/amp/grad_scaler.py'
  adding 'apex/transformer/functional/__init__.py'
  adding 'apex/transformer/functional/fused_softmax.py'
  adding 'apex/transformer/layers/__init__.py'
  adding 'apex/transformer/layers/layer_norm.py'
  adding 'apex/transformer/pipeline_parallel/__init__.py'
  adding 'apex/transformer/pipeline_parallel/_timers.py'
  adding 'apex/transformer/pipeline_parallel/p2p_communication.py'
  adding 'apex/transformer/pipeline_parallel/utils.py'
  adding 'apex/transformer/pipeline_parallel/schedules/__init__.py'
  adding 'apex/transformer/pipeline_parallel/schedules/common.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py'
  adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py'
  adding 'apex/transformer/tensor_parallel/__init__.py'
  adding 'apex/transformer/tensor_parallel/cross_entropy.py'
  adding 'apex/transformer/tensor_parallel/data.py'
  adding 'apex/transformer/tensor_parallel/layers.py'
  adding 'apex/transformer/tensor_parallel/mappings.py'
  adding 'apex/transformer/tensor_parallel/memory.py'
  adding 'apex/transformer/tensor_parallel/random.py'
  adding 'apex/transformer/tensor_parallel/utils.py'
  adding 'apex/transformer/testing/__init__.py'
  adding 'apex/transformer/testing/arguments.py'
  adding 'apex/transformer/testing/commons.py'
  adding 'apex/transformer/testing/distributed_test_base.py'
  adding 'apex/transformer/testing/global_vars.py'
  adding 'apex/transformer/testing/standalone_bert.py'
  adding 'apex/transformer/testing/standalone_gpt.py'
  adding 'apex/transformer/testing/standalone_transformer_lm.py'
  adding 'apex-0.1.dist-info/LICENSE'
  adding 'apex-0.1.dist-info/METADATA'
  adding 'apex-0.1.dist-info/WHEEL'
  adding 'apex-0.1.dist-info/top_level.txt'
  adding 'apex-0.1.dist-info/RECORD'
  removing build/bdist.linux-x86_64/wheel
done
  Created wheel for apex: filename=apex-0.1-py3-none-any.whl size=361346 sha256=bcb7f1a307c282a024de732f6c2ebbcff4c00807a5c27503b53f6f6ab70205b4
  Stored in directory: /tmp/pip-ephem-wheel-cache-3gyf6bgs/wheels/0e/9e/61/20762dce2ad2deec46512626627c2fe5487fc7306c71d6cfc8
Successfully built apex
Installing collected packages: apex
Successfully installed apex-0.1

And this hasn't built amp_C or apex_C. @loadams I have download the torch 1.13.1+cu116 i also pull the apex code in you repository but i can't install the apex can you help me the lasest error is
Processing /root/python_projects/PKD-for-BERT-Model-Compression-master/apex Running command pip subprocess to install build dependencies Collecting setuptools Using cached setuptools-68.2.2-py3-none-any.whl Collecting wheel Using cached wheel-0.41.2-py3-none-any.whl Installing collected packages: wheel, setuptools Successfully installed setuptools-68.2.2 wheel-0.41.2 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Installing build dependencies ... done Running command Getting requirements to build wheel Traceback (most recent call last): File "/root/anaconda3/envs/apex/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in main() File "/root/anaconda3/envs/apex/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/root/anaconda3/envs/apex/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel return hook(config_settings) File "/tmp/pip-build-env-tacnu7mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "/tmp/pip-build-env-tacnu7mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires self.run_setup() File "/tmp/pip-build-env-tacnu7mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 341, in run_setup exec(code, locals()) File "", line 4, in ModuleNotFoundError: No module named 'packaging' error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. full command: /root/anaconda3/envs/apex/bin/python /root/anaconda3/envs/apex/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmplkdqmays cwd: /root/python_projects/PKD-for-BERT-Model-Compression-master/apex Getting requirements to build wheel ... error error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

yuhuai4554 commented 10 months ago

pip install -v --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" --global-option="--xentropy" --global-option="--fast_multihead_attn" ./ it's works for me!

May I know what versions of Torch, CUDA, and Python you have?

romantic-black commented 8 months ago

pip install -v --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" --global-option="--xentropy" --global-option="--fast_multihead_attn" ./ it's works for me!

torch 1.9.0+cu111 cuda 11.3 python 3.9 this work for me

yuhuai4554 commented 8 months ago

Thank you for your advice. I'll try it later.

1589210472 @.***

 

------------------ 原始邮件 ------------------ 发件人: "NVIDIA/apex" @.>; 发送时间: 2023年11月2日(星期四) 晚上10:17 @.>; @.**@.>; 主题: Re: [NVIDIA/apex] pyproject.toml missing packaging dependency (Issue #1679)

pip install -v --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" --global-option="--xentropy" --global-option="--fast_multihead_attn" ./ it's works for me!

torch 1.9.0+cu111 cuda 11.3 this work for me

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Alwen-V commented 7 months ago

pip install -v --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" --global-option="--deprecated_fused_adam" --global-option="--xentropy" --global-option="--fast_multihead_attn" ./ 这对我有用！

it works for me!! thanks~

nebulanomad1134 commented 5 months ago

So, what I did was clone the repo and checkout to an older commit (something around April), and install it the regular way. Once it is successfully installed, I just git pull the latest changes.

@VarunGumma do you still get the warning: "amp_C fused kernels unavailable" when using fairseq-train?

Taskii-Lei commented 4 months ago

But when I run the : 

pip install -v --no-build-isolation --config-settings --build-option="--cpp_ext --cuda_ext --bnp --xentropy ..." ./

New error raises like below:

File "<string>", line 137, in <module>
    File "<string>", line 24, in get_cuda_bare_metal_version
  TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
  error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /lustre/S/huanglei/CondaEnv/Baichuan/bin/python /lustre/S/huanglei/CondaEnv/Baichuan/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpgof2lben
  cwd: /lustre/S/huanglei/apex
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
loadams commented 4 months ago

@Taskii-Lei - that looks like a new/different error, I'd recommend opening a new issue for that.

Taskii-Lei commented 4 months ago

@Taskii-Lei - that looks like a new/different error, I'd recommend opening a new issue for that.

I have solved it. It raises because the cuda installed by conda is not complete, and there's no nvcc. The solution is to install the nvcc by:

conda install -c nvidia cuda-nvcc

and by the way, if still not ok, one can try:

git clone https://github.com/ptrblck/apex.git
cd apex
git checkout apex_no_distributed
pip install -v --no-cache-dir ./
hayatkhan8660-maker commented 1 month ago

If you add pytorch and packaging, that will resolve those issues but the overall installation (at least for me) is failing with another issue of not installing the cpp_ext or cuda_ext.

@loadams I have the exact same issue, I want to build apex with cpp_ext and cuda_ext for mixed precision training.

I am using the following command:

pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

but it simply does not works.

when running my code on multiple gpus i get the following error: Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")

Any will be appreciated!

hayatkhan8660-maker commented 1 month ago

Using the below commands:

git clone https://github.com/NVIDIA/apex
cd apex
nvidia-smi
conda install -c nvidia cuda-nvcc
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./

It install the apex-0.1 but still it does not built with cpp_ext and cuda_ext

Sun May 26 14:11:38 2024
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3090 On | 00000000:A3:00.0 Off | N/A | | 30% 27C P8 28W / 350W | 1MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ Retrieving notices: ...working... done Channels:

All requested packages already installed.

Using pip 24.0 from /homes/hayatu/miniconda3/envs/focal/lib/python3.8/site-packages/pip (python 3.8) DEPRECATION: --build-option and --global-option are deprecated. pip 24.2 will enforce this behaviour change. A possible replacement is to use --config-settings. Discussion can be found at https://github.com/pypa/pip/issues/11859 WARNING: Implying --no-binary=:all: due to the presence of --build-option / --global-option. Processing /homes/hayatu/Video-FocalNets/apex Preparing metadata (pyproject.toml): started Running command Preparing metadata (pyproject.toml)

torch.version = 2.3.0+cu121

running dist_info creating /tmp/pip-modern-metadata-iljntb5s/apex.egg-info writing /tmp/pip-modern-metadata-iljntb5s/apex.egg-info/PKG-INFO writing dependency_links to /tmp/pip-modern-metadata-iljntb5s/apex.egg-info/dependency_links.txt writing requirements to /tmp/pip-modern-metadata-iljntb5s/apex.egg-info/requires.txt writing top-level names to /tmp/pip-modern-metadata-iljntb5s/apex.egg-info/top_level.txt writing manifest file '/tmp/pip-modern-metadata-iljntb5s/apex.egg-info/SOURCES.txt' reading manifest file '/tmp/pip-modern-metadata-iljntb5s/apex.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file '/tmp/pip-modern-metadata-iljntb5s/apex.egg-info/SOURCES.txt' creating '/tmp/pip-modern-metadata-iljntb5s/apex-0.1.dist-info' Preparing metadata (pyproject.toml): finished with status 'done' Requirement already satisfied: packaging>20.6 in /homes/hayatu/miniconda3/envs/focal/lib/python3.8/site-packages (from apex==0.1) (24.0) Building wheels for collected packages: apex WARNING: Ignoring --global-option when building apex using PEP 517 Building wheel for apex (pyproject.toml): started Running command Building wheel for apex (pyproject.toml)

torch.version = 2.3.0+cu121

running bdist_wheel running build running build_py installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/apex creating build/bdist.linux-x86_64/wheel/apex/normalization copying build/lib/apex/normalization/init.py -> build/bdist.linux-x86_64/wheel/apex/normalization copying build/lib/apex/normalization/fused_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/normalization creating build/bdist.linux-x86_64/wheel/apex/transformer copying build/lib/apex/transformer/log_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer creating build/bdist.linux-x86_64/wheel/apex/transformer/amp copying build/lib/apex/transformer/amp/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp copying build/lib/apex/transformer/amp/grad_scaler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/amp copying build/lib/apex/transformer/parallel_state.py -> build/bdist.linux-x86_64/wheel/apex/transformer creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel copying build/lib/apex/transformer/pipeline_parallel/p2p_communication.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel creating build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules copying build/lib/apex/transformer/pipeline_parallel/schedules/common.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules copying build/lib/apex/transformer/pipeline_parallel/schedules/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules copying build/lib/apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel/schedules copying build/lib/apex/transformer/pipeline_parallel/_timers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel copying build/lib/apex/transformer/pipeline_parallel/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel copying build/lib/apex/transformer/pipeline_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/pipeline_parallel copying build/lib/apex/transformer/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer copying build/lib/apex/transformer/enums.py -> build/bdist.linux-x86_64/wheel/apex/transformer creating build/bdist.linux-x86_64/wheel/apex/transformer/_data copying build/lib/apex/transformer/_data/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data copying build/lib/apex/transformer/_data/_batchsampler.py -> build/bdist.linux-x86_64/wheel/apex/transformer/_data creating build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/distributed_test_base.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/standalone_gpt.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/standalone_transformer_lm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/global_vars.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/standalone_bert.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/commons.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/testing/arguments.py -> build/bdist.linux-x86_64/wheel/apex/transformer/testing copying build/lib/apex/transformer/_ucc_util.py -> build/bdist.linux-x86_64/wheel/apex/transformer copying build/lib/apex/transformer/microbatches.py -> build/bdist.linux-x86_64/wheel/apex/transformer copying build/lib/apex/transformer/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer creating build/bdist.linux-x86_64/wheel/apex/transformer/functional copying build/lib/apex/transformer/functional/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional copying build/lib/apex/transformer/functional/fused_rope.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional copying build/lib/apex/transformer/functional/fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/transformer/functional creating build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/memory.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/data.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/utils.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/random.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/cross_entropy.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/mappings.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel copying build/lib/apex/transformer/tensor_parallel/layers.py -> build/bdist.linux-x86_64/wheel/apex/transformer/tensor_parallel creating build/bdist.linux-x86_64/wheel/apex/transformer/layers copying build/lib/apex/transformer/layers/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers copying build/lib/apex/transformer/layers/init.py -> build/bdist.linux-x86_64/wheel/apex/transformer/layers creating build/bdist.linux-x86_64/wheel/apex/fused_dense copying build/lib/apex/fused_dense/fused_dense.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense copying build/lib/apex/fused_dense/init.py -> build/bdist.linux-x86_64/wheel/apex/fused_dense creating build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/init.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/_process_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/handle.py -> build/bdist.linux-x86_64/wheel/apex/amp creating build/bdist.linux-x86_64/wheel/apex/amp/lists copying build/lib/apex/amp/lists/torch_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists copying build/lib/apex/amp/lists/init.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists copying build/lib/apex/amp/lists/functional_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists copying build/lib/apex/amp/lists/tensor_overrides.py -> build/bdist.linux-x86_64/wheel/apex/amp/lists copying build/lib/apex/amp/compat.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/rnn_compat.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/frontend.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/scaler.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/utils.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/opt.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/wrap.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/version.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/amp.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/_amp_state.py -> build/bdist.linux-x86_64/wheel/apex/amp copying build/lib/apex/amp/_initialize.py -> build/bdist.linux-x86_64/wheel/apex/amp creating build/bdist.linux-x86_64/wheel/apex/mlp copying build/lib/apex/mlp/init.py -> build/bdist.linux-x86_64/wheel/apex/mlp copying build/lib/apex/mlp/mlp.py -> build/bdist.linux-x86_64/wheel/apex/mlp copying build/lib/apex/init.py -> build/bdist.linux-x86_64/wheel/apex creating build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/distributed.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/init.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/optimized_sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/multiproc.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/optimized_sync_batchnorm_kernel.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/sync_batchnorm.py -> build/bdist.linux-x86_64/wheel/apex/parallel copying build/lib/apex/parallel/LARC.py -> build/bdist.linux-x86_64/wheel/apex/parallel creating build/bdist.linux-x86_64/wheel/apex/fp16_utils copying build/lib/apex/fp16_utils/fp16util.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils copying build/lib/apex/fp16_utils/init.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils copying build/lib/apex/fp16_utils/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils copying build/lib/apex/fp16_utils/loss_scaler.py -> build/bdist.linux-x86_64/wheel/apex/fp16_utils creating build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/init.py -> build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/fused_novograd.py -> build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/fused_adagrad.py -> build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers copying build/lib/apex/optimizers/fused_mixed_precision_lamb.py -> build/bdist.linux-x86_64/wheel/apex/optimizers creating build/bdist.linux-x86_64/wheel/apex/RNN copying build/lib/apex/RNN/init.py -> build/bdist.linux-x86_64/wheel/apex/RNN copying build/lib/apex/RNN/cells.py -> build/bdist.linux-x86_64/wheel/apex/RNN copying build/lib/apex/RNN/RNNBackend.py -> build/bdist.linux-x86_64/wheel/apex/RNN copying build/lib/apex/RNN/models.py -> build/bdist.linux-x86_64/wheel/apex/RNN copying build/lib/apex/_autocast_utils.py -> build/bdist.linux-x86_64/wheel/apex creating build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply copying build/lib/apex/multi_tensor_apply/init.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply copying build/lib/apex/multi_tensor_apply/multi_tensor_apply.py -> build/bdist.linux-x86_64/wheel/apex/multi_tensor_apply creating build/bdist.linux-x86_64/wheel/apex/contrib creating build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck copying build/lib/apex/contrib/bottleneck/bottleneck.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck copying build/lib/apex/contrib/bottleneck/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck copying build/lib/apex/contrib/bottleneck/halo_exchangers.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck copying build/lib/apex/contrib/bottleneck/test.py -> build/bdist.linux-x86_64/wheel/apex/contrib/bottleneck creating build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu copying build/lib/apex/contrib/conv_bias_relu/conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu copying build/lib/apex/contrib/conv_bias_relu/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/conv_bias_relu creating build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d copying build/lib/apex/contrib/index_mul_2d/index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d copying build/lib/apex/contrib/index_mul_2d/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/index_mul_2d creating build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/mask_softmax_dropout_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn copying build/lib/apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/bdist.linux-x86_64/wheel/apex/contrib/multihead_attn creating build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory copying build/lib/apex/contrib/peer_memory/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory copying build/lib/apex/contrib/peer_memory/peer_halo_exchanger_1d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory copying build/lib/apex/contrib/peer_memory/peer_memory.py -> build/bdist.linux-x86_64/wheel/apex/contrib/peer_memory creating build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm copying build/lib/apex/contrib/layer_norm/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm copying build/lib/apex/contrib/layer_norm/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/layer_norm creating build/bdist.linux-x86_64/wheel/apex/contrib/gpu_direct_storage copying build/lib/apex/contrib/gpu_direct_storage/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/gpu_direct_storage creating build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad copying build/lib/apex/contrib/clip_grad/clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad copying build/lib/apex/contrib/clip_grad/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/clip_grad creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity copying build/lib/apex/contrib/sparsity/asp.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity copying build/lib/apex/contrib/sparsity/permutation_lib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity copying build/lib/apex/contrib/sparsity/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity copying build/lib/apex/contrib/sparsity/sparse_masklib.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity creating build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels copying build/lib/apex/contrib/sparsity/permutation_search_kernels/channel_swap.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels copying build/lib/apex/contrib/sparsity/permutation_search_kernels/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels copying build/lib/apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels copying build/lib/apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels copying build/lib/apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py -> build/bdist.linux-x86_64/wheel/apex/contrib/sparsity/permutation_search_kernels copying build/lib/apex/contrib/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib creating build/bdist.linux-x86_64/wheel/apex/contrib/transducer copying build/lib/apex/contrib/transducer/transducer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer copying build/lib/apex/contrib/transducer/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer copying build/lib/apex/contrib/transducer/_transducer_ref.py -> build/bdist.linux-x86_64/wheel/apex/contrib/transducer creating build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/fp16_optimizer.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/fused_sgd.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/distributed_fused_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers copying build/lib/apex/contrib/optimizers/fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/optimizers creating build/bdist.linux-x86_64/wheel/apex/contrib/fmha copying build/lib/apex/contrib/fmha/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha copying build/lib/apex/contrib/fmha/fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/fmha creating build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn copying build/lib/apex/contrib/cudnn_gbn/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn copying build/lib/apex/contrib/cudnn_gbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/cudnn_gbn creating build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss copying build/lib/apex/contrib/focal_loss/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss copying build/lib/apex/contrib/focal_loss/focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/focal_loss creating build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/_layer_norm_forward_kernels.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/mha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/_layer_norm_config_ampere.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/fused_adam_swa.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/_layer_norm_backward_kernels.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/_layer_norm_config_hopper.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton copying build/lib/apex/contrib/openfold_triton/_mha_kernel.py -> build/bdist.linux-x86_64/wheel/apex/contrib/openfold_triton creating build/bdist.linux-x86_64/wheel/apex/contrib/nccl_allocator copying build/lib/apex/contrib/nccl_allocator/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/nccl_allocator copying build/lib/apex/contrib/nccl_allocator/nccl_allocator.py -> build/bdist.linux-x86_64/wheel/apex/contrib/nccl_allocator creating build/bdist.linux-x86_64/wheel/apex/contrib/groupbn copying build/lib/apex/contrib/groupbn/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn copying build/lib/apex/contrib/groupbn/batch_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/groupbn creating build/bdist.linux-x86_64/wheel/apex/contrib/group_norm copying build/lib/apex/contrib/group_norm/group_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/group_norm copying build/lib/apex/contrib/group_norm/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/group_norm creating build/bdist.linux-x86_64/wheel/apex/contrib/test creating build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck copying build/lib/apex/contrib/test/bottleneck/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck copying build/lib/apex/contrib/test/bottleneck/test_bottleneck_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/bottleneck creating build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu copying build/lib/apex/contrib/test/conv_bias_relu/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu copying build/lib/apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/conv_bias_relu creating build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d copying build/lib/apex/contrib/test/index_mul_2d/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d copying build/lib/apex/contrib/test/index_mul_2d/test_index_mul_2d.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/index_mul_2d creating build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/test_mha_fused_softmax.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn copying build/lib/apex/contrib/test/multihead_attn/test_self_multihead_attn.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/multihead_attn creating build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory copying build/lib/apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory copying build/lib/apex/contrib/test/peer_memory/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/peer_memory creating build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm copying build/lib/apex/contrib/test/layer_norm/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm copying build/lib/apex/contrib/test/layer_norm/test_fast_layer_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/layer_norm creating build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad copying build/lib/apex/contrib/test/clip_grad/test_clip_grad.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad copying build/lib/apex/contrib/test/clip_grad/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/clip_grad copying build/lib/apex/contrib/test/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test creating build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer copying build/lib/apex/contrib/test/transducer/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer copying build/lib/apex/contrib/test/transducer/test_transducer_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer copying build/lib/apex/contrib/test/transducer/test_transducer_joint.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/transducer creating build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers copying build/lib/apex/contrib/test/optimizers/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers copying build/lib/apex/contrib/test/optimizers/test_distributed_fused_lamb.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers copying build/lib/apex/contrib/test/optimizers/test_dist_adam.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/optimizers creating build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha copying build/lib/apex/contrib/test/fmha/test_fmha.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha copying build/lib/apex/contrib/test/fmha/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/fmha creating build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn copying build/lib/apex/contrib/test/cudnn_gbn/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn copying build/lib/apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/cudnn_gbn creating build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss copying build/lib/apex/contrib/test/focal_loss/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss copying build/lib/apex/contrib/test/focal_loss/test_focal_loss.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/focal_loss creating build/bdist.linux-x86_64/wheel/apex/contrib/test/group_norm copying build/lib/apex/contrib/test/group_norm/test_group_norm.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/group_norm copying build/lib/apex/contrib/test/group_norm/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/group_norm creating build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy copying build/lib/apex/contrib/test/xentropy/test_label_smoothing.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy copying build/lib/apex/contrib/test/xentropy/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/test/xentropy creating build/bdist.linux-x86_64/wheel/apex/contrib/xentropy copying build/lib/apex/contrib/xentropy/softmax_xentropy.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy copying build/lib/apex/contrib/xentropy/init.py -> build/bdist.linux-x86_64/wheel/apex/contrib/xentropy running install_egg_info running egg_info writing apex.egg-info/PKG-INFO writing dependency_links to apex.egg-info/dependency_links.txt writing requirements to apex.egg-info/requires.txt writing top-level names to apex.egg-info/top_level.txt reading manifest file 'apex.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'apex.egg-info/SOURCES.txt' Copying apex.egg-info to build/bdist.linux-x86_64/wheel/apex-0.1-py3.8.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/apex-0.1.dist-info/WHEEL creating '/tmp/pip-wheel-pysg0bfe/.tmp-nb5npm2i/apex-0.1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'apex/init.py' adding 'apex/_autocast_utils.py' adding 'apex/RNN/RNNBackend.py' adding 'apex/RNN/init.py' adding 'apex/RNN/cells.py' adding 'apex/RNN/models.py' adding 'apex/amp/init.py' adding 'apex/amp/version.py' adding 'apex/amp/_amp_state.py' adding 'apex/amp/_initialize.py' adding 'apex/amp/_process_optimizer.py' adding 'apex/amp/amp.py' adding 'apex/amp/compat.py' adding 'apex/amp/frontend.py' adding 'apex/amp/handle.py' adding 'apex/amp/opt.py' adding 'apex/amp/rnn_compat.py' adding 'apex/amp/scaler.py' adding 'apex/amp/utils.py' adding 'apex/amp/wrap.py' adding 'apex/amp/lists/init.py' adding 'apex/amp/lists/functional_overrides.py' adding 'apex/amp/lists/tensor_overrides.py' adding 'apex/amp/lists/torch_overrides.py' adding 'apex/contrib/init.py' adding 'apex/contrib/bottleneck/init.py' adding 'apex/contrib/bottleneck/bottleneck.py' adding 'apex/contrib/bottleneck/halo_exchangers.py' adding 'apex/contrib/bottleneck/test.py' adding 'apex/contrib/clip_grad/init.py' adding 'apex/contrib/clip_grad/clip_grad.py' adding 'apex/contrib/conv_bias_relu/init.py' adding 'apex/contrib/conv_bias_relu/conv_bias_relu.py' adding 'apex/contrib/cudnn_gbn/init.py' adding 'apex/contrib/cudnn_gbn/batch_norm.py' adding 'apex/contrib/fmha/init.py' adding 'apex/contrib/fmha/fmha.py' adding 'apex/contrib/focal_loss/init.py' adding 'apex/contrib/focal_loss/focal_loss.py' adding 'apex/contrib/gpu_direct_storage/init.py' adding 'apex/contrib/group_norm/init.py' adding 'apex/contrib/group_norm/group_norm.py' adding 'apex/contrib/groupbn/init.py' adding 'apex/contrib/groupbn/batch_norm.py' adding 'apex/contrib/index_mul_2d/init.py' adding 'apex/contrib/index_mul_2d/index_mul_2d.py' adding 'apex/contrib/layer_norm/init.py' adding 'apex/contrib/layer_norm/layer_norm.py' adding 'apex/contrib/multihead_attn/init.py' adding 'apex/contrib/multihead_attn/encdec_multihead_attn.py' adding 'apex/contrib/multihead_attn/encdec_multihead_attn_func.py' adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py' adding 'apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py' adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_func.py' adding 'apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py' adding 'apex/contrib/multihead_attn/mask_softmax_dropout_func.py' adding 'apex/contrib/multihead_attn/self_multihead_attn.py' adding 'apex/contrib/multihead_attn/self_multihead_attn_func.py' adding 'apex/contrib/nccl_allocator/init.py' adding 'apex/contrib/nccl_allocator/nccl_allocator.py' adding 'apex/contrib/openfold_triton/init.py' adding 'apex/contrib/openfold_triton/_layer_norm_backward_kernels.py' adding 'apex/contrib/openfold_triton/_layer_norm_config_ampere.py' adding 'apex/contrib/openfold_triton/_layer_norm_config_hopper.py' adding 'apex/contrib/openfold_triton/_layer_norm_forward_kernels.py' adding 'apex/contrib/openfold_triton/_mha_kernel.py' adding 'apex/contrib/openfold_triton/fused_adam_swa.py' adding 'apex/contrib/openfold_triton/layer_norm.py' adding 'apex/contrib/openfold_triton/mha.py' adding 'apex/contrib/optimizers/init.py' adding 'apex/contrib/optimizers/distributed_fused_adam.py' adding 'apex/contrib/optimizers/distributed_fused_lamb.py' adding 'apex/contrib/optimizers/fp16_optimizer.py' adding 'apex/contrib/optimizers/fused_adam.py' adding 'apex/contrib/optimizers/fused_lamb.py' adding 'apex/contrib/optimizers/fused_sgd.py' adding 'apex/contrib/peer_memory/init.py' adding 'apex/contrib/peer_memory/peer_halo_exchanger_1d.py' adding 'apex/contrib/peer_memory/peer_memory.py' adding 'apex/contrib/sparsity/init.py' adding 'apex/contrib/sparsity/asp.py' adding 'apex/contrib/sparsity/permutation_lib.py' adding 'apex/contrib/sparsity/sparse_masklib.py' adding 'apex/contrib/sparsity/permutation_search_kernels/init.py' adding 'apex/contrib/sparsity/permutation_search_kernels/call_permutation_search_kernels.py' adding 'apex/contrib/sparsity/permutation_search_kernels/channel_swap.py' adding 'apex/contrib/sparsity/permutation_search_kernels/exhaustive_search.py' adding 'apex/contrib/sparsity/permutation_search_kernels/permutation_utilities.py' adding 'apex/contrib/test/init.py' adding 'apex/contrib/test/bottleneck/init.py' adding 'apex/contrib/test/bottleneck/test_bottleneck_module.py' adding 'apex/contrib/test/clip_grad/init.py' adding 'apex/contrib/test/clip_grad/test_clip_grad.py' adding 'apex/contrib/test/conv_bias_relu/init.py' adding 'apex/contrib/test/conv_bias_relu/test_conv_bias_relu.py' adding 'apex/contrib/test/cudnn_gbn/init.py' adding 'apex/contrib/test/cudnn_gbn/test_cudnn_gbn_with_two_gpus.py' adding 'apex/contrib/test/fmha/init.py' adding 'apex/contrib/test/fmha/test_fmha.py' adding 'apex/contrib/test/focal_loss/init.py' adding 'apex/contrib/test/focal_loss/test_focal_loss.py' adding 'apex/contrib/test/group_norm/init.py' adding 'apex/contrib/test/group_norm/test_group_norm.py' adding 'apex/contrib/test/index_mul_2d/init.py' adding 'apex/contrib/test/index_mul_2d/test_index_mul_2d.py' adding 'apex/contrib/test/layer_norm/init.py' adding 'apex/contrib/test/layer_norm/test_fast_layer_norm.py' adding 'apex/contrib/test/multihead_attn/init.py' adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn.py' adding 'apex/contrib/test/multihead_attn/test_encdec_multihead_attn_norm_add.py' adding 'apex/contrib/test/multihead_attn/test_fast_self_multihead_attn_bias.py' adding 'apex/contrib/test/multihead_attn/test_mha_fused_softmax.py' adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn.py' adding 'apex/contrib/test/multihead_attn/test_self_multihead_attn_norm_add.py' adding 'apex/contrib/test/optimizers/init.py' adding 'apex/contrib/test/optimizers/test_dist_adam.py' adding 'apex/contrib/test/optimizers/test_distributed_fused_lamb.py' adding 'apex/contrib/test/peer_memory/init.py' adding 'apex/contrib/test/peer_memory/test_peer_halo_exchange_module.py' adding 'apex/contrib/test/transducer/init.py' adding 'apex/contrib/test/transducer/test_transducer_joint.py' adding 'apex/contrib/test/transducer/test_transducer_loss.py' adding 'apex/contrib/test/xentropy/init.py' adding 'apex/contrib/test/xentropy/test_label_smoothing.py' adding 'apex/contrib/transducer/init.py' adding 'apex/contrib/transducer/_transducer_ref.py' adding 'apex/contrib/transducer/transducer.py' adding 'apex/contrib/xentropy/init.py' adding 'apex/contrib/xentropy/softmax_xentropy.py' adding 'apex/fp16_utils/init.py' adding 'apex/fp16_utils/fp16_optimizer.py' adding 'apex/fp16_utils/fp16util.py' adding 'apex/fp16_utils/loss_scaler.py' adding 'apex/fused_dense/init.py' adding 'apex/fused_dense/fused_dense.py' adding 'apex/mlp/init.py' adding 'apex/mlp/mlp.py' adding 'apex/multi_tensor_apply/init.py' adding 'apex/multi_tensor_apply/multi_tensor_apply.py' adding 'apex/normalization/init.py' adding 'apex/normalization/fused_layer_norm.py' adding 'apex/optimizers/init.py' adding 'apex/optimizers/fused_adagrad.py' adding 'apex/optimizers/fused_adam.py' adding 'apex/optimizers/fused_lamb.py' adding 'apex/optimizers/fused_mixed_precision_lamb.py' adding 'apex/optimizers/fused_novograd.py' adding 'apex/optimizers/fused_sgd.py' adding 'apex/parallel/LARC.py' adding 'apex/parallel/init.py' adding 'apex/parallel/distributed.py' adding 'apex/parallel/multiproc.py' adding 'apex/parallel/optimized_sync_batchnorm.py' adding 'apex/parallel/optimized_sync_batchnorm_kernel.py' adding 'apex/parallel/sync_batchnorm.py' adding 'apex/parallel/sync_batchnorm_kernel.py' adding 'apex/transformer/init.py' adding 'apex/transformer/_ucc_util.py' adding 'apex/transformer/enums.py' adding 'apex/transformer/log_util.py' adding 'apex/transformer/microbatches.py' adding 'apex/transformer/parallel_state.py' adding 'apex/transformer/utils.py' adding 'apex/transformer/_data/init.py' adding 'apex/transformer/_data/_batchsampler.py' adding 'apex/transformer/amp/init.py' adding 'apex/transformer/amp/grad_scaler.py' adding 'apex/transformer/functional/init.py' adding 'apex/transformer/functional/fused_rope.py' adding 'apex/transformer/functional/fused_softmax.py' adding 'apex/transformer/layers/init.py' adding 'apex/transformer/layers/layer_norm.py' adding 'apex/transformer/pipeline_parallel/init.py' adding 'apex/transformer/pipeline_parallel/_timers.py' adding 'apex/transformer/pipeline_parallel/p2p_communication.py' adding 'apex/transformer/pipeline_parallel/utils.py' adding 'apex/transformer/pipeline_parallel/schedules/init.py' adding 'apex/transformer/pipeline_parallel/schedules/common.py' adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py' adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_with_interleaving.py' adding 'apex/transformer/pipeline_parallel/schedules/fwd_bwd_pipelining_without_interleaving.py' adding 'apex/transformer/tensor_parallel/init.py' adding 'apex/transformer/tensor_parallel/cross_entropy.py' adding 'apex/transformer/tensor_parallel/data.py' adding 'apex/transformer/tensor_parallel/layers.py' adding 'apex/transformer/tensor_parallel/mappings.py' adding 'apex/transformer/tensor_parallel/memory.py' adding 'apex/transformer/tensor_parallel/random.py' adding 'apex/transformer/tensor_parallel/utils.py' adding 'apex/transformer/testing/init.py' adding 'apex/transformer/testing/arguments.py' adding 'apex/transformer/testing/commons.py' adding 'apex/transformer/testing/distributed_test_base.py' adding 'apex/transformer/testing/global_vars.py' adding 'apex/transformer/testing/standalone_bert.py' adding 'apex/transformer/testing/standalone_gpt.py' adding 'apex/transformer/testing/standalone_transformer_lm.py' adding 'apex-0.1.dist-info/LICENSE' adding 'apex-0.1.dist-info/METADATA' adding 'apex-0.1.dist-info/WHEEL' adding 'apex-0.1.dist-info/top_level.txt' adding 'apex-0.1.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel Building wheel for apex (pyproject.toml): finished with status 'done' Created wheel for apex: filename=apex-0.1-py3-none-any.whl size=400504 sha256=d232a7002f725f1ec634700a257e122ea651c5032cd27f9718e1c77d19a559f0 Stored in directory: /tmp/pip-ephem-wheel-cache-90lnxh17/wheels/66/8c/3c/e5cb37c63d8f7e17ac11c36da73319a66f1cac5eeec36c8234 Successfully built apex Installing collected packages: apex Successfully installed apex-0.1

I am using HPC server, having PyTorch version 2.3.0+cu121 and CUDA 12.4.

I suspect this behavior might be caused by a mismatch between the installed CUDA version (12.4) on my server and the pre-compiled CUDA version (12.1) of PyTorch.

After installation, when I run my code I get the same warning:

Warning: multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback. Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")