Open brcisna opened 6 days ago
lw
commented
5 days ago Please post the entire error log, ideally as text. Otherwise we cannot help.
Also, please make sure you cloned all the submodules. Run git submodule update --init --recursive if you're not sure.
brcisna
commented
5 days ago This doing a build in a Python3.10.12 venv 'python setup.py build '
Yes.. I did do " git submodule update --init --recursive"
It appears all fails after the building process does not find the /thrust/complex.h file as evidenced in the error log. Installed PyTorch with pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.1
Error log
`/home/superuser/.pyenv/versions/wunjo/lib/python3.10/site-packages/torch/include/c10/util/complex.h:8:10: fatal error: 'thrust/complex.h' file not found
8 | #include <thrust/complex.h>
| ^~~~~~
1 error generated when compiling for gfx1032.
,,,,continued,,,,
/home/superuser/MyPrograms/wunjo/wunjo/portable/xformers/third_party/composable_kernel_tiled/include/ck_tile/ops/fmha/pipeline/block_fmha_bwd_pipeline_default_policy_hip.hpp:1118:16: note: in instantiation of function template specialization 'ck_tile::BlockFmhaBwdPipelineDefaultPolicy::MakeXTLdsBlockDescriptor<ck_tile::BlockFmhaBwdPipelineProblem<_Float16, _Float16, _Float16, _Float16, float, float, float, _Float16, unsigned short, _Float16, _Float16, _Float16, _Float16, _Float16, _Float16, FmhaBwdShape<32>, true, false, ck_tile::SimplifiedGenericAttentionMask<>, ck_tile::BlockDropoutBwd<false, true, false>, ck_tile::TileFmhaTraits<true, true, false, false, ck_tile::BlockAttentionBiasEnum::NO_BIAS, true, false, false, false, 1>>, 32, 32, 4, 2>' requested here
1118 | return MakeXTLdsBlockDescriptor<Problem, kNPerBlock, kKPerBlock, kKPack, kKPackT>();
| ^
/home/superuser/MyPrograms/wunjo/wunjo/portable/xformers/third_party/composable_kernel_tiled/include/ck_tile/ops/fmha/pipeline/block_fmha_bwd_pipeline_default_policy_hip.hpp:1606:13: note: in instantiation of function template specialization 'ck_tile::BlockFmhaBwdPipelineDefaultPolicy::MakeShuffledQLdsWriteBlockDescriptor<ck_tile::BlockFmhaBwdPipelineProblem<_Float16, _Float16, _Float16, _Float16, float, float, float, _Float16, unsigned short, _Float16, _Float16, _Float16, _Float16, _Float16, _Float16, FmhaBwdShape<32>, true, false, ck_tile::SimplifiedGenericAttentionMask<>, ck_tile::BlockDropoutBwd<false, true, false>, ck_tile::TileFmhaTraits<true, true, false, false, ck_tile::BlockAttentionBiasEnum::NO_BIAS, true, false, false, false, 1>>>' requested here
1606 | MakeShuffledQLdsWriteBlockDescriptor~~~~~~~~~~~~
/home/superuser/MyPrograms/wunjo/wunjo/portable/xformers/third_party/composable_kernel_tiled/include/ck_tile/ops/fmha/pipeline/block_fmha_bwd_pipeline_default_policy_hip.hpp:992:16: note: in instantiation of function template specialization 'ck_tile::BlockFmhaBwdPipelineDefaultPolicy::MakeXTLdsBlockDescriptor<ck_tile::BlockFmhaBwdPipelineProblem<_Float16, _Float16, _Float16, _Float16, float, float, float, _Float16, unsigned short, _Float16, _Float16, _Float16, _Float16, _Float16, _Float16, FmhaBwdShape<32>, true, false, ck_tile::SimplifiedGenericAttentionMask<>, ck_tile::BlockDropoutBwd<false, true, false>, ck_tile::TileFmhaTraits<true, true, false, false, ck_tile::BlockAttentionBiasEnum::NO_BIAS, true, false, false, false, 1>>, 32, 128, 8, 4>' requested here
992 | return MakeXTLdsBlockDescriptor<Problem, kNPerBlock, kKPerBlock, kKPack, kKPackT>();
| ^
/home/superuser/MyPrograms/wunjo/wunjo/portable/xformers/third_party/composable_kernel_tiled/include/ck_tile/ops/fmha/pipeline/block_fmha_bwd_pipeline_default_policy_hip.hpp:1001:42: note: in instantiation of function template specialization 'ck_tile::BlockFmhaBwdPipelineDefaultPolicy::MakeShuffledKLdsWriteBlockDescriptor<ck_tile::BlockFmhaBwdPipelineProblem<_Float16, _Float16, _Float16, _Float16, float, float, float, _Float16, unsigned short, _Float16, _Float16, _Float16, _Float16, _Float16, _Float16, FmhaBwdShape<32>, true, false, ck_tile::SimplifiedGenericAttentionMask<>, ck_tile::BlockDropoutBwd<false, true, false>, ck_tile::TileFmhaTraits<true, true, false, false, ck_tile::BlockAttentionBiasEnum::NO_BIAS, true, false, false, false, 1>>>' requested here
1001 | auto shuffled_k_lds_block_desc = MakeShuffledKLdsWriteBlockDescriptor
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/superuser/MyPrograms/wunjo/wunjo/portable/xformers/setup.py", line 584, in
TIA
lw
commented
5 days ago That seems to be an issue in upstream PyTorch: https://github.com/pytorch/pytorch/issues/72918
They claim that, for NVIDIA, it is due to a botched system setup, and installing the right packages should fix it. Could you verify your installation?
I don't know if this is also the case for AMD. Someone commented on that issue but no one answered. Could you try commenting there too or open a new issue on PyTorch?
brcisna
commented
5 days ago @lw
Thank You!! Very much! I will post the same error log there at PyTorch forums/issues and report back here. Am very green at this stuff..it goes without saying.
I think if i get xformers to compile correctly,,it may eliminate the float32 error i get when launching wunjo. BTW,, You should check out' wunjo AI V2' .. He doesnt get much ink,,,but is a very cool AI app.
Thanks again.
❓ Questions and Help
Hi All,
Debian 13 python3.10.12 venv PyTorch2.4.1_rocm
When I try and compile xformers against Pytorch2.4.1_rocm I am ending up with the common "no file found at /thrust/complex.h BUT,,,this may have something to do with issue https://github.com/facebookresearch/xformers/issues/1026
If I install the precompiled xformers_rocm This is what the xformers info looks like
python -m xformers.info WARNING[XFORMERS]: Need to compile C++ extensions to use all xFormers features. Please install xformers properly (see https://github.com/facebookresearch/xformers#installing-xformers) Memory-efficient attention, SwiGLU, sparse and more won't be available. Set XFORMERS_MORE_DETAILS=1 for more details xFormers 0.0.0 memory_efficient_attention.ckF: unavailable memory_efficient_attention.ckB: unavailable memory_efficient_attention.ck_decoderF: unavailable memory_efficient_attention.ck_splitKF: unavailable memory_efficient_attention.cutlassF: unavailable memory_efficient_attention.cutlassB: unavailable memory_efficient_attention.fa2F@0.0.0: unavailable memory_efficient_attention.fa2B@0.0.0: unavailable memory_efficient_attention.fa3F@0.0.0: unavailable memory_efficient_attention.fa3B@0.0.0: unavailable memory_efficient_attention.triton_splitKF: available indexing.scaled_index_addF: available indexing.scaled_index_addB: available indexing.index_select: available sequence_parallel_fused.write_values: unavailable sequence_parallel_fused.wait_values: unavailable sequence_parallel_fused.cuda_memset_32b_async: unavailable sp24.sparse24_sparsify_both_ways: unavailable sp24.sparse24_apply: unavailable sp24.sparse24_apply_dense_output: unavailable sp24._sparse24_gemm: unavailable sp24._cslt_sparse_mm@0.0.0: available swiglu.dual_gemm_silu: unavailable swiglu.gemm_fused_operand_sum: unavailable swiglu.fused.p.cpp: not built is_triton_available: True pytorch.version: 2.4.1+rocm6.1 pytorch.cuda: available gpu.compute_capability: 10.3 gpu.name: AMD Radeon Pro W6600 dcgm_profiler: unavailable build.info: none source.privacy: open source
AM trying to compile with an AMD Radeon Pro W6600 gfw1032,,,,which,,,as of May 2024 is still not supported.
Realize this is very experimental as it is.
TIA