repo fails to build using Triton Image 23.01

[Chris113113]

Description

main branch as of 02/13/2023
Build crashes at 57% with no additional information.

I was able to successfully build using 22.09 today to validate that nothing on my machine is causing the issue.

Tail of logs from the build:
...
kernels
[ 55%] Linking CUDA device code CMakeFiles/beam_search_topk_kernels.dir/cmake_device_link.o
[ 56%] Linking CUDA static library ../../../../../lib/libbeam_search_topk_kernels.a
[ 56%] Built target beam_search_topk_kernels
[ 56%] Linking CUDA device code CMakeFiles/fpA_intB_gemm.dir/cmake_device_link.o
[ 56%] Linking CXX static library ../../../../../../lib/libfpA_intB_gemm.a
[ 56%] Built target fpA_intB_gemm
[ 56%] Linking CUDA device code CMakeFiles/moe_gemm_kernels.dir/cmake_device_link.o
[ 56%] Linking CXX static library ../../../../../../lib/libmoe_gemm_kernels.a
[ 56%] Built target moe_gemm_kernels
[ 56%] Linking CUDA device code CMakeFiles/decoder_masked_multihead_attention.dir/cmake_device_link.o
[ 56%] Linking CUDA static library ../../../../../lib/libdecoder_masked_multihead_attention.a
[ 56%] Built target decoder_masked_multihead_attention
[ 57%] Linking CUDA device code CMakeFiles/online_softmax_beamsearch_kernels.dir/cmake_device_link.o
[ 57%] Linking CUDA static library ../../../../../lib/libonline_softmax_beamsearch_kernels.a
[ 57%] Built target online_softmax_beamsearch_kernels
make: *** [Makefile:136: all] Error 2
The command '/bin/sh -c cd _deps/repo-ft-src/ &&     git log | head -n 3 2>&1 | tee /workspace/build/fastertransformer_backend/FT_version.txt &&     cd /workspace/build/fastertransformer_backend/build &&     make -j"$(grep -c ^processor /proc/cpuinfo)" install &&     rm /workspace/build/fastertransformer_backend/build/bin/*_example -rf &&     rm /workspace/build/fastertransformer_backend/build/lib/lib*Backend.so -rf' returned a non-zero code: 2

Reproduced Steps

~/fastertransformer_backend$ echo $CONTAINER_VERSION
23.01
~/fastertransformer_backend$ echo $TRITON_DOCKER_IMAGE
triton_with_ft:23.01
~/fastertransformer_backend$ docker build --rm \
       --build-arg TRITON_VERSION=${CONTAINER_VERSION} \ 
       -t ${TRITON_DOCKER_IMAGE} \
       -f docker/Dockerfile \
       .

[byshiue]

FT does not support CUDA 12 now. You can try the docker image 22.12.

[SeungjaeLim]

I had the same problem as above. (I was running on version 23.04.) So I changed the two versions in the export as shown below and then ran it, but still got the same error.

~/fastertransformer_backend$ echo $CONTAINER_VERSION
22.12
~/fastertransformer_backend$ echo $TRITON_DOCKER_IMAGE
triton_with_ft:22.12

I thought that the docker cache might be the problem, so I deleted it and ran it again and got the same problem - can you help?