tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
400 stars 50 forks source link

Tracy profiling fails on TG #12927

Open kpaigwar opened 3 hours ago

kpaigwar commented 3 hours ago

Description

Seeing below error when tracy profiling is enabled on TG. This particular error appears on profiling any TG test. Clean build doesn't resolve the issue. The following error also doesn't appear on N150.

Error

           BuildKernels | INFO     | ncrisc link failure -- cmd: cd /proj_sw/user_dev/kpaigwar/tt-metal/built/4096/kernels/vc_packet_router/10795137441912823728/ncrisc/ && /proj_sw/user_dev
/kpaigwar/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,ma
x-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -Os -T/proj_sw/user_dev/kpaigwar/tt-metal/runtime/hw/toolchain/ncrisc.ld ncrisck.o /proj_sw/user_dev/kpaigwar/tt-metal/runtime/hw/lib
/tmu-crt0k.o /proj_sw/user_dev/kpaigwar/tt-metal/runtime/hw/lib/substitutes.o  -Xlinker "--just-symbols=/proj_sw/user_dev/kpaigwar/tt-metal/built/4096/firmware/ncrisc/ncrisc_weakened.elf" -
o /proj_sw/user_dev/kpaigwar/tt-metal/built/4096/kernels/vc_packet_router/10795137441912823728/ncrisc/ncrisc.elf                                                                             
/proj_sw/user_dev/kpaigwar/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: address 0xffb00c18 of /proj_sw/user_
dev/kpaigwar/tt-metal/built/4096/kernels/vc_packet_router/10795137441912823728/ncrisc/ncrisc.elf section `.ldm_data' is not within region `NCRISC_LOCAL_DATA_MEM'                            
/proj_sw/user_dev/kpaigwar/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: address 0xffb00c18 of /proj_sw/user_
dev/kpaigwar/tt-metal/built/4096/kernels/vc_packet_router/10795137441912823728/ncrisc/ncrisc.elf section `.ldm_data' is not within region `NCRISC_LOCAL_DATA_MEM'                            
/proj_sw/user_dev/kpaigwar/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: section .stack VMA [00000000ffb00bf0
,00000000ffb00fff] overlaps section .ldm_data VMA [00000000ffb00740,00000000ffb00c17]                                                                                                        
collect2: error: ld returned 1 exit status                                                                                                                                                   
                 Always | FATAL    | ncrisc build failed                                                                                                                                     
                 Always | FATAL    | Failed to generate binaries for vc_packet_router TT_THROW @ ../tt_metal/jit_build/build.cpp:408: tt::exception      
kpaigwar commented 3 hours ago

fyi @cglagovichTT