search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
366 stars 36 forks source link

Update install documentation to avoid commonly encountered errors for release tags #10275

Closed dvartaniansTT closed 23 hours ago

dvartaniansTT commented 1 month ago

Describe the bug Instruct user to do the following when checking out to a tag/branch:

git checkout [v0.50.0](https://github.com/tenstorrent/tt-metal/tree/v0.50.0) git submodule update --init --recursive build (either from src or whl)

It may not be obvious to an average user to do git submodule update --init --recursive prior to build and this causes model demos to fail with errors such as:

`At first: models/demos/wormhole/falcon7b/demo_wormhole.py::test_demo[user_input0-default_mode_stochastic] Device | INFO | Opening user mode device driver Detecting chips (found 1)
2024-07-15 05:10:00.321 | INFO | SiliconDriver - Detected 1 PCI device : [0] 2024-07-15 05:10:00.383 | INFO | SiliconDriver - Software version 6.0.0, Ethernet FW version 6.9.0 (Device 0) Metal | INFO | Initializing device 0. Program cache is NOT enabled Metal | INFO | AI CLK for device 0 is: 1000 MHz BuildKernels | INFO | trisc1 link failure -- cmd: cd /home/dvartanians/Metal/tt-metal/built/6144/firmware/trisc1/ && /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,max-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -O3 -T/home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/trisc1.ld substitutes.o trisc.o tmu-crt0.o -o /home/dvartanians/Metal/tt-metal/built/6144/firmware/trisc1/trisc1.elf BuildKernels | INFO | trisc0 link failure -- cmd: cd /home/dvartanians/Metal/tt-metal/built/6144/firmware/trisc0/ && /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,max-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -O3 -T/home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/trisc0.ld substitutes.o trisc.o tmu-crt0.o -o /home/dvartanians/Metal/tt-metal/built/6144/firmware/trisc0/trisc0.elf /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot open linker script file /home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/trisc1.ld: No such file or directory collect2: error: ld returned 1 exit status Always | FATAL | trisc1 build failed /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot open linker script file /home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/trisc0.ld: No such file or directory collect2: error: ld returned 1 exit status Always | FATAL | trisc0 build failed BuildKernels | INFO | trisc2 link failure -- cmd: cd /home/dvartanians/Metal/tt-metal/built/6144/firmware/trisc2/ && /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,max-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -O3 -T/home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/trisc2.ld substitutes.o trisc.o tmu-crt0.o -o /home/dvartanians/Metal/tt-metal/built/6144/firmware/trisc2/trisc2.elf /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot open linker script file /home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/trisc2.ld: No such file or directory collect2: error: ld returned 1 exit status Always | FATAL | trisc2 build failed BuildKernels | INFO | ncrisc link failure -- cmd: cd /home/dvartanians/Metal/tt-metal/built/6144/firmware/ncrisc/ && /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,max-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -Os -T/home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/ncrisc.ld substitutes.o tmu-crt0.o ncrisc.o ncrisc-halt.o -o /home/dvartanians/Metal/tt-metal/built/6144/firmware/ncrisc/ncrisc.elf /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot open linker script file /home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/ncrisc.ld: No such file or directory collect2: error: ld returned 1 exit status Always | FATAL | ncrisc build failed BuildKernels | INFO | idle_erisc link failure -- cmd: cd /home/dvartanians/Metal/tt-metal/built/6144/firmware/idle_erisc/ && /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,max-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -Os -T/home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/idle-erisc.ld substitutes.o noc.o tmu-crt0.o idle_erisc.o -o /home/dvartanians/Metal/tt-metal/built/6144/firmware/idle_erisc/idle_erisc.elf /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot open linker script file /home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/idle-erisc.ld: No such file or directory collect2: error: ld returned 1 exit status Always | FATAL | idle_erisc build failed BuildKernels | INFO | brisc link failure -- cmd: cd /home/dvartanians/Metal/tt-metal/built/6144/firmware/brisc/ && /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/riscv32-unknown-elf-g++ -mwormhole -march=rv32imw -mtune=rvtt-b1 -mabi=ilp32 -std=c++17 -flto -ffast-math -fno-exceptions -Wl,-z,max-page-size=16 -Wl,-z,common-page-size=16 -nostartfiles -Os -T/home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/brisc.ld substitutes.o tdma_xmov.o noc.o tmu-crt0.o brisc.o -o /home/dvartanians/Metal/tt-metal/built/6144/firmware/brisc/brisc.elf /home/dvartanians/Metal/tt-metal/tt_metal/third_party/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/10.2.0/../../../../riscv32-unknown-elf/bin/ld: cannot open linker script file /home/dvartanians/Metal/tt-metal/runtime/hw/toolchain/brisc.ld: No such file or directory collect2: error: ld returned 1 exit status Always | FATAL | brisc build failed libc++abi: terminating due to uncaught exception of type std::runtime_error: TT_THROW @ /home/runner/work/tt-metal/tt-metal/tt_metal/jit_build/build.cpp:396: tt::exception info: brisc build failed backtrace: --- /home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/tt_lib/build/lib/libtt_metal.so(+0xd094e) [0x7f05baa8794e] --- tt::tt_metal::JitBuildState::link(std::1::basic_string<char, std::__1::char_traits, std::1::allocator> const&, std::1::basic_string<char, std::__1::char_traits, std::1::allocator> const&) const --- tt::tt_metal::JitBuildState::build(tt::tt_metal::JitBuildSettings const*) const --- /home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/tt_lib/build/lib/libtt_metal.so(+0xd6af3) [0x7f05baa8daf3] --- /home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/tt_lib/build/lib/libtt_metal.so(+0xdadba) [0x7f05baa91dba] --- /home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/tt_lib/build/lib/libtt_metal.so(+0xd995d) [0x7f05baa9095d] --- /home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/tt_lib/build/lib/libtt_metal.so(+0xd8e69) [0x7f05baa8fe69] --- /home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/tt_lib/build/lib/libtt_metal.so(+0xd8cca) [0x7f05baa8fcca] --- /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f062a3fa609] --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f062a534353]

Fatal Python error: Aborted

Current thread 0x00007f062a249740 (most recent call first): File "/home/dvartanians/Metal/tt-metal/conftest.py", line 92 in device File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 901 in call_fixture_func File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 1129 in pytest_fixture_setup File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 1075 in execute File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 677 in _compute_fixture_value File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 591 in _get_active_fixturedef File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 569 in getfixturevalue File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/fixtures.py", line 550 in _fillfixtures File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/python.py", line 1792 in setup File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 492 in setup File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 155 in pytest_runtest_setup File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 260 in File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 339 in from_call File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 259 in call_runtest_hook File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 220 in call_and_report File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 125 in runtestprotocol File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in call File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/config/init.py", line 167 in main File "/home/dvartanians/Metal/tt-metal/venv/lib/python3.8/site-packages/_pytest/config/init.py", line 190 in console_main File "/home/dvartanians/Metal/tt-metal/venv/bin/pytest", line 8 in Aborted (core dumped) `

Expected behavior Make installation README comprehensive to avoid build errors.

Please complete the following environment information:

tt-rkim commented 1 month ago
Screenshot 2024-07-15 at 12 52 06 PM

We have had recurse submodules as a required option for a long time.

It seems like you pull the whole repo without doing this, and then checkout a tag. This is not the approach that's suggested in the installing doc.

We took out the part which shows you how can you can check out a specific tag as the very first step in Git because of earlier feedback that it was too much information.

tt-rkim commented 23 hours ago

closing as stale and will review PR feedback instead