tenstorrent / tt-buda

Tenstorrent TT-BUDA Repository
Other
207 stars 28 forks source link

How to install and run Netlist Analyzer (dpnra) #57

Open jhlee508 opened 2 weeks ago

jhlee508 commented 2 weeks ago

I am trying to run the Route UI tool and I figured out the file /netlist_analyzer/ is needed.

I found the TT_BACKEND_NETLIST_ANALYZER environment variable and set it to 1. However, I got an error that there is no dpnra in the path .../third_party/budabackend/build/bin/ which I think it should be there when correctly installing tt-buda. I have installed tt-buda using the official bash file here. (The release version has the dpnra in .../budabackend/build/bin/ and it works fine.)

Could you help me find the solution?

Error Log

2024-09-13 12:58:43.291 | INFO     | PerfInfra       - Running Performance Analyzer
2024-09-13 12:58:43.291 | INFO     | PerfInfra       - Performance Analyzer command: /home/n4/jaehwan/research/tenstorrent/tt-buda/third_party/budabackend//perf_lib/perf_analysis.py --analysis-mode overlay_decouple --skip-test-run --input-test-perf-results /tmp/jaehwan/4e2d342679e1/perf_results/
2024-09-13 12:58:43.291 | INFO     | PerfInfra       - Reading the performance results from /tmp/jaehwan/4e2d342679e1/perf_results/
2024-09-13 12:58:43.291 | INFO     | PerfInfra       - Writing the analyzer results to /tmp/jaehwan/4e2d342679e1/perf_results/
2024-09-13 12:58:43.422 | INFO     | PerfInfra       - Performance analyzer finished successfully
2024-09-13 12:58:43.422 | INFO     | Analyzer        - Running Netlist Analyzer
2024-09-13 12:58:43.422 | INFO     | Analyzer        - Netlist analyzer command: ARCH_NAME=wormhole_b0 /home/n4/jaehwan/research/tenstorrent/tt-buda/third_party/budabackend//build/bin/dpnra --arch wormhole_b0 --netlist /tmp/jaehwan/4e2d342679e1/linear_netlist.yaml --out-dir /tmp/jaehwan/4e2d342679e1 --cluster-desc /tmp/jaehwan/4e2d342679e1/cluster_desc.yaml
2024-09-13 12:58:43.423 | ERROR    | Always          - Error log: sh: 1: /home/n4/jaehwan/research/tenstorrent/tt-buda/third_party/budabackend//build/bin/dpnra: not found

2024-09-13 12:58:43.424 | FATAL    | Always          - Netlist analyzer failed. Full log under /tmp/jaehwan/4e2d342679e1/netlist_analyzer.err
2024-09-13 12:58:43.426 | ERROR    | Always          - Netlist analyzer failed. Full log under /tmp/jaehwan/4e2d342679e1/netlist_analyzer.err
backtrace:
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/third_party/budabackend/build/lib/libtt.so(+0x2396de) [0x7efc095986de]
 --- tt_runtime::finish()
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/pybuda/pybuda/_C.so(+0x8b0b9f) [0x7efc0a708b9f]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/pybuda/pybuda/_C.so(+0x2aef86) [0x7efc0a106f86]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(PyCFunction_Call+0x59) [0x5f58f9]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyObject_MakeTpCall+0x29e) [0x5f64ce]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3() [0x50b4b3]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x5777) [0x570b67]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x907) [0x56bcf7]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x907) [0x56bcf7]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x907) [0x56bcf7]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x907) [0x56bcf7]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3() [0x50ad5c]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(PyObject_Call+0x62) [0x5f5072]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x1f2d) [0x56d31d]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x907) [0x56bcf7]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x569dfa]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5e83]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x907) [0x56bcf7]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x1b6) [0x5f5ca6]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x725) [0x56bb15]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x569dfa]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyFunction_Vectorcall+0x393) [0x5f5e83]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalFrameDefault+0x1905) [0x56ccf5]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_PyEval_EvalCodeWithName+0x26a) [0x569dfa]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(PyEval_EvalCode+0x27) [0x68ce77]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3() [0x67e631]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3() [0x67e6af]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(PyRun_StringFlags+0x7f) [0x67e7ff]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(PyRun_SimpleStringFlags+0x3f) [0x67f20f]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(Py_RunMain+0x2cc) [0x6b687c]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(Py_BytesMain+0x2d) [0x6b6b4d]
 --- /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7efda0816083]
 --- /home/n4/jaehwan/research/tenstorrent/tt-buda/build/python_env/bin/python3(_start+0x2e) [0x5fa81e]
milank94 commented 2 weeks ago

@jhlee508 can you try a couple more things?

jhlee508 commented 2 weeks ago

I already have set TT_BACKEND_PERF_ANALYZER=1 and LD_LIBRARY_PATH to the path .../tt-buda/third_party/budabackend/build/lib/.

I think the problem of this error below is likely caused by an incorrect build (dpnra executable not found). May be something to modify the build script env_for_wormhole_b0.sh?

2024-09-15 15:48:29.176 | ERROR    | Always          - Error log: sh: 1: /home/n4/jaehwan/research/tenstorrent/tt-buda/third_party/budabackend//build/bin/dpnra: not found
milank94 commented 2 weeks ago

When you clone the tt-buda repo, did you run git submodule update --init --recursive? I'm wondering it the submodules did not all update.

For example,

git clone https://github.com/tenstorrent/tt-buda.git
cd tt-buda
git submodule update --init --recursive
./env_for_wormhole_b0.sh

Alternatively, you can use a prebuilt release: https://github.com/tenstorrent/tt-buda/releases. I've check the latest and that one does include the dpnra directory in build.

milank94 commented 5 days ago

@jhlee508 any follow up on the above?

jhlee508 commented 3 days ago

I am still in the process of testing which version works well and which one doesn't. Thank you.

jhlee508 commented 3 days ago

Buda version tags/0.18.2 builds successfully. However, still does not have ./tt-buda/third_party/budabackend/build/bin/dpnra so it is unable to run Netlist Analyzer (dpnra).

Build Command (branch: tags/v0.18.2)

git clone https://github.com/tenstorrent/tt-buda.git
cd tt-buda
git submodule update --init --recursive
git checkout tags/v0.18.2
./env_for_wormhole_b0.sh

Additionally I have tested different version and found out that building tt-buda version tags/v0.19.3 occurs a build error which is a different problem. This error also occurs in the main (last commit: 1e949f5) branch.

Built Error log (branch: tags/v0.19.3)

Downloading llvm
tar: /home/n4/jaehwan/buda-build1/tt-buda/third_party/clang+llvm-13.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
Release
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Hide private symbols...
-- Forbidding undefined symbols in shared library, using -Wl,--no-undefined on platform Linux
-- Build with RPC support...
-- Build with Graph Executor support...
-- Build with profiler...
-- Build with AOT Executor support...
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- Build Alloc alignment set to 64
-- Didn't find the path to CCACHE, disabling ccache
-- Performing Test SUPPORT_CXX17
-- Performing Test SUPPORT_CXX17 - Success
-- VTA build with VTA_HW_PATH=/home/n4/jaehwan/buda-build1/tt-buda/third_party/tvm/3rdparty/vta-hw
-- Build VTA runtime with target: sim
-- Use llvm-config=/home/n4/jaehwan/buda-build1/tt-buda/third_party/llvm/bin/llvm-config --link-static
CMake Error at cmake/utils/FindLLVM.cmake:76 (message):
  Fatal error executing:
  /home/n4/jaehwan/buda-build1/tt-buda/third_party/llvm/bin/llvm-config;--link-static
  --libfiles
Call Stack (most recent call first):
  cmake/modules/LLVM.cmake:31 (find_llvm)
  CMakeLists.txt:528 (include)

-- Configuring incomplete, errors occurred!
make: *** [Makefile:128: build/submodules/third_party/tvm.build] Error 1
Pybuda build failed!
./env_for_wormhole_b0.sh: line 22: return: can only `return' from a function or sourced script

Maybe open a different issue?

milank94 commented 4 hours ago

Thanks @jhlee508 one thing I can recommend is ensuring that you have all of the correct system dependencies installed when trying to build.

buda_build.md