Questions regarding local build

[mtsokol]

Hi @nullplay,

I rebuild Finch-mlir locally with the latest version of main branch in llvm-project. I've got two questions:

1. I followed README.md instructions and in cmake --build . --target check-finch four tests are failing (logs in details), is it expected to happen?

``` (llvm-dev) mtsokol@qgpu3:~/Finch-mlir/build$ cmake --build . --target check-finch [0/1] Running the finch regression tests UNRESOLVED: FINCH :: pdl.mlir (1 of 10) ******************** TEST 'FINCH :: pdl.mlir' FAILED ******************** Test has no 'RUN:' line ******************** FAIL: FINCH :: Finch/run-looplet-test.mlir (5 of 10) ******************** TEST 'FINCH :: Finch/run-looplet-test.mlir' FAILED ******************** Exit Code: 1 Command Output (stderr): -- RUN: at line 1: /home/mtsokol/Finch-mlir/build/bin/finch-opt /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir | /home/mtsokol/Finch-mlir/build/bin/finch-opt | /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir + /home/mtsokol/Finch-mlir/build/bin/finch-opt /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir + /home/mtsokol/Finch-mlir/build/bin/finch-opt + /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir:11:26: error: expected ',' %3 = finch.run %0 %0 %val : i32, i32, f32 to !finch.looplet ^ /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir:4:18: error: CHECK-LABEL: expected string not found in input // CHECK-LABEL: func @bar() ^ :1:1: note: scanning from here module { ^ :1:3: note: possible intended match here module { ^ Input file: Check file: /home/mtsokol/Finch-mlir/test/Finch/run-looplet-test.mlir -dump-input=help explains the following input dump. Input was: <<<<<< 1: module { label:4'0 X~~~~~~~~ error: no match found label:4'1 ? possible intended match 2: } label:4'0 ~~ 3: label:4'0 ~ >>>>>> -- ******************** FAIL: FINCH :: Finch/access.mlir (6 of 10) ******************** TEST 'FINCH :: Finch/access.mlir' FAILED ******************** Exit Code: 1 Command Output (stderr): -- RUN: at line 1: /home/mtsokol/Finch-mlir/build/bin/finch-opt /home/mtsokol/Finch-mlir/test/Finch/access.mlir | /home/mtsokol/Finch-mlir/build/bin/finch-opt | /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/access.mlir + /home/mtsokol/Finch-mlir/build/bin/finch-opt /home/mtsokol/Finch-mlir/test/Finch/access.mlir + /home/mtsokol/Finch-mlir/build/bin/finch-opt + /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/access.mlir /home/mtsokol/Finch-mlir/test/Finch/access.mlir:13:26: error: expected ',' %3 = finch.run %0 %0 %val : i32, i32, f32 to !finch.looplet ^ /home/mtsokol/Finch-mlir/test/Finch/access.mlir:4:18: error: CHECK-LABEL: expected string not found in input // CHECK-LABEL: func @bar() ^ :1:1: note: scanning from here module { ^ :1:3: note: possible intended match here module { ^ Input file: Check file: /home/mtsokol/Finch-mlir/test/Finch/access.mlir -dump-input=help explains the following input dump. Input was: <<<<<< 1: module { label:4'0 X~~~~~~~~ error: no match found label:4'1 ? possible intended match 2: } label:4'0 ~~ 3: label:4'0 ~ >>>>>> -- ******************** FAIL: FINCH :: Finch/finch-pass-plugin.mlir (7 of 10) ******************** TEST 'FINCH :: Finch/finch-pass-plugin.mlir' FAILED ******************** Exit Code: 1 Command Output (stderr): -- RUN: at line 1: /home/mtsokol/llvm-project/build-install/bin/mlir-opt /home/mtsokol/Finch-mlir/test/Finch/finch-pass-plugin.mlir --load-pass-plugin=/home/mtsokol/Finch-mlir/build/lib/FinchPlugin.so --pass-pipeline="builtin.module(finch-switch-bar-foo)" | /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/finch-pass-plugin.mlir + /home/mtsokol/llvm-project/build-install/bin/mlir-opt /home/mtsokol/Finch-mlir/test/Finch/finch-pass-plugin.mlir --load-pass-plugin=/home/mtsokol/Finch-mlir/build/lib/FinchPlugin.so '--pass-pipeline=builtin.module(finch-switch-bar-foo)' + /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/finch-pass-plugin.mlir /home/mtsokol/Finch-mlir/test/Finch/finch-pass-plugin.mlir:4:18: error: CHECK-LABEL: expected string not found in input // CHECK-LABEL: func @foo() ^ :1:1: note: scanning from here module { ^ :2:7: note: possible intended match here func.func @bar() { ^ Input file: Check file: /home/mtsokol/Finch-mlir/test/Finch/finch-pass-plugin.mlir -dump-input=help explains the following input dump. Input was: <<<<<< 1: module { label:4'0 X~~~~~~~~ error: no match found 2: func.func @bar() { label:4'0 ~~~~~~~~~~~~~~~~~~~~ label:4'1 ? possible intended match 3: return label:4'0 ~~~~~~~~ 4: } label:4'0 ~~~ 5: func.func @abar() { label:4'0 ~~~~~~~~~~~~~~~~~~~~~ 6: return label:4'0 ~~~~~~~~ 7: } label:4'0 ~~~ . . . >>>>>> -- ******************** FAIL: FINCH :: Finch/finch-plugin.mlir (8 of 10) ******************** TEST 'FINCH :: Finch/finch-plugin.mlir' FAILED ******************** Exit Code: 1 Command Output (stderr): -- RUN: at line 1: /home/mtsokol/llvm-project/build-install/bin/mlir-opt /home/mtsokol/Finch-mlir/test/Finch/finch-plugin.mlir --load-dialect-plugin=/home/mtsokol/Finch-mlir/build/lib/FinchPlugin.so --pass-pipeline="builtin.module(finch-switch-bar-foo)" | /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/finch-plugin.mlir + /home/mtsokol/llvm-project/build-install/bin/mlir-opt /home/mtsokol/Finch-mlir/test/Finch/finch-plugin.mlir --load-dialect-plugin=/home/mtsokol/Finch-mlir/build/lib/FinchPlugin.so '--pass-pipeline=builtin.module(finch-switch-bar-foo)' + /home/mtsokol/llvm-project/build-install/bin/FileCheck /home/mtsokol/Finch-mlir/test/Finch/finch-plugin.mlir /home/mtsokol/Finch-mlir/test/Finch/finch-plugin.mlir:4:18: error: CHECK-LABEL: expected string not found in input // CHECK-LABEL: func @foo() ^ :1:1: note: scanning from here module { ^ :2:7: note: possible intended match here func.func @bar() { ^ Input file: Check file: /home/mtsokol/Finch-mlir/test/Finch/finch-plugin.mlir -dump-input=help explains the following input dump. Input was: <<<<<< 1: module { label:4'0 X~~~~~~~~ error: no match found 2: func.func @bar() { label:4'0 ~~~~~~~~~~~~~~~~~~~~ label:4'1 ? possible intended match 3: return label:4'0 ~~~~~~~~ 4: } label:4'0 ~~~ 5: func.func @finch_types(%arg0: !finch.custom<"10">) { label:4'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 6: return label:4'0 ~~~~~~~~ 7: } label:4'0 ~~~ . . . >>>>>> -- ******************** ******************** Unresolved Tests (1): FINCH :: pdl.mlir ******************** Failed Tests (4): FINCH :: Finch/access.mlir FINCH :: Finch/finch-pass-plugin.mlir FINCH :: Finch/finch-plugin.mlir FINCH :: Finch/run-looplet-test.mlir Testing Time: 0.16s Total Discovered Tests: 10 Passed : 5 (50.00%) Unresolved: 1 (10.00%) Failed : 4 (40.00%) FAILED: test/CMakeFiles/check-finch /home/mtsokol/Finch-mlir/build/test/CMakeFiles/check-finch cd /home/mtsokol/Finch-mlir/build/test && /home/mtsokol/.conda/envs/llvm-dev/bin/python3.10 /home/mtsokol/llvm-project/build/bin/llvm-lit -sv /home/mtsokol/Finch-mlir/build/test ninja: build stopped: subcommand failed. ```

2. I started to look into Python bindings that are build in Finch-mlir, and I noticed that importing execution_engine fails. I didn't spend much time on this but I think there might be a missing dependency in DEPENDS or LINK_LIBS in one of the CMakeLists.txt files:

In [1]: from mlir_finch import execution_engine
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 from mlir_finch import execution_engine

File ~/Finch-mlir/build/python_packages/finch/mlir_finch/execution_engine.py:6, in <module>
      1 #  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
      2 #  See https://llvm.org/LICENSE.txt for license information.
      3 #  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
      4
      5 # Simply a wrapper around the extension module of the same name.
----> 6 from ._mlir_libs import _mlirExecutionEngine as _execution_engine
      7 import ctypes
      9 __all__ = [
     10     "ExecutionEngine",
     11 ]

ImportError: /home/mtsokol/Finch-mlir/build/python_packages/finch/mlir_finch/_mlir_libs/_mlirExecutionEngine.cpython-310-x86_64-linux-gnu.so: undefined symbol: mlirExecutionEngineRegisterSymbol

[nullplay]

1. Yes, this is expected; I'm not adding test cases in the check-finch now. I'll add tests at check-finch at some point. If you are annoyed by all the error messages, you can just build Finch-mlir with cmake --build . without testing.
2. I'm currently only playing with main cpp implementation and haven't checked the python part yet. What is the file and command that I can reproduce it?

[mtsokol]

Thank you for explanation!

2. I'm currently only playing with main cpp implementation and haven't checked the python part yet. What is the file and command that I can reproduce it?

In the repo there's test/python/smoketest.py file. I added from mlir_finch import execution_engine to the imports section to reproduce it:

from mlir_finch.ir import *
from mlir_finch.dialects import builtin as builtin_d, finch as finch_d
from mlir_finch import execution_engine

Having built llvm-project and Finch-mlir as component build (as described in README.md) I export PYTHONPATH:

export PYTHONPATH=/home/mtsokol/Finch-mlir/build/python_packages/finch

and then run:

python test/python/smoketest.py

---

@nullplay I've got one more question. We started to work on user-facing Python bindings layer (the API that the user will have access to for creating tensors etc.). Let's agree on a specific LLVM version (or git hash) that our code and Finch-mlir repo is going use, so once we plug the new dialect it goes smoothly. For example latest stable LLVM release 18.1.8 and latest main branch has some API differences in Python Bindings.

Latest pre-release is 19.1.0-rc2 (https://github.com/llvm/llvm-project/releases/tag/llvmorg-19.1.0-rc2). Maybe we can agree on using 19.x branch? and 19.1.0 once is released? Or 18.1.8 tag?

[mtsokol]

Maybe we can agree on using 19.x branch? and 19.1.0 once is released? Or 18.1.8 tag?

@nullplay In the meeting we agreed that 19.1.x version should be targeted for our development. It will be released in two weeks: https://discourse.llvm.org/t/llvm-19-1-0-rc3-released. Right now RC3 is available.

[nullplay]

I added appropriate cmake extension for execution_engine in python/CMakeLists.txt at latest push. I can now run from mlir_finch import execution_engine without an error. Could you check if it works?

[mtsokol]

Now it all works - thank you!