How to emitc compile for new Python frontend

[stellaraccident]

So, on the down-low, I've been building a native Python->IREE compiler frontend. And since I got the lowerings minimally working today (it doesn't do much, since most interesting things require runtime library integration which is coming next), I thought I'd see how it worked to transpile to a C program. It isn't super far off it seems. A few questions inline.

Program under test:

# python ~/scratch/pydme2e.py | ./llvm/bin/iree-dialects-opt -convert-iree-pydm-to-iree -canonicalize | ./iree/iree/tools/iree-translate --iree-mlir-to-vm-bytecode-module -iree-hal-target-backends=cpu

from typing import List
from mlir.dialects.iree_pydm.importer.test_util import *

@test_import_global
def weak_integer_arg_and_return(a: int) -> int:
  return a

But since you need to build non-default projects to get that, it may be easier to work from the imported/lowered form:

module  {
  func @weak_integer_arg_and_return(%arg0: i32) -> (i32, i32) {
    %c-4_i32 = constant -4 : i32
    %c0_i32 = constant 0 : i32
    %c9_i32 = constant 9 : i32
    %c0 = constant 0 : index
    %c1 = constant 1 : index
    %c256_i32 = constant 256 : i32
    %c2 = constant 2 : index
    %0 = iree.list.create : !iree.list<!iree.variant>
    %1 = iree.list.create : !iree.list<!iree.variant>
    iree.list.resize %1, %c2 : !iree.list<!iree.variant>
    iree.list.set %1[%c0], %c9_i32 : !iree.list<!iree.variant>, i32
    iree.list.set %1[%c1], %arg0 : !iree.list<!iree.variant>, i32
    iree.list.resize %0, %c2 : !iree.list<!iree.variant>
    iree.list.set %0[%c0], %c256_i32 : !iree.list<!iree.variant>, i32
    iree.list.set %0[%c1], %1 : !iree.list<!iree.variant>, !iree.list<!iree.variant>
    %2 = iree.list.get %0[%c1] : !iree.list<!iree.variant> -> !iree.list<!iree.variant>
    %3 = iree.list.get %2[%c0] : !iree.list<!iree.variant> -> i32
    %4 = cmpi eq, %c9_i32, %3 : i32
    cond_br %4, ^bb1, ^bb2(%c-4_i32, %c0_i32 : i32, i32)
  ^bb1:  // pred: ^bb0
    %5 = iree.list.get %2[%c1] : !iree.list<!iree.variant> -> i32
    br ^bb2(%c0_i32, %5 : i32, i32)
  ^bb2(%6: i32, %7: i32):  // 2 preds: ^bb0, ^bb1
    %8 = cmpi eq, %c0_i32, %6 : i32
    %9 = select %8, %7, %c0_i32 : i32
    return %6, %9 : i32, i32
  }
}

Then ./iree/iree/tools/iree-translate --iree-mlir-to-vm-c-module -iree-hal-target-backends=cpu > ~/scratch/pydme2e.cpp.

Running clang++ reveals:

clang++ ~/scratch/pydme2e.cpp -I .

/home/stella/scratch/pydme2e.cpp:229:9: warning: implicit conversion from 'long' to 'int32_t' (aka 'int') changes value from 4294967292 to -4 [-Wconstant-conversion]
  v18 = 4294967292;
      ~ ^~~~~~~~~~
/home/stella/scratch/pydme2e.cpp:726:34: error: use of undeclared identifier 'call_0i_ii_shim'
{(iree_vm_native_function_shim_t)call_0i_ii_shim, (iree_vm_native_function_target_t)module_weak_integer_arg_and_return},

Notes:

• For the first warning, the correct value is indeed -4, so the bit pattern is correct. I suspect some more robust literal emit/casting would clean this up.
• For the second, I don't know? Where do I get such a shim?

Finally, is anyone doing the build engineering to package up the headers and a mondo static-lib to link against? That seems like what I need once getting past the above error.

Also, given the name, I first tried to compile this as a pure C program but ran into the following:

/home/stella/scratch/pydme2e.c:229:9: warning: implicit conversion from 'long' to 'int32_t' (aka 'int') changes value from 4294967292 to -4 [-Wconstant-conversion]
  v18 = 4294967292;
      ~ ^~~~~~~~~~
/home/stella/scratch/pydme2e.c:716:2: error: initializer element is not a compile-time constant
{iree_make_cstring_view("weak_integer_arg_and_return"), iree_make_cstring_view("0i_ii"), 0, NULL},
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/stella/scratch/pydme2e.c:726:34: error: use of undeclared identifier 'call_0i_ii_shim'
{(iree_vm_native_function_shim_t)call_0i_ii_shim, (iree_vm_native_function_target_t)module_weak_integer_arg_and_return},
                                 ^
/home/stella/scratch/pydme2e.c:730:1: error: initializer element is not a compile-time constant
iree_make_cstring_view("module"),

Seems like the C version bails on other errors but allows the implicit definition of the shim.

Anyway, was really neat to get this far.

[marbre]

[..] Notes:

* For the first warning, the correct value is indeed `-4`, so the bit pattern is correct. I suspect some more robust literal emit/casting would clean this up.

We'll take a look. Seems like a bug in the emitter.

* For the second, I don't know? Where do I get such a shim?

Right now, those are hard-coded and we do not yet auto-generate those. What he have so far is in iree/vm/shims_emitc.h. So this is still a TODO to auto-generated those (after we got some other things working).

Finally, is anyone doing the build engineering to package up the headers and a mondo static-lib to link against? That seems like what I need once getting past the above error.

Simon and me were working on an example replicating the iree/samples/static_library one. However, a mono static-lib was not yet on our agenda. Feel free to create an issue and assign it to me :)

Also, given the name, I first tried to compile this as a pure C program but ran into the following:

/home/stella/scratch/pydme2e.c:229:9: warning: implicit conversion from 'long' to 'int32_t' (aka 'int') changes value from 4294967292 to -4 [-Wconstant-conversion]
  v18 = 4294967292;
      ~ ^~~~~~~~~~
/home/stella/scratch/pydme2e.c:716:2: error: initializer element is not a compile-time constant
{iree_make_cstring_view("weak_integer_arg_and_return"), iree_make_cstring_view("0i_ii"), 0, NULL},
 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/stella/scratch/pydme2e.c:726:34: error: use of undeclared identifier 'call_0i_ii_shim'
{(iree_vm_native_function_shim_t)call_0i_ii_shim, (iree_vm_native_function_target_t)module_weak_integer_arg_and_return},
                                 ^
/home/stella/scratch/pydme2e.c:730:1: error: initializer element is not a compile-time constant
iree_make_cstring_view("module"),

Seems like the C version bails on other errors but allows the implicit definition of the shim.

We already noticed that error and Simon has a fix for it. Should be ready to review soon.

Anyway, was really neat to get this far.

Thanks a lot! We're really looking forward to get this running.

[marbre]

To give a short update, we have fixed some of the issues you were facing. The two points not addresses yet are the implementation of a shim (in addition to the ones we have in iree/vm/shims_emitc.h) and a mono static library (which should be convenient but not blocking).