compiler: eval_kernel() to create kernel function from string

[dnadlinger]

With the lack of generics support in the compiler (while requiring an unique set of inferred types per class), the option to brute-force one's way through metaprogramming situations by generating the code for @kernel functions in string form would be useful.

There currently doesn't seem a way to do that (apart from possibly generating files on the fly) because inspect.getsource only works for functions that originate from a source file.

It seems quite feasible to add something along the lines of an eval_kernel() function that stores the passed source code in the artiq_embedded metadata for later use by the compiler. The Python side host function body could just be set to something like raise NotImplementedError("eval_kernel functions are not available on host") in the first instance, since it should never be called anyway. @portable support could be added later by actually using exec on the host.

[whitequark]

Implementing genericism through eval seems quite depressing. We already implement monomorphization (it's required for handling Python inheritance; superclass methods are polymorphic on the self argument), so why not simply expose it? To aid inference, polymorphic methods will require an explicit type annotation, something like:

def list_add_poly(a: TList(TVar("a")), b: TList(TVar("a"))) -> TList(TVar("a")):
    return a + b

[dnadlinger]

Oh, string eval is only ever going to be a fallback to fill in the gaps between language features. Still, this can be very useful.

Even in D, which has a good claim to be the language with the most expressive metaprogramming capabilities among the mainstream compiled languages (C++/Rust/…, spotting expansive compile-time reflection, static if/foreach, etc.), the ability to generate code at compile time is handy to fill the gaps. Same goes for Rust macros.

---

For another example where this would be useful for me right now, consider the following situation:

class Fragment(HasEnvironment):
  def build(self):
    self.children = []

  def setattr_fragment(self, name, fragment):
    setattr(self, name, fragment)
    self.children.append(fragment)

  @kernel
  def setup(self):
    # By default just recurse.
    for c in self.children:
      c.setup()

class FooFragment(Fragment):
  def build(self):
    super.build()
    self.setattr_fragment("bar", BarFragment())

  @kernel
  def do_things(self):
    self.bar.do_the_thing(5)

class BarFragment(Fragment):
  @kernel
  def setup(self):
    super.setup()
    self._do_some_core_device_stuff()

  @kernel
  def do_the_thing(self, n):
    for i in range(n):
      ...    

This is obviously greatly simplified pseudocode, but hopefully gets the point across. The wider context for this is that scanning framework I mentioned some months ago, where users can compose their experiments out of different blocks with their own parameters, result data channels, etc., and the system handles parameter aggregation, (re)initialisation, plotting, etc. behind the scenes.

There might be a clean way of making this work, but currently I can't quite see how to. children can't be represented on the core device because of lack of polymorphism. I also tried building an array of function pointers to the child setup calls on the host, but that relies on having delegates (closures) with the context type erased, which I also couldn't get to work.

Eval would allow for a simple implementation of this, by just saving the child names in setattr_fragment, and generating code for a setup_recurse function per class that directly calls all the child setup methods.

Again, there might be an extension to the type system that makes this particular case work. Still, ARTIQ Python is (by design) a fairly restricted language, and it is very helpful to be able to resort to code generation (rather than extending the compiler) in order to provide a clean and simple design to the user.

[dnadlinger]

By the way, regarding your earlier generics example, my code looks somewhat more like this:

from typing import *

T = TypeVar("T")
class Param(Generic[T]):
  def __init__(self, value: T):
    self.value = value

  @portable
  def get(self) -> T:
    return self.value

I got halfway through implementing support for this (using the standard Python type hints, and __orig_class__ and __args__ on the host runtime value to resolve the type variables in the compiler), but I didn't find a clean way to handle duplicating the classes in the compiler yet without major reworks.