Assess support for MicroPython

[krzema12]

See https://kotlinlang.slack.com/archives/C0289CS37AS/p1637517359061900?thread_ts=1637517359.061900&cid=C0289CS37AS

https://core-electronics.com.au/tutorials/Python_and_MicroPython.html

Let's check how costly it would be to support both regular Python and Micropython (MP).

Ideas/steps:

• [X] check if there's a regular Linux interpreter for MP
• [X] smoke test: try running a sample output from current Kotlin Python compiler and see how it goes
• [x] run box tests with MP interpreter and see what's the success rate comparing to what we have on Python
• [ ] hook MP interpreter into the box tests so that CI runs both
• [ ] think of cases where it would be challenging to support both at the same time

[krzema12]

Running on Linux: see

• https://snapcraft.io/micropython
• https://github.com/micropython/micropython/tree/master/ports/unix

[krzema12]

Running on my STM32F429 DISCOVERY board: try the image provided after this change: https://github.com/micropython/micropython/issues/8018, once available. To learn if it's available, check the files on this page: https://micropython.org/download/STM32F429DISC/.

Nightly build already exposes a HEX: https://micropython.org/resources/firmware/STM32F429DISC-20211123-unstable-v1.17-195-gbb7aae557.hex .

[krzema12]

Running example code from here that works with regular Python:

def factorial(n):
    tmp0_subject = n <= 1
    if tmp0_subject == True:
        tmp = 1
    elif tmp0_subject == False:
        tmp = n * factorial((n - 1).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000))
    else:
        noWhenBranchMatchedException()

    return tmp

def numberOfCombinations(n, k):
    return (factorial(n) // (factorial(k) * factorial((n - k).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000))).__add__
(0x8000_0000_0000_0000).__and__(0xffff_ffff_ffff_ffff).__sub__(0x8000_0000_0000_0000)).__add__(0x8000_0000_0000_0000).__and__(0xffff_f
fff_ffff_ffff).__sub__(0x8000_0000_0000_0000)

def sumOverflowDemo(a, b):
    return (a + b).__add__(0x8000_0000).__and__(0xffff_ffff).__sub__(0x8000_0000)

print(isPowerOfTwo(32))
print(isPowerOfTwo(33))

print(factorial(5))

print(numberOfCombinations(4, 3))

print(sumOverflowDemo(2 ** 31 - 1, 2 ** 31 - 10))

gives

$ micropython calculations.py 
Traceback (most recent call last):
  File "calculations.py", line 21, in <module>
  File "calculations.py", line 2, in isPowerOfTwo
AttributeError: 'int' object has no attribute '__add__'

[krzema12]

@SerVB I remember that using these __add__ is a workaround to some issue. Do you remember any details? How hard would it be to do something more casual here that MicroPython would support? Fortunately, it's easy to run MP, at least on Linux. It's available in SnapStore.

[SerVB]

It's for overflow simulation, here is an explanation:

https://github.com/krzema12/kotlin-python/blob/df5531bf989c258bcfeb4b100925f3cb350bce40/compiler/ir/backend.py/src/org/jetbrains/kotlin/ir/backend/py/transformers/irToPy/PyIntrinsicTransformers.kt#L132-L136

We can replace magic methods (like __add__) with ordinary operators (like +) easily and it should work in MP. The initial idea of using these methods was to show that these calls are some internal pieces.

We can, for example, just replace them. Or support a new compilation flag like enum class CompilationMode { PYTHON3_9, MICRO_PYTHON }.

[krzema12]

Thanks! I propose (especially in such seemingly simple cases) is to emit such Python code that works for both Pythons. The compilation flag introduces some extra complexity, maybe we'll need it in the future, though.

My first step towards supporting MicroPython is adding some flag to the box tests that selects Python interpreter (#75). Let's then compare the outputs and see what tests pass in regular Python and fail in MP so far, then we can learn about more differences in practice.

Two good news I see:

• passing https://github.com/krzema12/kotlin-python/blob/python-backend/python/experiments/out_ir.py to micropython yields no parsing issues, which means that the Python generated for the JS IR stdlib is correctly parsed by MicroPython (not yet executed!)
• we're just starting implementing our stdlib, so we'll be able to take into account both pythons already

[krzema12]

228 more failed tests on MIcroPython: https://github.com/krzema12/kotlin-python/commit/73ed31239e7c1f1511ea02c000c3aba89c485a7f

Failure causes:

• lots of MemoryError: memory allocation failed, allocating XXX bytes - https://forum.micropython.org/viewtopic.php?t=4912

Differences in failure reasons:

• minor: wording in case of unknown name: is not VS isn't - https://github.com/micropython/micropython/pull/8024

[LouisCAD]

I think it'd be best to have a MicroPython specific compiler mode, that also doesn't bring the full Kotlin stdlib, given how far apart the RAM quantities with regular Python running devices and even mobile phones.

[LouisCAD]

Particularly, the kotlin.collections package is probably not something you want to bring into a MicroPython running device. I'd make an embedded systems alternative with typealiases if I want to share some code with other platforms that can use kotlin.collections.

[LouisCAD]

Kotlin to MicroPython for me would be mostly a way to have Kotlin's syntax for MicroPython APIs. Note that you can embed assembly instructions right into MicroPython. This is the kind of thing I'd like to abstract away under an inlined function.

This 30min talk from Damien George, the author of MicroPython himself, is quite informative about optimizations you can do in MicroPython to ensure fastest code: https://www.youtube.com/watch?v=hHec4qL00x0

[krzema12]

I'm afraid it's beyond the current scope of this project. We already struggle with delivering a target for regular Python, and focusing on such optimizations (even if they are necessary sooner or later, I don't deny it) would make the regular Python support way less likely to appear within even 1-2 years.

For now, what I want to do towards MicroPython support, is ensuring correctness. I. e. emitting correct MicroPython code, not really paying attention to the performance and memory. That's the first, mandatory step that I think we can afford. What happens next is in fact optimization that by rule shouldn't happen prematurely. From optimization that we plan to do even for regular Python is dead code elimination, so it would bring only necessary stdlib functions.

Emitting assembly is an entirely different chapter which brings concrete architectures into the picture, and the feature of Python is abstracting them out. Kotlin/Native seems to be closer to this, well, native world. Here we deal with Python and its beauty of being portable.

[krzema12]

We are, of course, open to contributions :) But then we'd have to agree on some architectural stuff as well.

[SerVB]

Now as https://github.com/micropython/micropython/pull/8024 is closed without merging, we can just replace inconsistencies in our test report generator, right?

[krzema12]

Now as micropython/micropython#8024 is closed without merging, we can just replace inconsistencies in our test report generator, right?

Yes, however this is some minor thing out of the whole picture of supporting MicroPython. I'm more worried about MemoryError: memory allocation failed, allocating XXX bytes.

[LouisCAD]

I wasn't suggesting emitting assembly as in having the compiler compile everything to assembly, but simply allow embedded assembly instructions from Kotlin code that would be carried over into the MicroPython output, since it supports embedded assembly code.

I was also thinking about having an ability to bring the the @microptyhon.native decorators that make the code compiled AOT instead of JIT if I understood correctly.

About the MemoryError: memory allocation failed, allocating XXX bytes: Are they happening with code involving parts of the Kotlin stdlib?

[krzema12]

Ok, got it! I have an idea for a generic feature that would allow embedding any text in the Kotlin code, which would then be carried over to the Python output. In particular, it could be any MicroPython-specific stuff. Do you think it would do the job? Like:

val test = listOf(1, 2, 3)

verbatim(
    """
    Any Python/MicroPython code
    """)

The indent in the output would have to be set correctly, but it's a minor issue.

About the memory issue, not sure about what these test cases have in common. Generally we don't yet expect stdlib to work, maybe in some 1% of trivial cases. It's the next step for me to check out why these fail (#84).

[LouisCAD]

That'd work fine. It'd allow to use Kotlin functions to abstract away plain python code.

Could be named python(…), just like we have js(…) in Kotlin/JS.

That said, it'd not be enough on its own for MicroPython: inlining functions, and allowing addition of the decorator I mentioned would be necessary to be doable for code originally written in Kotlin.

[krzema12]

Summary: supporting MicroPython looks realistic, and we generally do want to support it in Kotlin/Python.

I created a milestone to track related work: https://github.com/krzema12/kotlin-python/milestone/2 (its scope will grow as the time goes). We'll be developing Kotlin/Python with MicroPython in mind, and for stuff that doesn't work already while it works on Python, we also have tasks to fix it.