Feature: On-target execution tests [WIP]

[agnat]

Still work in progress, but fun to play with. From the README:

AVR LLVM Integrated Tester

This tool builds an AVR executable from test code lowered by our backend and the libavrlit test suite library using a known good toolchain (avr-gcc). The resulting binary is uploaded to a development board. Test results are collected using a virtual tty.

Setup

Things you will need:

• ATmega32U4 board with USB and AVR-109 type bootloader. (Arduino Leonardo, RedBear Blend (Micro), &c.)
• pySerial python module
• Fire extinguisher

Set the AVRLIT_PORT environment variable to the tty path of your board.

> export AVRLIT_PORT=/dev/tty.usbmodemfd141

If your board currently runs an Arduino sketch that uses the serial port, you are all set. Otherwise, you need to reset the board manually for the first run.

> bin/llvm-lit -v ../llvm/test/CodeGen/AVR/

Writing Tests

The on-target execution tests reside in llvm/test/CodeGen/AVR. Like other lit tests they contain a RUN: line calling llvm-avrlit:

// RUN: llvm-avrlit %s %p/add.ll 

#include <avrlit.h>

using namespace avrlit;

extern "C" {  // actually this is extern "IR" but Bjarne forgot.
  i8 add8_reg_reg(i8, i8);
}

AVRLIT_TEST(llvm_avr) {
  reenter (this) {  // don't worry about the coroutine
    plan(3);
    ok(_(add8_reg_reg( 23, 42) ==  23 + 42)); yield;
    ok(_(add8_reg_reg(-23, 42) == -23 + 42)); yield;
    ok(_(add8_reg_reg(-23, 42) == 0));        yield;
  }
}

All of this is still in flux. I'll explain it if I decide to keep it. ;)

[agnat]

Ready for review... and test drive ;)

[dylanmckay]

Where is add8_reg_reg &co defined?

Also, do you think this would be worth putting into a seperate repository? It is quite a bit less lightweight than any other test, and won't be useful to users.

[agnat]

add8_reg_reg &co are defined in add.ll &co.

Take a look at the // RUN: ... line above and $OBJ/test/CodeGen/AVR/execute_on_target_test/Makefile. Yeah... I know ... totally cool... :-D

I'd rather keep it in-tree. It's only a hand full of files and we're talking about the LLVM repository. It's huge no matter what. Let's drown out the static overhead by writing a bazillion tests.

In the long run I'd like to make it part of the CI. All we need is a build bot and an Arduino.

[agnat]

Ah, right. you don't have a board yet. Here is the makefile...

# Generated by llvm-avrlit. Do not edit.
AVRLIT_BOARD  ?= leonardo
AVRLITD = /Users/david/projects/avr-llvm/llvm/utils/avr/avrlit/libavrlit

include $(AVRLITD)/$(AVRLIT_BOARD)/board.mk

execute_on_target_test.$(AVRLIT_BOARD).hex:

clean:
    rm -f *.hex *.elf *.o

.PHONY: clean

execute_on_target_test.$(AVRLIT_BOARD).hex : execute_on_target_test.$(AVRLIT_BOARD).elf
    avr-objcopy -O ihex -R .eeprom -R .fuse -R .lock -R .signature $< $@

execute_on_target_test.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/execute_on_target_test.cpp
    avr-g++ -Os -mmcu=$(MCU) -std=c++11 -I$(AVRLITD) -c $< -ffunction-sections -o $@

add.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/add.ll
    llc -mtriple=avr-atmel-none $< -filetype=obj -o $@

and.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/and.ll
    llc -mtriple=avr-atmel-none $< -filetype=obj -o $@

xor.$(AVRLIT_BOARD).o : /Users/david/projects/avr-llvm/llvm/test/CodeGen/AVR/xor.ll
    llc -mtriple=avr-atmel-none $< -filetype=obj -o $@

execute_on_target_test.$(AVRLIT_BOARD).elf: execute_on_target_test.$(AVRLIT_BOARD).o add.$(AVRLIT_BOARD).o and.$(AVRLIT_BOARD).o xor.$(AVRLIT_BOARD).o
    avr-g++ -Os -mmcu=$(MCU) -std=c++11 $^ -L$(AVRLITD)/$(AVRLIT_BOARD) -lavrlit -Wl,--gc-sections -Wl,--relax -o $@

Also note that usb.h, usb.c and LUFAConfig.h are pretty boilerplate. The interesting things happen in avrlit.h, avrlit.cpp and llvm-avrlit.in.

[agnat]

I found this post regarding the PIC backend a while back, lost it, and found it again.

Quote:

However, a back-end maintainer has to provide a few "guarantees" to continue supporting in tree:

1. You must have a public buildbot that can test your code. Since you'll always cross compile, having a good set of tests in x86 mode would do, but having execution tests would increase the quality by a large amount. If at all possible, I'd recommend a cross-compilation buildbot that runs some embedded code in a proper PIC.

[agnat]

One defect of the current design is that it allows only one target execution test. That's because llvm-lit runs tests in parallel. Since llvm-avrlit does not perform any locking there currently is a race on the tty device (chaos).

We have a little time on this. In the long run we need some sort of serialization.

[dylanmckay]

Can you add e0cc7d56a66d20204b50d1c562c6d1e884664271, I can't seem to push a commit to your PR (probably doing it wrong).

Note that I must run llvm-lit as root otherwise the script is unable to access the tty device.

When I do, llvm-lit succeeds - but the on_target_execution_test.cpp test is marked unsupported.

[agnat]

Can you add e0cc7d5 [...]

Done.

Note that I must run llvm-lit as root otherwise the script is unable to access the tty device.

Hm, not good. There probably is a cleaner (distro specific) way to do this. There often it is a user group you need to member in to access the ttys.

When I do, llvm-lit succeeds - but the on_target_execution_test.cpp test is marked unsupported.

Unsupported means AVRLIT_PORT is not set. This is handled in lit.local.cfg. You probably use sudo?... which creates a fresh environment loosing your setting in the outer shell? I'm guessing...

[dylanmckay]

Could you also add 8982a31d2401005666a643fbac19f8c448eca09f.

I got it to run by adding my user to the relevant groups, however llvm-lit hangs when running the test. I am not sure if this has something to do with the board I'm using - a Freetronics LeoStick. It is a leonardo compatible however.

Feel free to merge,

[agnat]

Pretty sure it is not the board. I'd guess it's either a timing issue (I tested it on OS X only) or more python incompetence on my end.

If you don't feel like debugging it just drop your distro name. Then I'll set up a vbox.

If you want to look into it, here are a few things to try:

• Try running llvm-avrlit ... manually to get console output. You'll need to add $OBJ/bin to your path, so it does find our llc.
• Try this line as a standalone script. It should throw your stick into the boot loader. If it doesn't, add a bit of delay before closing the tty. (These boards reset when the tty is closed while being set to 1200bps).
• Upload $OBJ/test/CodeGen/AVR/execute_on_target_test/execute_on_target_test.leonardo.hex manually using avrdude
• Look at the test output using your favorite terminal (CoolTerm, minicom, screen)
• print(...)-debug the living :poop: out of it...

Could you also add 8982a31.

Done.

Feel free to merge,

Will do.

[dylanmckay]

I'll follow your steps in the next few days, I'll update once I get around to it :)