Code sharing

[dhylands]

OK - I'd like to start looking a bit more seriously at code sharing between boards and ports. Some related threads are in #11 #19 #83

I'd like to propose that we create something like

micropython
|-- common
    |-- cortex-m4
        |-- put common m4 files, like gchelper.s here
    |-- bindings
        |-- put common python bindings here
    |-- hal
        |-- hal_gpio.h
        |-- other hal header files which describes the HAL API
    |-- malloc0.c
    |-- string0.c
    |-- other shared files that are more or less independent.
    |-- other directories which might contain other common processor family files.

The share python bindings would be abstract and would refer to HAL_xxx for doing things.

So for gpio, for example, we could have a HAL_gpio_init() function and HAL_GPIO_SET(pin,val) HAL_GPIO_GET(pin), HAL_GPIO_SET_DIR(pin, mode) (or maybe HAL_GPIO_SET_MODE(pin, mode)).

common/hal/hal_gpio.h would describe the API and would #include "hal_gpio_impl.h" stm/hal_gpio_impl.h would implement stm-specific macros stm/hal_gpio.c would implement stm-specific functions (optional)

Each board or board family would implement the appropriate hal functions and macros.

Or maybe we should just use CMSIS, which cover the M3/M4. I also think that FreeRTOS uses CMSIS for the cortex ports (I'm not suggesting that we necessarily want to use FreeRTOS, but some people might, and I wouldn't complain :).

We need to decide how to map pins. I think having some type of external board identifier (i.e. X1, Y1, or for the teensy just an integer number) can map to the internal designator (PORTA pin 13). I think that @Neon22 had some ideas on this front.

We should be able to share the same python bindings for almost all of the functionality (UARTs, LCD, LEDs, GPIOs, Servo, timers, ADC, etc) as long as they aren't too processor specific.

Some things probably need to be split up into 2 layers. For example, the common 4x20, 2x16, etc character based LCDs all use the same commands, but they can be connected directly via GPIO pins, or they be sitting on a gpio expander. I'm aware of at least a couple common GPIO expander variations, like the PCF8574 or the MCP23008, and even there, the mappings from the i2c expander pins to the LCD pins often differ slightly. So you wind up with one layer which is for talking to the LCD, and a second layer for delivering the LCD commands to the actual LCD device (over gpio, i2c expander A, i2x expander B, etc).

I happen to have: http://www.adafruit.com/products/292 and http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=370802573051&ssPageName=ADME:L:OU:CA:3160 but there are many other variations.

[pfalcon]

But as previous tickets suggest, it's hard to plan such migration "from top" - there're too many questions to consider.

Why not start from opposite direction - find specific pain point, resolve it, the repeat.

Here's specific pain points I may suggest:

1. garbage collector now lives in stm port; unix port doesn't have GC at all. GC is generic component, which should be moved to generic code (with the idea that there may be alternative implementations; I'd consider adding top-level dir for them, gc/).
2. I started introducing file object hierarchy (io module). Don't you want to port stm/teensy to use it, which will allow it enjoy lot of functionality "for free", once basic stream interface (read()/write()) is implemented.

These are very specific and useful tasks which will allow to collect experience how to proceed further.

[pfalcon]

Additionally:

|-- common

There's already dir for common files, py/ . But given context, it's not really "common", but something else ("bare-metal", "embedded", note that even that is confusing - you can run under linux and have direct access to hardware registers).

|-- cortex-m4
   |-- put common m4 files, like gchelper.s here

gchelper.s is not "cortex-m4", it's not even "cortex-m", it's "cortex", though I'd guess it's just a few changes would make it just "arm".

[dhylands]

But as previous tickets suggest, it's hard to plan such migration "from top" - there're too many questions to consider.

Why not start from opposite direction - find specific pain point, resolve it, the repeat.

I have run into a specific pain point. Right now none of the python bindings are shared between the teensy and the stm. I'd also like to port MicroPython to the BeagleBoneBlack (running linux) and I'd like to use the same bindings on all 3 platforms. But I figured I'll wait until we have 2 platforms sharing bindings before I go and add a third.

It seems to me that a huge portion of whats in stm/main.c should be factored into common code.

I'm also very interested in exploring the file object hierarchy stuff.

I ordered all 3 STM discovery boards (the 401, 407 and 429 and have the 401 and 429 in my possession and should have the 407 on Tuesday).

[dhylands]

Or maybe we should just use CMSIS, which cover the M3/M4. I also think that FreeRTOS uses CMSIS for the cortex ports (I'm not suggesting that we necessarily want to use FreeRTOS, but some people might, and I wouldn't complain :).

Or maybe not. I went and downloaded CMSIS from the arm site and it hardly has any peripherals. I think I was thinking of the CMSIS library that comes from NXP (I brought up FreeRTOS on the mbed a while back).

[pfalcon]

CMSIS is nothing but register definitions, the only thing it mandates is usage of structures to represent MCU blocks. CMSIS is individual per specific MCU family/model. What ARM provides is generic "Cortex-M" CMSIS, and peripherals of generic Cortex-M end with SysTick and NVIC. So, with vendor CMSIS, you would need to write "drivers" using those definitions yourself - in general, individually for each vendor's MCUs. And that's rather far goal for a project like MicroPython.

11 gives much better idea how to deal with it - find a project which provides "drivers" with consistent interface across as many MCU families as possible. Use it as upstream. If adding feature or new MCU, work with upstream. Of course, that's "idealized" approach, but trying to ignore it means getting swamped with low-level hardware stuff instead of just enabling small efficient Python on hardware.

[dhylands]

@pfalcon Thanks for prodding me to look into the mbed hal stuff. That's the idea of what I wanted to do, but already implemented. So that seems like the API to focus on.

It looks generic enough to work for all of the chips I'm interested in working with. I'll get it up and running with my mbed and then look at porting to the teensy and stm32f.

Oh it looks like the STM32F4xx is already supported.

[Neon22]

Well maybe we could start with virtual pins/ports to physical pin mapping. I suggest this because even on the pyboard - we want this. E.g. port A(0-7) is really pins (1,15,16,17,20,21,22,23) And Damien has a different mapping on his pyb4 version using Ports X, Y E.g. (trying to capture the pyb4 data)

# public edge Label, physical pin, [capabilitiues]
# Y1 = 29, PB10/SCL2/PU/TX3/PWM
# Y2 = 30, PB11/SDA2/PU/RX3/PWM
# Y3 = 26, PB0/ADC/PWM
# Y4 = 27, PB1/ADC/PWM
# Y5 = RST
# Y6 = GND
# Y7 = 3V3
# Y8 = VIN
# Y9  = 37, PC6/TX6/PWM
# Y10 = 38, PC7/RX6/PWM
# Y11 = 61, PB8/CANRX1/PWM
# Y12 = 62, PB9/CANTX1/PWM
# Y13 = 33, PB12/SS2/CANRX2
# Y14 = 34, PB13/SCK2/CANTX2/PWM
# Y15 = 35, PB14/MISO2/PWM
# Y16 = 36, PB15/MOSI2/PWM
#
# X1 = 58, PB6/SCL1/PU/TXD1/PWM/MMA-SCL
# X2 = 59, PB7/SDA1/PU/RXD1/PWM/MMA-SDA
# X3 = 24, PC4/ADC
# X4 = 25, PC5/ADC
# X5 = RST
# X6 = GND
# X7 = 3V3
# X8 = VIN
# X9  = 1,  PA0/TX4/ADC/PWM/SERVO
# X10 = 15, PA1/RX4/ADC/PWM/SERVO
# X11 = 16, PA2/TX2/ADC/PWM/SERVO
# X12 = 17, PA3/TX2/ADC/PWM/SERVO
# X13 = 20, PA4/SS1/ADC/DAC
# X14 = 21, PA5/SCK1/ADC/DAC/PWM
# X15 = 22, PA6/MISO1/ADC/PWM
# X16 = 23, PA7/MOSI1/ADC/PWM

# X17  = 55, PB3/USB_SW/TRACESWD/TD0
# X17A = 3V3
# X18  = 2,  PC13/SDIO_SW
# X18A = 60, BOOT0
# X19  = 8,  PC0/ADC
# X19A = 46, PA13/LED_R/SWDIO/JTAG-TMS
# X20  = 9,  PC1/ADC
# X20A = 49, PA14/LED_G/SWCLK/TCK
# X21  = 10, PC2/ADC
# X21A = VBAT
# X22  = 11, PC3/ADC
# X22A = GND

# INT = 42, PA9/OTG_VBUS
# INT = 43, PA10/OTG-ID
# INT = 44, PA11/USB-OM
# INT = 45, PA12/USB-OP
# INT = 39, PC8/SDIO_D0
# INT = 40, PC9/SDIO_D1
# INT = 51, PC10/SDIO_D2
# INT = 52, PC11/SDIO_D3
# INT = 53, PC12/SDIO_CK
# INT = 54, PD2/SDIO_CMD
# INT = 28, PB2/BOOT1/MMA-INT
# INT = 57, PB5/MMA-AV00
# INT = 56, PB4/LED_B/JTAG-TRST
# INT = 50, PA15/LED_Y/TDI

If I might suggest we implement this as one or more c structs but we also expose it as a python class so Python can inspect and use same data defined in one place. e.g.

class hardware_map():
  pins = {}
  ports = {}
 def __init__(self):
    pass

I put the initial pinpool class and idea in here: https://github.com/micropython/micropython/issues/125 This would need to change to use pin labels strings instead of virtual pin numbers (or both)

[dhylands]

@Neon22 So if we proceed with using the mbed hal layer, then it has already defined the pin to number mapping used by the hal layer. See: https://github.com/mbedmicro/mbed/tree/master/libraries/mbed/targets/hal/TARGET_STM/TARGET_STM32F4XX and look at PinNames.h

So that basically does a port/pin to integer mapping.

I think that it makes sense to map the Xx ant Yy to their corresponding pins (power ground and the like don't need a mapping).

So on the PYBOARD, "Y1" is PB10, which winds up having the integer pin number 26 (PB_10 or since B = 2, 2 * 16 + 10 = 26)

So from python, we can have a dictionary that maps "Y1" to 26 and 26 is what the underlying hal layer would use.

Does that make sense?

[dhylands]

ahh @dhylands you mean this: https://github.com/mbedmicro/mbed/tree/master/libraries/mbed

Yeah - I think our emails crossed paths

[Neon22]

Yes I just found that. looks neat and tidy. I agree no need for voltages etc I just put them all in in a hurry. Looks pretty concise - is it ? making a python module on top to expose the same info also looks quite doable...

[Neon22]

once we get port/pin mapping sortedout - I'm hoping some of the others will become clearer too - malloc...

[dhylands]

Looks pretty clean to me.

It needs to be fleshed out some more - like I don't seen any PWM yet (although there are some definitions in device.h and PeripheralNames.h)

But this seems like a decent base to use.

[Neon22]

will you 'import' it like the github cloning approach used for micromusl (?) - so inheritance is clear and if you add PWM etc - then it might get dragged back...?

[dhylands]

I haven't looked at the micromusl stuff yet.

I'd expect that the mbed git repository would just be a dependency and you'd need to checkout both to get micropython to build. It makes sense to record the "working" hashes so that you can get a known good version.

The other option would be to fork the mbed repository as say micropython/mbed and then merge periodically from upstream.

I'd be inclined to add new target upstream though and just pull them into micropython, but I haven't really done enough of this type of project.

If you clone it and check it in new, then it becomes much more difficult to pull in upstream stuff I think.

[chrismas9]

Another potential source of device drivers is CoX from coocox.org. They are part of Embest which is owned by Farnell/Element14. They don't support many MCUs yet but more coming this year. I probably favour mbed but CoX might fill in some gaps.

[Neon22]

The Coocox work seems interesting. E.g. here's a link to their I2C implementation: http://coocox.org/cox/manual/interface/group__x_i2_c___exported___a_p_is.html

So useful for exposing the on-chip functionality... Timer, UART, SPI - all done there... E.g. setting up the Watchdog timer: http://coocox.org/cox/manual/interface/group___w_d_t.html

All under BSD license: http://coocox.org/cox/manual/interface/index.html

[dpgeorge]

I would lean towards using mbed HAL. It is being actively worked on. My concern is that it is not mature yet and not that featureful. Also, I'd like to know more about how thick/thin it is. Eg, do you have to allocate memory just to create an I2C object? How many layers are there to switch a GPIO on/off? I would prefer using macros so as to avoid a long chain of function calls glueing everything together.

@pfalcon there is already a lot of common GC stuff in py/gc.c that can be used on the unix port.

@Neon22 I don't think you need to worry about the physical pin number that a GPIO port is on. You just reference it by, eg, PB10.

I think organising the file hierarchy is a difficult thing. Let's just try something minimal and sensible for now, and not be afraid to change it later when we outgrow it.

How about we start by making a embed/ directory for everything related to embedded devices (I2C, USART, servo, etc), and that is not specific to a board/chip.

[dhylands]

I would lean towards using mbed HAL. It is being actively worked on. My concern is that it is not mature yet and not that featureful. Also, I'd like to know more about how thick/thin it is. Eg, do you have to allocate memory just to create an I2C object? How many layers are there to switch a GPIO on/off? I would prefer using macros so as to avoid a long chain of function calls glueing everything together.

@dpgeorge So I took a look at the mbed HAL layer, and it looks pretty thin.

Each module, gpio spi, i2c, analog, pwm, etc has a structure and a bunch of associated functions.

The code expects the caller to allocate and mange the structure, so there is no memory allocations happening internally.

For GPIO, due to the way the abstraction is done they could have chosen a a fast method or a slower method that uses less memory. The current implementation requires 6 words (24 bytes) of storage for each gpio pin that you need to use as GPIO. On the plus side, the functions for reading and writing a pin are very small and written inline, so should give you similar performance to using a macro.

The pyb_gpio function doesn't have an object associate with it, but if it were reimplemented like the pyb_Led code, then the mbed HAL implementation would fit right in.

The current functionality includes:

• setting the pinmux to configure a pin as gpio
• setting the direction of a pin
• allows the mode (none, pullup, pulldown, opendrain) to be set
• setting/getting the value of a pin

It would be possible to provide an alternate implementation of the API which used much less memory per pin, but did a bit more processing in order to set/get the values.

The i2c read and write functions are synchronous, so they don't return until the operation is completed.

The complete list of HAL APIs is: analogin, analogout, can, ethernet, gpio, gpio_irq, i2c, pinmap, port, pwmout, rtc, serial, sleep, spi, us_ticker.

The STM32F4xx target currently implements analogin, gpio, i2c, pinmap, port, spi, us_ticker.

The following are not currently implemented on STM32F4xx: analogout, can, ethernet, gpio_irq, pwmout, rtc, serial, sleep,

So as long you're ok with creating the "pythonic" implementations (which would have some small amount of memory per peripheral object - typically only 1 word, but varies by object) and we're willing to use the HAL API, then we can use it and create common python bindings.

[Neon22]

For an embedded cpu with ability to adjust clock - it would also be nice if we could also get access to sleeping the chip (and the watchdog timer - if not in the mbed HAL then from the coocox). Probably a diff thread...

[dpgeorge]

To wrap the mbed HAL code in Micro Python bindings would be pretty easy. For the GPIO, one would make a uPy object like:

struct mp_obj_gpio_t {
    mp_obj_base_t base;
    gpio_t gpio;
};

That means 8 words per pin. It could be reduced at the cost of speed, but that means rewriting the HAL code.

mbed HAL is clean, but also lacking STM functionality. It would be nice to support, but there would be some work to get all the features and I don't want to turn the uPy project into a "let's implement a clean and portable HAL" project.

[dhylands]

mbed HAL is clean, but also lacking STM functionality. It would be nice to support, but there would be some work to get all the features and I don't want to turn the uPy project into a "let's implement a clean and portable HAL" project.

So it looks like a bunch of STM functionality is present (https://github.com/mbedmicro/mbed/tree/master/libraries/mbed/targets/hal/TARGET_STM/TARGET_STM32F4XX) . So I was going to start with that, and expand as required.

The mbed stuff produces a lib which we could link against, although the sources that go into that lib are a mixture of C and C++ (the hal appears to be all C).

So then the question becomes how would you like to import this into micropython?

Some options that come to mind:

1 - We could checkout the mbed tree, build the libs using their build system, which produces a lib per target. We could reference that lib, or check it into the python tree

2 - We could checkout the mbed tree, reference the appropriate source files and build them using the micropython build.

3 - We could copy the mbed hal directories into the micropython source tree.

I'm inclined to use 2, but I'm happy to do whatever you'd prefer.

The mbed hal provides a bunch of headers which duplicate ones found in micropython. For example micropython/stm/lib/stm32f4xx.h (which is version 1.3) mbed/libraries/mbed/targets/cmsis/TARGET_STM/TARGET_STM32F4XX/stm32f4xx.h (which is version 1.1)

I'm also happy to try and get the 1.3 headers into the mbed tree.

In any case, we should probably fork the mbed tree into the micropython org and use a "micropython" branch. Then we can upstream any improvements and advance the micropython branch appropriately and not break micropython project due to some incompatible change in the mbed tree.

I'm currently in the process of cleaning up the Makefiles so that I can add the proper dependency support, and I think adding the hal stuff should wait until thats done.

[dpgeorge]

I don't want to link against a library (anyway you need to do some compiling of the HAL for all the inlined header code).

I don't want to use any C++ code.

Would be best if we could track and merge any changes they make. I like your suggestion of forking mbed into the MP organisation and keeping our own version at least somewhat stable.

Let's wait until the depdency support is done.

[dpgeorge]

Since this discussion was had, lots of things have changed and moved on. In particular extmod/ and lib/ directories are used to store code common to all the ports. Discussion about further refactoring of the directory hierarchy is in #2401.

Regarding a HAL: from experience with the microbit port mbed is actually a bit bloated for uPy's use, since it pulls in a lot of newlib, has its own dedicated heap, and uses RAM for basic peripherals when it could be avoided.