Standardizing hardware access

[pfalcon]

stmhal has (undocumented) "stm" module which provides physical memory access. I'd like to propose to standardize it - under different name of course. As "stm" is not documented, moving mem* to a new module hopefully shouldn't be a big problem. I propose "physmem" as a module name. Alternatively, to save on modules, we can put it into uctypes (not sure that's ideal though).

[dpgeorge]

I recall that you did not like so much the stm.mem* accessors. Did you have a suggestion to change how they work, or just want to move them as-is into physmem (or other module)?

Can uctypes be used to do the same thing, albeit in a slightly more round-about way? Note also that viper mode can access raw memory: p = ptr16(0x2000000); p[0] = 123

[pfalcon]

I recall that you did not like so much the stm.mem* accessors. Did you have a suggestion to change how they work, or just want to move them as-is into physmem (or other module)?

Yep, I had, but I had to adjust - it's hard to argue it's easy to use and people use it, and I myself could use it for easy hack. So, just moving .mem* to physmem, leaving specific constants in "stm".

Can uctypes be used to do the same thing, albeit in a slightly more round-about way?

It can be, yes. But again, it's hard to argue against need to have something dead simple for quick hacks. (Just the same as uctypes can do much more than accessing untyped values in "raw memory").

Note also that viper mode can access raw memory: p = ptr16(0x2000000); p[0] = 123

OMG ;-)

[dpgeorge]

I propose "physmem" as a module name. Alternatively, to save on modules, we can put it into uctypes (not sure that's ideal though).

Could use micropython module, but then that might end up being a ground for all ad-hoc functions.

But I fear that physmem is such a tiny module that it'll only even have these mem access functions.

What about something that we can imagine having some broader functionality/scope in the future, something like machine for all low-level stuff? machine.mem8 is very readable and understandable. Could have port8 etc methods for special port memory that is not memory mapped.

[dhylands]

Some other names to consider: mem and cpu (other than machine)

[dhylands]

I think physmem is actually misleading, since from linux user space you can't actually access phymem, only virtual memory. You can use memmap to map some physical pages to a virtual memory space, but you still access using virtual addresses.

[pfalcon]

I think physmem is actually misleading, since from linux user space you can't actually access phymem, only virtual memory.

And yet that's the intention of that module. So, Linux implementation will mmap /dev/mem underneath, etc.

[pfalcon]

Could use micropython module, but then that might end up being a ground for all ad-hoc functions.

IMHO not, micropython is to control interpreter/implementation.

But I fear that physmem is such a tiny module that it'll only even have these mem access functions.

Yes, but it will be clean and pure. Other option mentioned is "uctypes". Light reasons against it is: 1) putting similar (competing) means into a single module; 2) as mentioned above, similarity is surface, actually 2 means deal with pretty different matters: hardware access (specifically, memory) vs foreign data structures access.

something like machine for all low-level stuff?

Well, with such formulation, it sounds like cleaned and purified refactor of "pyb". If we're ready for that, let's do it (but of course, carefully consider each piece added there).

[dpgeorge]

I would prefer machine. It would be a place for methods that are "board" agnostic. I guess only pyboards should have the pyb module.

[dpgeorge]

If we go with machine I think we can put power-control modes in there:

• machine.sleep() or machine.idle() - enter low power mode waiting for something to happen. On ST MCUs this is a wfi instruction. On machines that don't support this, it is a nop.
• machine.suspend() - halt the machine, waiting for a special wakeup event, and continue execution from this point after waking. On ST MCUs this is "stop mode".
• machine.standby() - halt the machine and go into a deep power saving mode. After waking up the machine resets.

Also, we can add hard_reset() to this module (perhaps renaming it to simply reset or reboot).

[danicampora]

I think the machine naming is fine. I like machine.sleep() better the idle() :-), also, I like the idea of renaming hard_reset() to just reset()

[dpgeorge]

@danicampora does the CC3200 have low-power modes that these names/functions can be easily mapped to?

[danicampora]

@dpgeorge Yes!

sleep() it's wfi as well suspend() can be mapped to the deep_sleep_mode of the CC3200 standby() will be mapped to hibernate_mode

Actually, I would suggest to use something like machine.hibernate() instead of machine.standby(), since I think it's more clear what the intention is. standby sounds more like what sleep() or idle() would do (simply halting the processor).

[dpgeorge]

Ok. We should do a small survey of other MCUs to see if they can can map to these functions easily.

Does the CC3200 have backup RAM that does not get lost when waking from hibernate? Does hibernate reset the MCU upon waking? What events can trigger a wakeup from deep_sleep_mode and hibernate_mode?

[danicampora]

Actually the CC3200 has a fourth low-power mode, I will explain them briefly:

1. Sleep (wfi)
2. DeepSleep, which is like Sleep but consumes a bit less. DMA does not work under deepsleep but it does under "normal" sleep.
3. Low Power Deep Sleep. RAM retention is configurable, and the whole RAM can be retained if desired. SoC current consumption goes down to 700uA with WiFi enabled (and still connected to the AP). The chip can be configured to wake up and continue execution from any where. WiFi connectivity is not affected when entering and leaving this mode. GPIOs, timers, and also WiFi events can be used to wake up.
4. Hibernate . No RAM retention, no WiFi enabled. After wake up, the machine resets. GPIOs and a special timer can be used as the wake up source.

I was planning to use Sleep for machine.sleep(), LowPowerDeepSleep for machine.suspend(), and hibernate for the last mode.

[danicampora]

The ability to quickly wake up from LowPowerDeepSleep without loosing the WiFi connection is one of the strong points of the CC3200.

[dpgeorge]

@pfalcon any objections to adding the machine module and using it for low power modes as well?

[pfalcon]

@pfalcon any objections to adding the machine module and using it for low power modes as well?

Ad I wrote above, I'm all for making this "machine" module a cleaned-up, fully target-independent interface to hardware (portable and recapitalized replacement for pyb module).

[pfalcon]

Specific additions:

Also, we can add hard_reset() to this module (perhaps renaming it to simply reset or reboot).

+1 to make it just reset().

[pfalcon]

machine.sleep() or machine.idle() - enter low power mode waiting for something to happen. On ST MCUs this is a wfi instruction. On machines that don't support this, it is a nop.

I wouldn't treat "wfi" as real low-power, and thus find using sleep() for that confusing. idle() better corresponds to what actually happens.

[pfalcon]

machine.standby() - halt the machine and go into a deep power saving mode. After waking up the machine resets.

Maybe it's my non-native English, but I also don't find it intuitive that standby() is the highest power-saving method. Maybe deepsleep() is more intuitive?

[danicampora]

What about: idle() suspend() hibernate() ?

[pfalcon]

And otherwise, creating this machine module is in my TODO, but I didn't get to it so far, so feel free to beat me on it.

But as discussed above, it's peculiar module - some parts are reusable across ports, some not, and to a different degree. (For example, mem* will be the same for all baremetal ports (yes, let's remove any bound-checking), but different for unix).

[dpgeorge]

But as discussed above, it's peculiar module - some parts are reusable across ports, some not, and to a different degree.

The important thing is the Python API, not the underlying implementation. And because we want to target multiple machines with this, designing a good API is going to be tough.

Maybe it's my non-native English, but I also don't find it intuitive that standby() is the highest power-saving method. Maybe deepsleep() is more intuitive?

I like @danicampora's suggestion of hibernate for this function.

I wouldn't treat "wfi" as real low-power, and thus find using sleep() for that confusing. idle() better corresponds to what actually happens.

Calling idle() doesn't really sound like an active verb, it could be mistaken for an "am I idle?" call. wfi does actually sleep the CPU. If there are no interrupts (eg irqs are disabled) then it halts forever. Power consumption drops by about 1/2 just putting wfi in a busy wait loop. Given all this, I'd rather call it sleep().

[dpgeorge]

portable and recapitalized replacement for pyb module

I'd like to keep pyb as a specific module for all "pyboards". I don't think it's realistic to try and make a generic module that will cover all possible peripherals of all possible boards. At least, it will take a lot of research into different boards capabilities, and then a lot of time coming up with a good design that is general and flexible.

But, the idea of starting with machine module and gradually introducing very generic functions, is a good thing.

[pfalcon]

and recapitalized replacement

Weird, I was typing "perfectalized" ;-)

[pfalcon]

But, the idea of starting with machine module and gradually introducing very generic functions, is a good thing.

Yes, that's what I mean either.

[dpgeorge]

Note that if we put idle/sleep/suspend/hibernate in machine then one still needs board-specific functions to wake the machine from the sleep state. On pyboard Pin and RTC can be used to wake up, eg:

pyb.RTC().wakeup(10000) # trigger a wakeup every 10s
machine.suspend() # suspend, waking up after 10s

Does it seem strange to need to use both pyb and machine modules for low-power functionality?

@danicampora how does the cc3200 wake up?

[danicampora]

Maybe it seems a bit strange.

The CC3200 wakes up with Pin, RTC and WLAN.

[dpgeorge]

Also, we can add hard_reset() to this module (perhaps renaming it to simply reset or reboot).

+1 to make it just reset().

How about reboot()? Reset is rather generic and weak sounding. Reboot really tells you the machine is going to start from scratch.

[pfalcon]

Reset is rather generic and weak sounding. Reboot really tells you the machine is going to start from scratch.

All I can say is that sleep() for not really a sleep (just imagine that every place where it's used will require a comment that no, it's not really a sleep, it's just a pause till something happens), hibernate(), reboot() sound strange for me when applied to deeply embedded systems. It sounds more naturally when applied to "big" machine. where sleep() may be indeed implemented via sleep() syscall, hibernate can indeed hibernate after saving state to non-volatile storage (that's how I'd expect operation named "hibernate" to work), and reboot is indeed a reboot. I'm not sure if this "similarity" to big machine operations is helpful or misleading.

Well, I guess something should be just selected, it's just I don't feel that the selections will be intuitive.

[dpgeorge]

This discussion regarding low-power modes is off-topic, but here is more fuel for the fire. Atmel's modes for the SAM3 family are:

• sleep: =wfi/wfe; CPU clock stopped, peripherals still going
• wait: =suspend; CPU and peripheral clocks stopped, SRAM retained, wake on RTC or pin event
• backup: =hibernate; everything stopped, reset on waking from RTC alarm

That corresponds pretty much exactly to ST's sleep(=wfi/wfe), stop and standby modes.

[danicampora]

Every manufacturer has its own naming, and some are less intuitive than others. I vote for:

sleep()
suspend()
hibernate()

By the way, reboot sounds good to me.

[dpgeorge]

Atmel's AVR32 CPU has 1 generic "SLEEP" instruction that takes an argument to set the sleep mode. The different modes depend on the implementation of a given MCU. The UC3A has:

• idle: only CPU clock stopped, wake by irq
• frozen: CPU and bus stopped, wake by peripheral irq
• standby: all clocks stopped (PLL still running), wake by RTC or ext int
• stop: all clocks stopped including PLL, wake by RTC or ext int
• deepstop: similar to stop but a few more things turned off
• static: everything off, wake by async ext int or reset pin

[dpgeorge]

Fleescale Kinetis family has 11 low power modes:

• wait
• stop
• VLPR (very low power run)
• VLPW (very low power wait)
• VLPS (very low power stop)
• LLS (low leakage stop)
• VLLS3, VLLS2, VLLS1, VLLS0 (very low leakage stop 3, 2, 1, 0)
• BAT (backup battery only)

"wait" looks to be similar to wfi/wfe. "stop" looks to be like a suspend. "VLPx" modes scale the clock down to 4MHz. "LLS" seems to be a low power version of "stop" where SRAM is still retained. "VLLSx" modes are like hibernate, in that the MCU goes through a reset upon waking, although some of them partially retain the SRAM.

[dpgeorge]

I don't see how to provide a generic enough API in machine to handle all these different cases. And it's not good enough to just provide 3 generic modes (eg idle, suspend, hibernate) and hope that an MCU can map them to something sensible. This is because in the case of low power you really want to squeeze the last microamp out, and therefore you need a full understanding of the MCU you're running on, as well as full and fine-grained control of all its low-power modes.

Providing generic functions in machine will only lead us to provide additional MCU-specific functions in another module, in which case the generic machine ones will go unused.

Could just have 1 single function machine.sleep(...) with the arguments to this function unspecified, ie MCU-specific. But I don't think that's a good way to do it.

[pfalcon]

And it's not good enough to just provide 3 generic modes (eg idle, suspend, hibernate)

I don't see any problem with that, and would assume that's exactly what people using portable language like Python want.

[danicampora]

I don't see any problem with that, and would assume that's exactly what people using portable language like Python want.

I agree with that.

But...

This is because in the case of low power you really want to squeeze the last microamp out, and therefore you need a full understanding of the MCU you're running on, as well as full and fine-grained control of all its low-power modes.

Also with this other statement.

So, it is not easy to find the balance between a simple/friendly API, and one that gives full control over the hardware.

[danicampora]

Even though most processors and SoCs support several low power modes, the great majority of applications only use 2 or 3, since handling sleep/wake cycles is tricky when the software becomes complex, specially when it involves stopping and re-initializing peripherals, preserving only a part of the RAM, and so on. People using uPy most likely want a simple yet flexible API, so we need to find the balance to provide that. In my experience regarding low power modes, it boils down (most of the time) to 3 use cases:

1. The application is active, but not with a 100% duty cycle, so you want to save power while the processors is idle. ARM's wfi fits well here (other architectures like the AVR32 also have something analogue to it).
2. The application needs to power down periodically (or sporadically), but needs to wake-up quickly, perform some operations continuing from the point where the code entered the low power mode. So, needs RAM retention and hopefully peripherals remember their state after leaving the low power state.
3. The application needs to power down for long periods, so it needs to use the most drastic low power mode, and it can afford to spend a bit more time during the sleep/wake procedures.

It is our responsibility as developers of each port to select the low power modes of the platform that better map to those 3 uses cases, and implement them in a way that is transparent from the MicroPython perspective.

Just my 2 cents.

[dpgeorge]

I understand the above points regarding relative simplicity, but it's been my/our experience that all the pyboard peripheral classes are moving towards exposing every little detail of the MCU. Eg I initially thought for pyb.Timer that simple counting and PWM would be enough, but now it has almost everything exposed including eg deadtime. And users are using the features. Other examples: I2C mem_read/write operations wanted memory width selection, SPI wanted DMA (and still wants callback when DMA is done), UART wants rx callbacks, CAN wants lots, DAC wants 12 bit support #1130, SPI wants callbacks #1124. All of these are useful things. Mome fit a generic model while others need very specific MCU support.

With low power modes, what happens when we need the wfe instruction? Where does it go? Do we make machine.idle take an option which selects the type of idle (wfi or wfe in Cortex-M case)? As I said above, I'm afraid we will soon outgrow the machine.{idle,suspend,deepsleep/hibernate} API and then it becomes unused legacy code.

Things like delay, udelay, millis, micros, elapsed_millis and elapsed_micros seem much more suited to generic machine functions.

[danicampora]

Well, those are indeed very good points. I guess that if we want to expose every little detail of the MCU, we can not go for a generic API.

[dpgeorge]

For reference to a real-life example where naming these things is important (pyb.wfi), see http://forum.micropython.org/viewtopic.php?f=2&t=656#p3758

[danicampora]

I also commented on your recent commit that documents the RTC wake-up functionality. Maybe we should try to reach an agreement on a common generic API for all callbacks. I already implemented one version of this for the CC3200 that even allows the user to configure from which power mode(s) the interrupt can wake the device.

https://github.com/micropython/micropython/blob/master/cc3200/misc/mpcallback.c

It's quite flexible so I think we can work on top of that to make something that will suit stmhal and other platforms as well.

[pfalcon]

"machine" module was introduced, with basic support for accessing (unprotected) physical memory. Per the discussion above, this ticket is kept open and renamed to "Standardizing hardware access".

[vitiral]

just my 2c

I would have the machine module do the general use cases (sleep, suspend, hibernate) and have specialized sleeping done in modules like pyb or equivalent for the target platform. That way everyone has a standard way to do the standard things, but there is also room to expose more specialized features on different platforms.

As for code like this:

pyb.RTC().wakeup(10000) # trigger a wakeup every 10s
machine.suspend() # suspend, waking up after 10s

Things like a RTC wakeups should be implemented in machine or maybe time. Pin wakeup should be implemented in whatever module interfaces with pins (currently pyb), and other wakeups should be implemented in their native modules (serial wakeup should be done in serial module, network/wifi in their respective modules, etc).

[pfalcon]

Done/further continued in https://github.com/micropython/micropython/issues/1430