`utimeq` is not available by default in the standard library

[stroobandt]

Two other users plus myself are reporting this in detail on the forum.

utimeq is required by uasyncio, a very useful module for creating event loops, available in the «standard» MicroPython library.

[rkeiii]

So just for fun I took and copied micropython/extmod/utimeq.c and tossed it into pycom-micropython-sigfox/esp32/modutimeq.c and added it to the Makefile configuration. When it tries to build I hit this error first...

CC mods/modutimeq.c
mods/modutimeq.c:191:33: error: unknown type name 'mp_unary_op_t'
 STATIC mp_obj_t utimeq_unary_op(mp_unary_op_t op, mp_obj_t self_in) {
                                 ^
mods/modutimeq.c:215:17: error: 'utimeq_unary_op' undeclared here (not in a function)
     .unary_op = utimeq_unary_op,
                 ^
../py/mkrules.mk:47: recipe for target 'build/WIPY/release/mods/modutimeq.o' failed
make: *** [build/WIPY/release/mods/modutimeq.o] Error 1
rkeiii@vulcan:~/wipy2/pycom-micropython-sigfox/esp32$ 

At first glance this looked like something that originates from micropython/py/obj.c|.h but I'm far from certain. I diffed the v1.8.7 tagged version of that file from Micropython with Pycom's version and they're almost identical. However by v1.9.x they are far from identical. I might spend some more time playing with this but I'm not sure I really need asyncio at the moment.

[robert-hh]

mp_unary_op_t is defined in py/runtime0.h. Yo if you include that into py/obj.h, like it is done in the micropython.org port, at least that error should go away.

[robert-hh]

I've placed modtimeq.c into extmod/modtimeq.c. To get it visible, you have to: a) add the following line to MPCONFIGBOARD.H:

#define MICROPY_PY_UTIMEQ                           (1)

b) add the following code to py/objmodule.c

#if MICROPY_PY_UTIMEQ
    { MP_ROM_QSTR(MP_QSTR_utimeq), MP_ROM_PTR(&mp_module_utimeq) },
#endif

c) add the following line to py/builtin.h

extern const mp_obj_module_t mp_module_utimeq;

d) add this line to py/py.mk, in the extmod section

    ../extmod/modutimeq.o \

e) add the following line to py/obj.h:

#include "py/runtime0.h"

f) to make it clean, add the following lines to py/mpconfig.h

#ifndef MICROPY_PY_UTIMEQ
#define MICROPY_PY_UTIMEQ (0)
#endif

I placed that all close to the definitions of uheapq, which is somehow related. After that, you can import utimeq. Whether it works, has to be tested.

[robert-hh]

OK. Some progress, after stumbling over two small hiccups: utime.ticks_add() did not exists. Can be replaced for testing by: time + delta utime.ticks_diff() returns the other values than the MP port. Can be replaced for testing by: t - tnow After fixing that, simple examples based on uasyncio.core worked! I did not try others.

[stroobandt]

That is amazing! Thank you for your continued efforts. I am currently working on something else, but will return my attention to this as soon I fully grasp what has been done.

[rkeiii]

@robert-hh Thanks much for showing us how to build utimeq! I've confirmed I am able to build timeq using your instructions. I also tried loading up uasyncio and using it. I hit this issue trying some example code.

Traceback (most recent call last):
  File "main.py", line 69, in <module>
  File "uasyncio/core.py", line 149, in run_until_complete
  File "uasyncio/core.py", line 142, in run_forever
  File "uasyncio/core.py", line 46, in call_later_ms
AttributeError: 'module' object has no attribute 'ticks_add'

Test code

import uasyncio.core as asyncio
import datetime
from machine import RTC

async def display_date(loop):
    end_time = loop.time() + 5.0
    while True:
        print(RTC().now())
        if (loop.time() + 1.0) >= end_time:
            break
        await asyncio.sleep(1)

loop = asyncio.get_event_loop()
# Blocking call which returns when the display_date() coroutine is done
loop.run_until_complete(display_date(loop))
loop.close()

I'll try to take a look at the implementation of time and see about adding the ticks_add function.

[robert-hh]

That's what I planned to tell above, and now have corrected. ticks_add() does not exist. You may for test replace ticks_add(a, b) by a+b ticks_diff() in the pycom esp32 port has a different semantic than in the MP port. You may replace ticks_diff(a,b) by a-b Both fixes are not clean, since they do not respect overflow, but will do for tests lasting not longer than 2**30 ms after reset (298 hours). For the implementation of utime.ticks_add, you have to look at extmod/modutime.c. I made a simple version:

STATIC mp_obj_t time_ticks_add(mp_obj_t ticks_in, mp_obj_t delta_in) {
    uint32_t ticks = mp_obj_get_int(ticks_in);
    uint32_t delta = mp_obj_get_int(delta_in);
    return mp_obj_new_int_from_uint((ticks + delta));
}
STATIC MP_DEFINE_CONST_FUN_OBJ_2(time_ticks_add_obj, time_ticks_add);

and in the list of methods, add

    { MP_OBJ_NEW_QSTR(MP_QSTR_ticks_add),           (mp_obj_t)&time_ticks_add_obj },

[rkeiii]

I completely missed that you already addressed ticks_add not existing. I'll try to build it again tomorrow with your fix and rerun my test.

[robert-hh]

I made a PR to fix the utime issues with the OverflowError and missing ticks_add(). Still with this PR the order of parameters of ticks_diff() in uasyncio/core.py has to be reversed.

[robert-hh]

I tested just with uasyncio.core. Importing uasyncio.py resilts in two errors: a) uselect not found. But select exists. If needed, an alias for uselect can easily be installed, by adding the line:

    { MP_OBJ_NEW_QSTR(MP_QSTR_uselect),         (mp_obj_t)&mp_module_uselect },   \

to esp32/mpconfigport.h. b) ipoll is not implemented. That is not easily added, since the way iterators are implemented in pycom's port is somewhat different to the MP.org port. But instead, in uasyncio simply poll can be used, at a small memory penalty.

[robert-hh]

Made a PR #116 for these changes.

[rkeiii]

Well I have to say, robert-hh is the man. I tested his changes and I'm able to get various parts (uasyncio,core, uasyncio.synchro and uasyncio.queues) of uasyncio working on my WiPy 2.0 running 1.11.0b1.

Regarding the part about replacing ipoll with poll. I read in the MP uasyncio source/uselect docs that they were using ipoll for the oneshot behaviour and that this was equivalent to calling poll.modify(obj, 0) on socks/streams/whatever who have received events. I made the following tiny change to the wait function in https://github.com/micropython/micropython-lib/blob/master/uasyncio/uasyncio/__init__.py to try and reproduce the one-shot behaviour with poll. Did I call self.poller.modify(sock, 0) in a sane place? Sorry I don't have a great test case (in fact I have no sockets or other stream-type-things) for this at the moment in my codebase.

    def wait(self, delay):
        if DEBUG and __debug__:
            log.debug("poll.wait(%d)", delay)

        # change to work with pycom-sigfox-micropython        
        # since we don't have ipoll's one-shot behaviour
        # we have to reset the event mask on any sock
        # that receives an event
        res = self.poller.poll(delay)

        #log.debug("poll result: %s", res)
        # Remove "if res" workaround after
        # https://github.com/micropython/micropython/issues/2716 fixed.
        if res:
            for sock, ev in res:

                # as mentioned above we reset the event mask 
                # manually on each sock that's received an
                # event, i guess this is the right place to do this?
                self.poller.modify(sock, 0)

                cb = self.objmap[id(sock)]
                if ev & (select.POLLHUP | select.POLLERR):
                    # These events are returned even if not requested, and
                    # are sticky, i.e. will be returned again and again.
                    # If the caller doesn't do proper error handling and
                    # unregister this sock, we'll busy-loop on it, so we
                    # as well can unregister it now "just in case".
                    self.remove_reader(sock)
                if DEBUG and __debug__:
                    log.debug("Calling IO callback: %r", cb)
                if isinstance(cb, tuple):
                    cb[0](*cb[1])
                else:
                    cb.pend_throw(None)
                    self.call_soon(cb)

[robert-hh]

Edit upfront: I would say, Paul and Peter are the men, where Paul and Peter implemented the code and Peter wrote this fantastic tutorial.

That's not clear to me. The respective C-code in the Microypthon port reads like:

            if (self->flags & FLAG_ONESHOT) {
                // Don't poll next time, until new event flags will be set explicitly
                poll_obj->flags = 0;
            }

where flag is, as I read it, the event mask, and FLAG_ONESHOT has the value of 1. But the same piece of code is in the Pycom section (moduselect.c, line 271), so it should not make a difference, and the change you made is not required. Alas, you just have to give it a try.

[robert-hh]

utimeq was added today with version 1.15.0.b1.

[husigeza]

Hello, Uasyncio support added in development release 1.19.0.b4. utimeq is part of the firmware since the mentioned version, 1.15.0.b1 thus closing this issue.