Cross-compilation for MicroPython 1.12

[poesel]

I've bumped mpy-cross to 1.12 because vanilla MicroPython is at 1.12 now, which is what I am using.

[poesel]

Hmm, two problems: Make is missing the definition for MPY_VERSION and MPY_TARGET I can set:

MPY_VERSION=1.12 
MPY_TARGET=bytecode

but where is the 'right' place for that? And is 'bytecode' the correct option for an ESP32?

[amotl]

Thanks!

The designated command for the most recent Pycom firmware is

make recycle-ng MPY_CROSS=true MPY_TARGET=pycom MPY_VERSION=1.11

Please keep the mpy-cross 1.11 version as we need it for bytecode-compilation for the Pycom firmware.

According to https://github.com/hiveeyes/micropython/commit/762ab8ac78c96e71aea1f1d647448d4b3180136d, using MPY_TARGET=bytecode should be the appropriate option here, right. See also [1].

make pyboard-recycle MPY_CROSS=true MPY_TARGET=bytecode MPY_VERSION=1.11

[1] https://community.hiveeyes.org/t/running-the-terkin-datalogger-on-stm32/2576/2

[amotl]

So, the appropriate command for vanilla MicroPython could be

export MCU_PORT=/dev/ttyS3
make install-ng MPY_CROSS=true MPY_TARGET=bytecode MPY_VERSION=1.12

[poesel]

I have a problem importing the compiled mpy files. It seems uPy ignores them, I have deliberately taken away the /terkin sys.path to force uPy to use the mpy files in /lib... This was the result:

[boot.py] INFO: Python module search path is: ['', '/lib'] [boot.py] INFO: Universal MicroPython Application Loader (umal) [umal] INFO: Python module search path is: ['/lib-mpy-112-bytecode', '', '/lib'] [main.py] INFO: Loading settings [main.py] INFO: Starting logging Traceback (most recent call last): File "main.py", line 21, in File "logging.py", line 7, in ImportError: no module named 'logging' MicroPython v1.12 on 2019-12-20; ESP32 module (spiram) with ESP32 Type "help()" for more information.

Another problem: the pycom module in VSC cannot handle dots ('.') in directory names. So I had to rename 'lib-mpy-1.12-bytecode' to 'lib-mpy-112-bytecode'.

However I can import mpy files that are in the top directory. Wasn't there an issue with mpy files and subdirectories?

[amotl]

Hmmm, I believe it worked for me when trying this on a Pyboard. It looks like we have to investigate further.

[poesel]

I checked if it is possible to import .mpy's from sub- and subsub directories. It works. No bug here. Still I don't know why it fails to import.

Other questions: Why is 'dist-packages' only compiled for target 'pycom'? Shouldn't there be the option '-march=xtensa' for the pycom & ESP boards?

[amotl]

I checked if it is possible to import .mpy's from sub- and subsub directories. It works. No bug here.

Good to hear it works.

Still I don't know why it fails to import.

So importing works but then it fails?

Why is 'dist-packages' only compiled for target 'pycom'?

Is it? Maybe some wiring is wrong within the Makefile?

Shouldn't there be the option '-march=xtensa' for the pycom & ESP boards?

According to bin/mpy-cross/1.11/mpy_cross/mpy-cross --help, the -march option is only used when emitting native code. Until we go into that direction, we are only emitting bytecode for now.

-march=<arch> : set architecture for native emitter; x86, x64, armv6, armv7m, xtensa

[amotl]

With 7f2111b, dist-packages will be compiled unconditionally and mpy-cross==1.12 will be installed side-by-side with version 1.11.

So, this command populates the lib-mpy-1.12-bytecode folder successfully with all of the bytecode.

make mpy-compile MPY_CROSS=true MPY_TARGET=bytecode MPY_VERSION=1.12

[poesel]

Is it? Maybe some wiring is wrong within the Makefile?

Thats the Makefile:

    @if test "${MPY_TARGET}" = "pycom"; then \
            $(MAKE) mpy-cross what="--out $(mpy_path) dist-packages"; \
    fi
    @$(MAKE) mpy-cross what="--out $(mpy_path) lib"
    @$(MAKE) mpy-cross what="--out $(mpy_path)/terkin terkin"

And that was the cause of my import problem. I tried to force reading the mpy files and removed /terkin and /dist-packages from sys.path. Since the packages weren't compiled they couldn't be imported... d'oh!

I've changed the Makefile and removed the 'if pycom...'. mpy-compile will now populate the /lib-... folder with the content of /dist-packages. Thats a bit ugly but seems to work. Shaved of 6 seconds of boot time (from 22 to 16 until the first line of the logger comes up). Honestly I had hoped for more. Looks like it is necessary to freeze.

[thiasB]

Also, tools/upload-mpy.rshell still hard-wires thelib-mpy-1.11-pycom folder and needs dynamic version handling or version specific .rshell files, respectively.

[amotl]

tools/upload-mpy.rshell still hard-wires the lib-mpy-1.11-pycom folder.

So true, thanks for mentioning! Are you also working with vanilla MicroPython, Matthias? @poesel decided to opt-in for compile-only and will transfer the files using VSCode.

[amotl]

Dear @poesel,

while I like the config.mk approach for having a shortcut in there, will it be possible to still pass arguments on the command line?

While the main target is still Pycom,

make mpy-compile MPY_CROSS=true MPY_TARGET=pycom MPY_VERSION=1.11

I will be happy to support all platforms equally.

I am just thinking about the appropriate implementation here which doesn't add bias into one or another direction. A while ago, I was planning to rewrite the whole build driver using Python which would give us way more flexibility on many levels. Right know, we are probably stuck with the Makefile-based approach and all I want to achieve here is at least some convenience/elegance without disrupting the existing approach too much.

Thanks for listening and with kind regards, Andreas.

[poesel]

while I like the config.mk approach for having a shortcut in there, will it be possible to still pass arguments on the command line?

Since the include is in the Makefile I guess that anything in config.mk would overwrite parameters given on the command line.

But IIRC there is this ?= assignment syntax which only sets a variable if it is not set. Maybe that's the way to go?

[amotl]

But IIRC there is this ?= assignment syntax which only sets a variable if it is not set. Maybe that's the way to go?

That would be awesome. Will you be able to test this detail if you find some minutes and eventually update the pull request afterwards?

[poesel]

That should be it if I didn't mess up with Git...

[amotl]

Can I ask you to add https://github.com/poesel/terkin-datalogger/pull/1 by @thiasB beforehand?

[poesel]

You have to be more explicit wrt Git with me :) I was wondering to what I should be added to :D

[amotl]

Dear @poesel and @thiasB,

I have deliberately squashed these updates and integrated them through dbfa32d6. Thanks a bunch for your efforts!

With kind regards, Andreas.