Flash example hangs when system frequency is increased to 800 MHz

andresovela commented 4 months ago

I'm not sure if this is necessarily an issue appropriate for this repo, but since it affects one of the examples I'll open it here. I also brought this up in the community forum.

In the interest of running models as fast as possible, I increased the system frequency to 800 MHz. The app_flash_single_model seems to hang after the prepare stage.

These are the steps to reproduce:

Copy /path/to/xtc-tools/targets/XK-EVK-XU316/XK-EVK-XU316.xn to examples/app_flash_single_model/src/XK-EVK-XU316-custom.xn
Change SystemFrequency="600MHz" to SystemFrequency="800MHz" in line 15
Change the TARGET in the Makefile from XCORE-AI-EXPLORER to XK-EVK-XU316-custom.xn
(Optional) Add APP_FLAGS += -DTFLMC_XCORE_PROFILE to see where it gets stuck
Follow steps to compile and run as described in the README, but add --target-file src/XK-EVK-XU316-custom.xn to the xflash command

The program seems to hang after printing this:

Cumulative times for prepare()...
OP_XC_ld_flash                   296          0.00ms
OP_XC_conv2d_v2                  78           0.00ms
OP_UNPACK                        0            0.00ms
OP_SPLIT                         104          0.00ms
OP_XC_add                        126          0.00ms
OP_XC_lookup                     2276         0.02ms
OP_XC_mul                        66           0.00ms
OP_RESHAPE                       558          0.01ms

Total time for prepare() - 3504       0.04ms

I can see the STATUS LED near the flash on the EVK flashing rapidly.

I wasn't able to get more information using xgdb.

andresovela commented 4 months ago

Note that app_no_flash runs without issues, and I can see the expected speed up when the system frequency is set to 800 MHz

panickal-xmos commented 4 months ago

Thank you @andresovela. Fixed by https://github.com/xmos/lib_tflite_micro/pull/165 and will be merged in by https://github.com/xmos/ai_tools/pull/919

andresovela commented 4 months ago

Hi @panickal-xmos, I tried to test this but I got unexpected results.

When I tested the app_flash_single_model example, it doesn't hang anymore, but I don't see any speed up either.

I tried to test an app of my own, but it still hangs (this is likely due to my own mistake somewhere). Not sure what do I need to update to get these changes.

I pulled develop and ran git submodule update --init --recursive in ai_tools. Is there anything else I need to do?

panickal-xmos commented 4 months ago

If you're building the source yourself, will have to build the compiler and libs, and then build a wheel, and then install the wheel into your env. Could you please try the prebuilt beta version from PyPI, pip install xmos-ai-tools==1.2.1.dev160 and try the app?

andresovela commented 4 months ago

Build a wheel? What does that mean?

andresovela commented 4 months ago

May I ask why is it necessary to rebuild the compiler just to include a change in a lib?

panickal-xmos commented 4 months ago

Good point. For this fix, the compiler doesn't have to be rebuilt. Wheel is a Python package format, https://peps.python.org/pep-0427/ . For building from source and installing it, follow these steps, https://github.com/xmos/ai_tools/blob/develop/docs/rst/build-from-source.rst. The last step builds a wheel and installs it.

andresovela commented 4 months ago

I was able to run the example with 800 MHz and it looks good to me. I'm closing this issue :) Thanks!

xmos / ai_tools

Flash example hangs when system frequency is increased to 800 MHz #917