Reproducing the results for tiny v1.0 by OctoML, CM Error: Portable CM script failed (name = get-microtvm, return code = 256)

[sahelib25]

On running cm run script --tags=generate,tiny,mlperf,octoml,submission

...
         * Searching for collective script(s) with tags "detect,os"
          - Number of scripts found: 1
          - Found script::detect-os,863735b7db8c44fc in /home/saheli/CM/repos/mlcommons@ck/cm-mlops/script/detect-os
          - Running preprocess ...

          - Running native script "/home/saheli/CM/repos/mlcommons@ck/cm-mlops/script/detect-os/run.sh" from temporal script "tmp-run.sh" in "/home/saheli/CM/repos/local/cache/7d3ffc2a074d4c76" ...

          - Running postprocess ...
          - running time of script "detect-os,detect,os,info": 0.21 sec.
      - Processing env after dependencies ...
        # potential PIP version string (if needed): ==main
      - Running preprocess ...

      - Running native script "/home/saheli/CM/repos/mlcommons@ck/cm-mlops/script/get-microtvm/run.sh" from temporal script "tmp-run.sh" in "/home/saheli/CM/repos/local/cache/7d3ffc2a074d4c76" ...

******************************************************
Cloning microtvm from https://github.com/mlcommons/submissions_tiny_v1.0 with branch main  ...
Cloning into 'microtvm'...
Username for 'https://github.com': sahelib25
Password for 'https://sahelib25@github.com':
remote: Repository not found.
fatal: repository 'https://github.com/mlcommons/submissions_tiny_v1.0/' not found

CM error: Portable CM script failed (name = get-microtvm, return code = 256)

Note that it is often a portability problem of the third-party tool or native script that is wrapped and unified by this CM script.
The CM concept is to collaboratively fix such issues inside portable CM scripts to make existing tools and native script more portable, interoperable, deterministic and reproducible.

Please help the community by reporting the full log with the command line here:
* https://github.com/mlcommons/ck/issues
* https://bit.ly/mlperf-edu-wg

Thank you!

[arjunsuresh]

Thank you @sahelib25 for raising this issue. Actually that repository was used only until the submission time and we had forgot to update it once this new repository is created. This change should fix this issue.

[sahelib25]

Thank you @arjunsuresh. I have made the changes accordingly and re run it. Now it is showing the following error.

CMake Warning at /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/cmake/generic_toolchain.cmake:17 (message):
  The environment variable 'LDFLAGS' was set to -O2 ,

  but Zephyr ignores flags from the environment.  Use 'cmake
  -DEXTRA_LDFLAGS=-O2 ' instead.
Call Stack (most recent call first):
  /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/cmake/app/boilerplate.cmake:540 (include)
  /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:102 (include_boilerplate)
  CMakeLists.txt:28 (find_package)

-- Found toolchain: zephyr 0.13.2 (/home/saheli/zephyr-sdk)
-- Found BOARD.dts: /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/boards/arm/nucleo_l4r5zi/nucleo_l4r5zi.dts
-- Found devicetree overlay: /home/saheli/CM/repos/local/cache/7d3ffc2a074d4c76/microtvm/closed/OctoML/code/microtvm_cmsis_nn/NUCLEO_L4R5ZI/ad/nucleo_l4r5zi.overlay
In file included from <command-line>:
/home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/boards/arm/nucleo_l4r5zi/nucleo_l4r5zi.dts:9:10: fatal error: st/l4/stm32l4r5z(g-i)tx-pinctrl.dtsi: No such file or directory
    9 | #include <st/l4/stm32l4r5z(g-i)tx-pinctrl.dtsi>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMake Error at /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/cmake/dts.cmake:167 (message):
  command failed with return code: 1
Call Stack (most recent call first):
  /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/cmake/app/boilerplate.cmake:541 (include)
  /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:24 (include)
  /home/saheli/CM/repos/local/cache/c078d20858374e82/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:102 (include_boilerplate)
  CMakeLists.txt:28 (find_package)

-- Configuring incomplete, errors occurred!

CM error: Portable CM script failed (name = reproduce-mlperf-octoml-tinyml-results, return code = 256)

Note that it is often a portability problem of the third-party tool or native script that is wrapped and unified by this CM script.
The CM concept is to collaboratively fix such issues inside portable CM scripts to make existing tools and native script more portable, interoperable, deterministic and reproducible.

Please help the community by reporting the full log with the command line here:
* https://github.com/mlcommons/ck/issues
* https://bit.ly/mlperf-edu-wg

Thank you!

[arjunsuresh]

Hi @sahelib25 it seems the code we had submitted is not compatible with the latest cmsis_5 GIT repository. I have no updated the cmsis_5 repo to point to a previous working repository. Please do cm pull repo mlcommons@ck && cm rm cache --tags=get,cmsis -f

[gfursin]

By the way, @sahelib25, we have just created a Discord "Reproducibility" channel to help the community reproduce ML Systems experiments. Feel free to join here - it may be easier to have an interactive discussion there...

We will use this Discord to discuss with the community how to provide a common CM interface to access and reproduce various public research projects and benchmarks ...