Generate requirements.txt from enabled components in environment

[PhilippvK]

With an increasing number of components (frameworks, backends, frontends, platforms, targets, features,…) in MLonMCU the list of packages in the requirements.txt file is growing…

Especially since a dependency to the tensorflow package was added, installing the Python dependencies for the first time (I.e. in CI) does often take an unreasonable amount of time while also consuming more than one GB of disk space.

We might be at a point now where the “user” should decide which features we wants to use and which not. We already have those information in an environments environment.yml and would just need a script which maintains a mapping for the required packages for each component and writes all those which are needed to a requirements.txt file inside the environment directory.

TVM does a similar thing we you probably use as inspiration:

https://github.com/apache/tvm/blob/main/python/gen_requirements.py

[zhouxinyu0723]

check the branch less_pip_install_issue51.

[zhouxinyu0723]

The command is "mlonmcu setup -g".

[PhilippvK]

First some feedback, which is not related to your implementation:

It seems like, a lot of packages are required even if ther are not used just because they are included in some mlonmcu source files such i.e.:

• tflite
• flatbuffers
• matplotlib

I will think about if we can do anything about this.

[PhilippvK]

In

https://github.com/tum-ei-eda/mlonmcu/blob/main/mlonmcu/cli/init.py#L64-L69

I initially wanted to create a MLonMCU-managed virtual environment for the user (optional), however I never bothered implementing it. With this new feature we could take care of this automatically by some degree.

[PhilippvK]

@zhouxinyu0723

The following does not work as expected when installed into an python environment (e.g. using make install):

https://github.com/tum-ei-eda/mlonmcu/blob/eeb161607876b5d0c5301205ae1f63b31cccca5f/mlonmcu/setup/setup.py#L170

Could you write the requirements_addition.txt to the environment directory instead?

[PhilippvK]

@zhouxinyu0723 Is there a way where we do not have to hardcode the following lines?

https://github.com/tum-ei-eda/mlonmcu/blob/less_pip_install_issue51/mlonmcu/setup/setup.py#L191-L239

Maybe something like this could be used:

if config.name in requirements:
    ...

[PhilippvK]

After adressing these comments, feel free to open a PR

[zhouxinyu0723]

@zhouxinyu0723

The following does not work as expected when installed into an python environment (e.g. using make install):

https://github.com/tum-ei-eda/mlonmcu/blob/eeb161607876b5d0c5301205ae1f63b31cccca5f/mlonmcu/setup/setup.py#L170

Could you write the requirements_addition.txt to the environment directory instead?

will check it.

[zhouxinyu0723]

@zhouxinyu0723 Is there a way where we do not have to hardcode the following lines?

https://github.com/tum-ei-eda/mlonmcu/blob/less_pip_install_issue51/mlonmcu/setup/setup.py#L191-L239

Maybe something like this could be used:

if config.name in requirements:
    ...

I doubt whether it is necessary to do it without hardcode. The problem is that the config structures are nested.

For example how to check whether autotuning is activated?

It is also possible that certain feature names can be shared by two different modules. So I think it will be safe to check both the module name and feature name for requirement groups like "tvm-autotuning" "tflite-visualization".

Then this style of naming conventions like "tvm-autotuning" need to be distinguished from "tvm". Which means branching is needed.

[PhilippvK]

@zhouxinyu0723 Is there a way where we do not have to hardcode the following lines? https://github.com/tum-ei-eda/mlonmcu/blob/less_pip_install_issue51/mlonmcu/setup/setup.py#L191-L239 Maybe something like this could be used:

if config.name in requirements:
    ...

I can hardly think of a method to do this without hardcode because the config structures are nested.

For example how to check whether autotuning is activated?

It is also possible that certain feature names can be shared by two different modules. So I think it will be safe to check both the module name and feature name for requirements group like "tvm-autotuning" "tflite-visualization"

Then let’s keep it hardcoded as-is until we have resolved #33 which removes the nested structure of the config.