search

mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) to make it easier to build and benchmark AI systems across diverse models, data sets, software and hardware
http://docs.mlcommons.org/cm4mlops/
Apache License 2.0
15 stars 24 forks source link

Improving cm4mlops PYPI package #149

Open gfursin opened 4 months ago

gfursin commented 4 months ago

After testing the latest cm4mlops PYPI package, I have a few suggestions:

This package hides extra installation of various system dependencies and CM repositories and it is impossible to customize it (foe example, changing CM4MLOps repository or specifying different branches) or debug issues if something goes wrong inside (PYPI projects do not provide a user-friendly way to report issues inside since they are supposed to work in an autonomous way.

While looking at other projects, I think that a better way is to use the original cmind package with min deps and have a function to bootstrap/init cm4mlops, cm4mlperf and any other repository with a standard way to customize and debug it.

For example:


pip install cmind
cm init cm4mlperf
cm init cm4mlops --branch=mlperf-inference
...

That will perform the same functions as cm4mlops package but will be understandable to users and will provide a standard and easy way to debug it and trace errors. It will be also easier to use it GitHub actions and other CI frameworks.

Let's discuss it during the next CM dev meeting.
arjunsuresh commented 4 months ago

Hi @gfursin the main use of pip install cm4mlops is to give a one liner for users to use it in further projects like cm4mlops, cm4btf etc. It is not meant for those working on developing cm4mlops or its forks.

gfursin commented 4 months ago

Sure, I am not speaking about developments. But one liner is not always good. In cm4mlops, cm4abtf, cm4mlperf cases, we still need to check system sub-dependencies (platform dependent and project dependent that are not supposed to be done inside pip install) and extra Git repositories or stable ZIP/TAR downloads with customization that is not possible. PIP is not handling such errors and cases properly. That's why usually you install a main package and then provide init function that can handle all the situations and errors properly outside PIP install. We can discuss it later ...