Improving CM script automation and CM scripts

[gfursin]

Aggregating tasks to improve CM script automation and CM scripts based on user feedback (this ticket is being gradually updated and tasks resolved based on our bandwidth and engineering resources):

Automation

• [x] Add --silent / -s mode to avoiding printing CM workflow execution
• [x] Add cm_min_version check for automation and scripts
• [x] Check current dumping of all versions of all dependencies Commit
• [x] Check readme generation with all deps Commit
  • [x] Add --version when available Commit
• [x] Prepare tmp_run_final_script.sh and stop to let user run it manually without CM if needed
• [x] Generate sample Docker container during --repro
• [x] Add cfg to load default env, state and keys for all CM scripts
  • [x] cm set cfg default --key.cm-script.silent --key.cm-script.env.CM_SUDO="no" ...
  • [x] Add silent mode as default
  • [x] Support add/update cfg keys ...
• [x] Add cm status to show version, paths to current repositories and if the version is up-to-date (useful for virtual environments)
• [x] Add to meta where to report errors if not default repository (for MLPerf inference -> report https://github.com/mlcommons/inference)
• [x] Add proper version detection for cuDNN

Documentation

• [ ] Check individual documentation and input description for all main CM scripts
• [ ] Add individual tests for all main CM scripts

Tutorials

Need to explain how to extend CM scripts, add new ones, unify inputs, etc)

• [ ] Prepare tutorial about CM basics and CM scripts
• [ ] Prepare tutorial about CM automation for basic inference
• [ ] Prepare tutorial about CM automation for MLPerf loadgen
• [ ] Prepare tutorial about CM automation for MLPerf inference
• [ ] Prepare tutorial about CM automation for ABTF
• [ ] Prepare tutorial about CM automation for SCC'24

Longer term

• [ ] Use logging instead of print
• [ ] Refactor and simplify CM script automation - it was heavily prototyped and it is stable but the implementation can be considerably improved and simplified
• [ ] Refactor docker container generation and execution - the prototype implementation is very complex and can be dramatically simplified