mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) to make it easier to build and benchmark AI systems across diverse models, data sets, software and hardware
http://docs.mlcommons.org/cm4mlops/
Apache License 2.0
10 stars 17 forks source link

[CK2/CM] Adding test cases for individual scripts #241

Open arjunsuresh opened 2 years ago

arjunsuresh commented 2 years ago

As discussed during the conf-calls, we should ass test.sh files inside individual script folders which can test the basic script functionality. Need to think how to handle variations and versions here.

gfursin commented 2 years ago

Brain dump just for development purposes from our meeting in London (@gfursin, @arjunsuresh):

TESTS:
 ??? VIRTUAL CM NOT TO MIX UP USER 

 cm test script 

  ENV TO AVOID WRITING TO CACHE 

 cm test script --tags=image-corner-detection --min --sandbox
   (GCC, LLVM)

    _test.yaml | _test.json

      matrix_min:
       CM_VERSION=3.10

      matrix:
       CM_PYTHON_VERSION: 3.9,3.10
       CM_TEST_COMPILER: gcc,llvm
       CM_TEST_VARIATIONS: _openmp, _cuda
      cmd:
        --tags={SELF_TAG},${CM_TEST_VARIATIONS} --add_deps.compiler.tags=${CM_TEST_COMPILER} --quiet
        RECORD JSON output of run
      customize.py (postprocess_test)
       ENV
       diff files ..

     "test_cmd":"--tags=.. --add_deps.compiler.tags=,=gcc,llvm"

   cm run script --tags=.. --add_deps.compiler.tags=,=gcc,llvm
   tmp.sh 

 cm test script 

  [1/108] get-os 
    clean test.og
      _test.json/yaml
       MATRIX:
        test.sh (ENV)
     ENV: 
      CM_TEST_SCRIPT_TAGS
      CM_TEST_SCRIPT_UID
    run test.sh / test.bat 
      cm run script --tags=$CM_TEST_SCRIPT_TAGS $CM_TEST_SCRIPT_FLAGS --quiet > test.log
      diff original new 
     NO interaction (test.log)
     OK or ERROR (test.log)