CM error: Windows is not supported in this script yet!

mwangistan commented 1 year ago

Trying to run cm run script --tags=app,vision,language,mlcommons,mlperf,inference,generic --json=true on windows 11. I'm getting an error that the CM script isn't supported on windows yet. Not sure if I'm missing anything. Thanks

* Searching for collective script(s) with tags "app,vision,language,mlcommons,mlperf,inference,generic"
  - Number of scripts found: 1
  - Found script::app-mlperf-inference,d775cac873ee4231 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\app-mlperf-inference
    Prepared variations: _reference,_resnet50,_onnxruntime,_cpu,_test,_fp32
  - Checking dependencies on other CM scripts:

    * Searching for collective script(s) with tags "detect,os"
      - Number of scripts found: 1
      - Found script::detect-os,863735b7db8c44fc in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os
      - Running preprocess ...

      - Running native script "C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os\run.bat" from temporal script "tmp-run.bat" in "C:\Users\<>\source\repos\mlperf" ...

      - Running postprocess ...

      * Searching for collective script(s) with tags "get,sys-utils-min"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
          - Number of cached script outputs found: 1
        - Found script::get-sys-utils-min,a9af7714d3d94779 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-min
        - Checking if script execution is already cached ...
          - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\f986f803f495404b
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,sys-utils-min": 1.01 sec.
      - running time of script "detect-os,detect,os,info": 1.81 sec.

    * Searching for collective script(s) with tags "get,sys-utils-cm"
      - Number of scripts found: 1
      - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-cm
        - Number of cached script outputs found: 1
      - Found script::get-sys-utils-cm,bc90993277e84b8e in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-cm
      - Checking if script execution is already cached ...
        - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-cm
        - Found cached script output: C:\Users\<>\CM\repos\local\cache\36b113d54cce43ed
      - Checking dynamic dependencies on other CM scripts:
      - Processing env after dependencies ...
        - Checking prehook dependencies on other CM scripts:
          - Loading state from cached entry ...
        - Checking posthook dependencies on other CM scripts:
        - Checking post dependencies on other CM scripts:
      - running time of script "get,sys-utils-cm": 0.60 sec.

    * Searching for collective script(s) with tags "get,python"
      - Number of scripts found: 1
      - Searching for cached script outputs with the following tags: -tmp,get,python
        - Number of cached script outputs found: 1
      - Found script::get-python3,d0b5dd74373f4a62 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-python3
      - Checking if script execution is already cached ...
        - Searching for cached script outputs with the following tags: -tmp,get,python,python3,get-python,get-python3
        - Found cached script output: C:\Users\<>\CM\repos\local\cache\3b1fe588b7944001
        - Checking prehook dependencies on other CM scripts:
          - Loading state from cached entry ...
        - Checking posthook dependencies on other CM scripts:
        - Checking post dependencies on other CM scripts:
      - running time of script "get,python,python3,get-python,get-python3": 0.61 sec.

    * Searching for collective script(s) with tags "get,mlcommons,inference,src"
      - Number of scripts found: 1
      - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src
        - Number of cached script outputs found: 2
      - Found script::get-mlperf-inference-src,4b57186581024797 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-mlperf-inference-src
        Prepared variations: _default,_short-history
      - Checking if script execution is already cached ...
        - Prepared variations: _default,_short-history
        - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src,source,inference-src,inference-source,mlperf,_default,_short-history
        - Found cached script output: C:\Users\<>\CM\repos\local\cache\4dbbd315cf154998
      - Checking dynamic dependencies on other CM scripts:
      - Processing env after dependencies ...
        - Checking prehook dependencies on other CM scripts:
          - Loading state from cached entry ...
        - Checking posthook dependencies on other CM scripts:
        - Checking post dependencies on other CM scripts:
      - running time of script "get,src,source,inference,inference-src,inference-source,mlperf,mlcommons": 0.74 sec.

    * Searching for collective script(s) with tags "get,dataset-aux,imagenet-aux"
      - Number of scripts found: 1
      - Searching for cached script outputs with the following tags: -tmp,get,dataset-aux,imagenet-aux
        - Number of cached script outputs found: 1
      - Found script::get-dataset-imagenet-aux,bb2c6dd8c8c64217 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-dataset-imagenet-aux
        Prepared variations: _from.dropbox,_2012
      - Checking if script execution is already cached ...
        - Prepared variations: _from.dropbox,_2012
        - Searching for cached script outputs with the following tags: -tmp,get,dataset-aux,imagenet-aux,aux,image-classification,_from.dropbox,_2012
        - Found cached script output: C:\Users\<>\CM\repos\local\cache\de7777a6be2d4f85
        - Checking prehook dependencies on other CM scripts:
          - Loading state from cached entry ...
        - Checking posthook dependencies on other CM scripts:
        - Checking post dependencies on other CM scripts:
      - running time of script "get,aux,dataset-aux,image-classification,imagenet-aux": 0.63 sec.
  - Processing env after dependencies ...
  - Running preprocess ...

  * Searching for collective script(s) with tags "app,mlperf,reference,inference" and variations "resnet50,cpu,onnxruntime,fp32"
    - Number of scripts found: 1
    - Found script::app-mlperf-inference-reference,ff149e9781fc4b65 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\app-mlperf-inference-reference
      Prepared variations: _resnet50,_cpu,_onnxruntime,_fp32,_python
    - Checking dependencies on other CM scripts:

      * Searching for collective script(s) with tags "detect,os"
        - Number of scripts found: 1
        - Found script::detect-os,863735b7db8c44fc in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os
        - Running preprocess ...

        - Running native script "C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os\run.bat" from temporal script "tmp-run.bat" in "C:\Users\<>\source\repos\mlperf" ...

        - Running postprocess ...

        * Searching for collective script(s) with tags "get,sys-utils-min"
          - Number of scripts found: 1
          - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
            - Number of cached script outputs found: 1
          - Found script::get-sys-utils-min,a9af7714d3d94779 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-min
          - Checking if script execution is already cached ...
            - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
            - Found cached script output: C:\Users\<>\CM\repos\local\cache\f986f803f495404b
            - Checking prehook dependencies on other CM scripts:
              - Loading state from cached entry ...
            - Checking posthook dependencies on other CM scripts:
            - Checking post dependencies on other CM scripts:
          - running time of script "get,sys-utils-min": 0.83 sec.
        - running time of script "detect-os,detect,os,info": 1.60 sec.

      * Searching for collective script(s) with tags "detect,cpu"
        - Number of scripts found: 1
        - Found script::detect-cpu,586c8a43320142f7 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-cpu
        - Checking dependencies on other CM scripts:

          * Searching for collective script(s) with tags "detect,os"
            - Number of scripts found: 1
            - Found script::detect-os,863735b7db8c44fc in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os
            - Running preprocess ...

            - Running native script "C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os\run.bat" from temporal script "tmp-run.bat" in "C:\Users\<>\source\repos\mlperf" ...

            - Running postprocess ...

            * Searching for collective script(s) with tags "get,sys-utils-min"
              - Number of scripts found: 1
              - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                - Number of cached script outputs found: 1
              - Found script::get-sys-utils-min,a9af7714d3d94779 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-min
              - Checking if script execution is already cached ...
                - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                - Found cached script output: C:\Users\<>\CM\repos\local\cache\f986f803f495404b
                - Checking prehook dependencies on other CM scripts:
                  - Loading state from cached entry ...
                - Checking posthook dependencies on other CM scripts:
                - Checking post dependencies on other CM scripts:
              - running time of script "get,sys-utils-min": 0.75 sec.
            - running time of script "detect-os,detect,os,info": 1.56 sec.
        - Processing env after dependencies ...
        - Running preprocess ...
        - Running postprocess ...
WARNING TBD: Add Windows support in the "detect-cpu" script ...
        - running time of script "detect,cpu,detect-cpu,info": 2.08 sec.

      * Searching for collective script(s) with tags "get,sys-utils-cm"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-cm
          - Number of cached script outputs found: 1
        - Found script::get-sys-utils-cm,bc90993277e84b8e in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-cm
        - Checking if script execution is already cached ...
          - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-cm
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\36b113d54cce43ed
        - Checking dynamic dependencies on other CM scripts:
        - Processing env after dependencies ...
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,sys-utils-cm": 0.63 sec.

      * Searching for collective script(s) with tags "get,python"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,python
          - Number of cached script outputs found: 1
        - Found script::get-python3,d0b5dd74373f4a62 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-python3
        - Checking if script execution is already cached ...
          - Searching for cached script outputs with the following tags: -tmp,get,python,python3,get-python,get-python3
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\3b1fe588b7944001
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,python,python3,get-python,get-python3": 0.75 sec.

      * Searching for collective script(s) with tags "get,generic-python-lib" and variations "onnxruntime"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,generic-python-lib,_onnxruntime
          - Number of cached script outputs found: 1
        - Found script::get-generic-python-lib,94b62a682bc44791 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-generic-python-lib
          Prepared variations: _onnxruntime
        - Checking if script execution is already cached ...
          - Prepared variations: _onnxruntime
          - Searching for cached script outputs with the following tags: -tmp,get,generic-python-lib,install,generic,_onnxruntime,deps-python-version-3.10.7,deps-python-non-virtual
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\a5145533ab254524
        - Checking dynamic dependencies on other CM scripts:
        - Processing env after dependencies ...
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,install,generic,generic-python-lib": 0.65 sec.

      * Searching for collective script(s) with tags "get,ml-model,image-classification,resnet50,raw" and variations "fp32,onnx"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,ml-model,image-classification,resnet50,raw,_fp32,_onnx
          - Number of cached script outputs found: 1
        - Found script::get-ml-model-resnet50,56203e4e998b4bc0 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-ml-model-resnet50
          Prepared variations: _fp32,_onnx,_opset-11,_argmax
        - Checking if script execution is already cached ...
          - Prepared variations: _fp32,_onnx,_opset-11,_argmax
          - Searching for cached script outputs with the following tags: -tmp,get,ml-model,image-classification,resnet50,raw,ml-model-resnet50,_fp32,_onnx,_opset-11,_argmax
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\69a0ff5d3f4d4e5a
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,raw,ml-model,resnet50,ml-model-resnet50,image-classification": 0.79 sec.

      * Searching for collective script(s) with tags "get,dataset,image-classification,imagenet,preprocessed" and variations "default,NCHW"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,dataset,image-classification,imagenet,preprocessed,_default,_NCHW
          - Number of cached script outputs found: 1
        - Found script::get-preprocessed-dataset-imagenet,f259d490bbaf45f5 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-preprocessed-dataset-imagenet
          Prepared variations: _default,_NCHW,_mlcommons-reference-preprocessor,_500,_resolution.224
        - Checking if script execution is already cached ...
          - Prepared variations: _default,_NCHW,_mlcommons-reference-preprocessor,_500,_resolution.224
          - Searching for cached script outputs with the following tags: -tmp,get,dataset,image-classification,imagenet,preprocessed,ILSVRC,_default,_NCHW,_mlcommons-reference-preprocessor,_500,_resolution.224
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\88541b6d038f4be5
        - Checking dynamic dependencies on other CM scripts:
        - Processing env after dependencies ...
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,dataset,imagenet,ILSVRC,image-classification,preprocessed": 0.69 sec.

      * Searching for collective script(s) with tags "get,dataset-aux,image-classification,imagenet-aux"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,dataset-aux,image-classification,imagenet-aux
          - Number of cached script outputs found: 1
        - Found script::get-dataset-imagenet-aux,bb2c6dd8c8c64217 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-dataset-imagenet-aux
          Prepared variations: _from.dropbox,_2012
        - Checking if script execution is already cached ...
          - Prepared variations: _from.dropbox,_2012
          - Searching for cached script outputs with the following tags: -tmp,get,dataset-aux,image-classification,imagenet-aux,aux,_from.dropbox,_2012
          - Found cached script output: C:\Users\<>\CM\repos\local\cache\de7777a6be2d4f85
          - Checking prehook dependencies on other CM scripts:
            - Loading state from cached entry ...
          - Checking posthook dependencies on other CM scripts:
          - Checking post dependencies on other CM scripts:
        - running time of script "get,aux,dataset-aux,image-classification,imagenet-aux": 0.79 sec.

      * Searching for collective script(s) with tags "generate,user-conf,mlperf,inference"
        - Number of scripts found: 1
        - Found script::generate-mlperf-inference-user-conf,3af4475745964b93 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\generate-mlperf-inference-user-conf
        - Checking dependencies on other CM scripts:

          * Searching for collective script(s) with tags "detect,os"
            - Number of scripts found: 1
            - Found script::detect-os,863735b7db8c44fc in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os
            - Running preprocess ...

            - Running native script "C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os\run.bat" from temporal script "tmp-run.bat" in "C:\Users\<>\source\repos\mlperf" ...

            - Running postprocess ...

            * Searching for collective script(s) with tags "get,sys-utils-min"
              - Number of scripts found: 1
              - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                - Number of cached script outputs found: 1
              - Found script::get-sys-utils-min,a9af7714d3d94779 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-min
              - Checking if script execution is already cached ...
                - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                - Found cached script output: C:\Users\<>\CM\repos\local\cache\f986f803f495404b
                - Checking prehook dependencies on other CM scripts:
                  - Loading state from cached entry ...
                - Checking posthook dependencies on other CM scripts:
                - Checking post dependencies on other CM scripts:
              - running time of script "get,sys-utils-min": 0.76 sec.
            - running time of script "detect-os,detect,os,info": 1.65 sec.

          * Searching for collective script(s) with tags "detect,cpu"
            - Number of scripts found: 1
            - Found script::detect-cpu,586c8a43320142f7 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-cpu
            - Checking dependencies on other CM scripts:

              * Searching for collective script(s) with tags "detect,os"
                - Number of scripts found: 1
                - Found script::detect-os,863735b7db8c44fc in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os
                - Running preprocess ...

                - Running native script "C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os\run.bat" from temporal script "tmp-run.bat" in "C:\Users\<>\source\repos\mlperf" ...

                - Running postprocess ...

                * Searching for collective script(s) with tags "get,sys-utils-min"
                  - Number of scripts found: 1
                  - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                    - Number of cached script outputs found: 1
                  - Found script::get-sys-utils-min,a9af7714d3d94779 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-min
                  - Checking if script execution is already cached ...
                    - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                    - Found cached script output: C:\Users\<>\CM\repos\local\cache\f986f803f495404b
                    - Checking prehook dependencies on other CM scripts:
                      - Loading state from cached entry ...
                    - Checking posthook dependencies on other CM scripts:
                    - Checking post dependencies on other CM scripts:
                  - running time of script "get,sys-utils-min": 0.88 sec.
                - running time of script "detect-os,detect,os,info": 1.78 sec.
            - Processing env after dependencies ...
            - Running preprocess ...
            - Running postprocess ...
WARNING TBD: Add Windows support in the "detect-cpu" script ...
            - running time of script "detect,cpu,detect-cpu,info": 2.41 sec.

          * Searching for collective script(s) with tags "get,python"
            - Number of scripts found: 1
            - Searching for cached script outputs with the following tags: -tmp,get,python
              - Number of cached script outputs found: 1
            - Found script::get-python3,d0b5dd74373f4a62 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-python3
            - Checking if script execution is already cached ...
              - Searching for cached script outputs with the following tags: -tmp,get,python,python3,get-python,get-python3
              - Found cached script output: C:\Users\<>\CM\repos\local\cache\3b1fe588b7944001
              - Checking prehook dependencies on other CM scripts:
                - Loading state from cached entry ...
              - Checking posthook dependencies on other CM scripts:
              - Checking post dependencies on other CM scripts:
            - running time of script "get,python,python3,get-python,get-python3": 0.83 sec.

          * Searching for collective script(s) with tags "get,mlcommons,inference,src"
            - Number of scripts found: 1
            - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src
              - Number of cached script outputs found: 2
            - Found script::get-mlperf-inference-src,4b57186581024797 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-mlperf-inference-src
              Prepared variations: _default,_short-history
            - Checking if script execution is already cached ...
              - Prepared variations: _default,_short-history
              - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src,source,inference-src,inference-source,mlperf,_default,_short-history
              - Found cached script output: C:\Users\<>\CM\repos\local\cache\4dbbd315cf154998
            - Checking dynamic dependencies on other CM scripts:
            - Processing env after dependencies ...
              - Checking prehook dependencies on other CM scripts:
                - Loading state from cached entry ...
              - Checking posthook dependencies on other CM scripts:
              - Checking post dependencies on other CM scripts:
            - running time of script "get,src,source,inference,inference-src,inference-source,mlperf,mlcommons": 0.89 sec.

          * Searching for collective script(s) with tags "get,sut,configs"
            - Number of scripts found: 1
            - Found script::get-mlperf-inference-sut-configs,c2fbf72009e2445b in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-mlperf-inference-sut-configs
            - Running postprocess ...
            - running time of script "get,mlperf,inference,sut,configs,sut-configs": 0.76 sec.
        - Processing env after dependencies ...
        - Running preprocess ...
Using MLCommons Inference source from 'C:\Users\<>\CM\repos\local\cache\4dbbd315cf154998\inference'
Output Dir: 'C:\Users\<>\source\repos\mlperf\test_results\DESKTOP_EKP862Q-reference-cpu-onnxruntime-v1.14.1-default_config\resnet50\offline\accuracy'
resnet50.Offline.target_qps = 1
resnet50.Offline.max_query_count = 10
resnet50.Offline.min_query_count = 10
resnet50.Offline.min_duration = 0

        - running time of script "generate,mlperf,inference,user-conf,inference-user-conf": 7.29 sec.

      * Searching for collective script(s) with tags "get,loadgen"
        - Number of scripts found: 1
        - Searching for cached script outputs with the following tags: -tmp,get,loadgen
          - Number of cached script outputs found: 0
        - Found script::get-mlperf-inference-loadgen,64c3d98d0ba04950 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-mlperf-inference-loadgen
        - Checking if script execution is already cached ...
          - Searching for cached script outputs with the following tags: -tmp,get,loadgen,inference,inference-loadgen,mlperf,mlcommons,deps-python-version-3.10.7,deps-python-non-virtual
        - Creating new "cache" script artifact in the CM local repository ...
          - Tags: tmp,get,loadgen,inference,inference-loadgen,mlperf,mlcommons,deps-python-version-3.10.7,deps-python-non-virtual,script-artifact-64c3d98d0ba04950
        - Changing to C:\Users\<>\CM\repos\local\cache\033d94b6c61f4788
        - Version is not specified - use either default_version from meta or min/max/usable: master
        - Checking dependencies on other CM scripts:

          * Searching for collective script(s) with tags "detect,os"
            - Number of scripts found: 1
            - Found script::detect-os,863735b7db8c44fc in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os
            - Running preprocess ...

            - Running native script "C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\detect-os\run.bat" from temporal script "tmp-run.bat" in "C:\Users\<>\CM\repos\local\cache\033d94b6c61f4788" ...

            - Running postprocess ...

            * Searching for collective script(s) with tags "get,sys-utils-min"
              - Number of scripts found: 1
              - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                - Number of cached script outputs found: 1
              - Found script::get-sys-utils-min,a9af7714d3d94779 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-sys-utils-min
              - Checking if script execution is already cached ...
                - Searching for cached script outputs with the following tags: -tmp,get,sys-utils-min
                - Found cached script output: C:\Users\<>\CM\repos\local\cache\f986f803f495404b
                - Checking prehook dependencies on other CM scripts:
                  - Loading state from cached entry ...
                - Checking posthook dependencies on other CM scripts:
                - Checking post dependencies on other CM scripts:
              - running time of script "get,sys-utils-min": 0.94 sec.
            - running time of script "detect-os,detect,os,info": 1.89 sec.

          * Searching for collective script(s) with tags "get,python3"
            - Number of scripts found: 1
            - Searching for cached script outputs with the following tags: -tmp,get,python3
              - Number of cached script outputs found: 1
            - Found script::get-python3,d0b5dd74373f4a62 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-python3
            - Checking if script execution is already cached ...
              - Searching for cached script outputs with the following tags: -tmp,get,python3,python,get-python,get-python3
              - Found cached script output: C:\Users\<>\CM\repos\local\cache\3b1fe588b7944001
              - Checking prehook dependencies on other CM scripts:
                - Loading state from cached entry ...
              - Checking posthook dependencies on other CM scripts:
              - Checking post dependencies on other CM scripts:
            - running time of script "get,python,python3,get-python,get-python3": 0.88 sec.

          * Searching for collective script(s) with tags "get,mlcommons,inference,src" and variations "pybind"
            - Number of scripts found: 1
            - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src,_pybind
              - Number of cached script outputs found: 1
            - Found script::get-mlperf-inference-src,4b57186581024797 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-mlperf-inference-src
              Prepared variations: _pybind
              - Requested version:  == master
            - Checking if script execution is already cached ...
              - Prepared variations: _pybind
              - Searching for cached script outputs with the following tags: -tmp,get,mlcommons,inference,src,source,inference-src,inference-source,mlperf,_pybind,version-master
              - Found cached script output: C:\Users\<>\CM\repos\local\cache\9bb76720ddb842ce
            - Checking dynamic dependencies on other CM scripts:
            - Processing env after dependencies ...
              - Checking prehook dependencies on other CM scripts:
                - Loading state from cached entry ...
              - Checking posthook dependencies on other CM scripts:
              - Checking post dependencies on other CM scripts:
            - running time of script "get,src,source,inference,inference-src,inference-source,mlperf,mlcommons": 0.88 sec.

          * Searching for collective script(s) with tags "get,compiler"
            - Number of scripts found: 3
            - Searching for cached script outputs with the following tags: -tmp,get,compiler
              - Number of cached script outputs found: 1
            - Found script::get-cl,7dbb770faff947c0 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-cl
            - Checking if script execution is already cached ...
              - Searching for cached script outputs with the following tags: -tmp,get,compiler,cl,c-compiler,cpp-compiler,get-cl
              - Found cached script output: C:\Users\<>\CM\repos\local\cache\18553779650a4120
              - Checking prehook dependencies on other CM scripts:
                - Loading state from cached entry ...
              - Checking posthook dependencies on other CM scripts:
              - Checking post dependencies on other CM scripts:
            - running time of script "get,cl,compiler,c-compiler,cpp-compiler,get-cl": 0.88 sec.

          * Searching for collective script(s) with tags "get,cmake"
            - Number of scripts found: 1
            - Searching for cached script outputs with the following tags: -tmp,get,cmake
              - Number of cached script outputs found: 1
            - Found script::get-cmake,52bf974d791b4fc8 in C:\Users\<>\CM\repos\mlcommons@ck\cm-mlops\script\get-cmake
              - Requested version:  >= 3.1
            - Checking if script execution is already cached ...
              - Searching for cached script outputs with the following tags: -tmp,get,cmake,get-cmake
              - Found cached script output: C:\Users\<>\CM\repos\local\cache\1203e5f69d6f41ee
            - Checking dynamic dependencies on other CM scripts:
            - Processing env after dependencies ...
              - Checking prehook dependencies on other CM scripts:
                - Loading state from cached entry ...
              - Checking posthook dependencies on other CM scripts:
              - Checking post dependencies on other CM scripts:
            - running time of script "get,cmake,get-cmake": 0.93 sec.
        - Processing env after dependencies ...
          # potential PIP version string (if needed): ==master
        - Running preprocess ...

gfursin commented 1 year ago

Hi @mwangistan,

Thank you for your feedback. The CK2 (CM) concept is to provide wrappers (CM scripts) for main DevOps/MLOps tasks and let the community gradually add portability across all SW/HW stacks. Some of the CM scripts support all OSes (Linux, MacOS, Windows) and some are not.

We have added and tests CM script for MLPerf inference for Linux and MacOS but we did not fully ported it to Windows because there was no requests. Can you please say why would you like to run MLPerf inference on Windows and if you will be interested to help the community by providing missing support for Windows?

Please feel free to join our Discord server to discuss that: https://discord.gg/JjWNWXKxwT .

Thank you and have a good week, Grigori

mwangistan commented 1 year ago

Hi @gfursin , I appreciate the feedback. I was looking into comparing inference speeds across Linux, Mac and Windows. Are there any scripts that currently work on windows that I can start with to get a feel of it?

I would be happy to help the community. Thanks

gfursin commented 1 year ago

Hi @mwangistan - I would like to check these scripts with you on my Windows machine.

Will you be interested to set up a conf-call?

By the way, I created a Collective Knowledge challenge to run MLPerf inference v3.1 on Windows: https://access.cknowledge.org/playground/?action=challenges&name=53e56d714c7649c7

mwangistan commented 1 year ago

Great, thanks. I've sent you a message on discord to schedule the call.

gfursin commented 1 year ago

Hi Stanley, It was nice talking to you this week. As we discussed, please share a README about how you ran MLPerf inference on Windows first without CM/CK and later we can work together to add support for CM/CK. Note, that we are resuming our regular weekly conf-calls starting from this Thursday: https://github.com/mlcommons/ck/blob/master/docs/taskforce.md#weekly-conf-calls . Feel free to join to share your feedback. Thanks!

mwangistan commented 1 year ago

Hi @gfursin. Thanks for having the chat. I've created a fork with some changes I had made to the mlperf source repo. You can find a readme here: https://github.com/mwangistan/inference/blob/master/vision/classification_and_detection/GettingStarted.md I'll join the call this week to share my feedback. Thanks

gfursin commented 1 year ago

Great! Thank you very much @mwangistan . Let's check it during the conf-call tomorrow!

gfursin commented 1 year ago

Hi @mwangistan . Due to the overlap with the MLCommons community meeting, we decided to cancel our weekly conf-call tomorrow and will have another one next week ... Sorry about that and looking forward to talking to you soon!

mwangistan commented 1 year ago

Hi @gfursin sounds good. Thanks

gfursin commented 1 year ago

Hi @mwangistan,

I've managed to run MLPerf on Windows following these docs (https://github.com/mwangistan/inference/blob/master/vision/classification_and_detection/GettingStarted.md and https://github.com/mwangistan/inference/blob/master/vision/classification_and_detection/README.md#usage) and reusing RetinaNet ONNX model from the CM cache:

D:\grigori\inference\vision\classification_and_detection>python python/main.py --profile retinanet-onnxruntime --scenario Offline --model D:\CM\repos\local\cache\b5df9a3024564ba1\resnext50_32x4d_fpn.onnx --dataset-path D:\grigori\inference\vision\classification_and_detection\downloaded_dataset --accuracy

INFO:main:Namespace(dataset='openimages-800-retinanet-onnx', dataset_path='D:\\grigori\\inference\\vision\\classification_and_detection\\downloaded_dataset', dataset_list=None, data_format=None, profile='retinanet-onn
xruntime', scenario='Offline', max_batchsize=1, model='D:\\Work1\\CM\\repos\\local\\cache\\b5df9a3024564ba1\\resnext50_32x4d_fpn.onnx', output='output', inputs=['images'], outputs=['boxes', 'labels',
'scores'], backend='onnxruntime', model_name='retinanet', threads=8, qps=None, cache=0, cache_dir=None, preprocessed_dir=None, use_preprocessed_dataset=False, accuracy=True, find_peak_performance=Fals
e, debug=False, mlperf_conf='../../mlperf.conf', user_conf='user.conf', audit_conf='audit.config', time=None, count=None, performance_sample_count=None, max_latency=None, samples_per_query=8)
INFO:coco:loaded 100 images, cache=0, already_preprocessed=False, took=0.0sec

C:\!Progs\Python39\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
  warnings.warn(INFO:main:starting TestScenario.Offline

loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Loading and preparing results...
Converting ndarray to lists...
(12228, 7)
0/12228
DONE (t=0.07s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=1.50s).
Accumulating evaluation results...
DONE (t=0.63s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.439
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.610
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.452
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.030
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.240
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.496
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.455
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.597
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.618
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.152
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.410
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.666
TestScenario.Offline qps=0.96, mean=69.5604, time=104.445, acc=44.194%, mAP=43.860%, queries=100, tiles=50.0:75.1461,80.0:101.4416,90.0:107.0209,95.0:108.2511,99.0:113.3382,99.9:114.7423

It should be possible to support MLPerf for Windows in our CM workflows then! I will try to update CM workflows soon and will get back in touch!

mwangistan commented 1 year ago

Hi @gfursin that's great to hear. Looking forward to it. Thanks

gfursin commented 1 year ago

Hi again @mwangistan .

I've added support for Windows for scripts required for MLPerf-RetinaNet. I also added a CM test script to run it on Windows: https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-mlperf-inference-retinanet-win .

You can try it as follows:

python3 -m pip install cmind
cm pull repo mlcommons@ck
cm run script "test mlperf-inference-win retinanet windows"

I didn't add the full support for Windows to our universal CM-MLPerf inference script since it will require much more updates and we don't have time to do it right now. But if it's of interest, you can use it as example and maybe update other scripts ... We can discuss it during one of our conf-calls. Thank you very much for your feedback and have a good weekend!

mwangistan commented 1 year ago

Hi @gfursin thanks for adding the support. I'm getting an error when running cm run script "test mlperf-inference-win retinanet windows Error: Running python.exe setup.py develop bindings/python_api.cc(27): fatal error C1083: Cannot open include file: 'pybind11/functional.h': No such file or directory CM error: Portable CM script failed (name = get-mlperf-inference-loadgen, return code = 1)

gfursin commented 1 year ago

Hi @mwangistan . Just a note, did you clean the cache and update the mlcommons@ck repo? Can you please try again:

python -m pip uninstall mlperf_loadgen
cm pull repo mlcommons@ck
cm rm cache -f
cm run script "test mlperf-inference-win retinanet windows"

Otherwise, I think CM picks up older loadgen version without a patch ...

I just checked above steps and it worked on my Windows 10 with Python 3.9.6 and Visual Studio Community Edition 2022.

However, if it doesn't work, then I will check it again ... Can you please then send the whole log from the start? Thanks!

mwangistan commented 1 year ago

@gfursin It works now. I hadn't cleared the cache. Thanks

gfursin commented 1 year ago

Cool! I close this ticket. Thank you for your feedback @mwangistan !

mlcommons / ck

CM error: Windows is not supported in this script yet! #696