CM error: Portable CM script failed (name = benchmark-program, return code = 1)

[RommelCF]

I'm attempting to run Bert from the MLperf Edge on an AMD CPU under windows11, using the following cmds:

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1 \
   --model=bert-99 \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=Offline \
   --execution_mode=test \
   --device=cpu  \
   --quiet \
   --test_query_count=100

I get the following error:

Constructing QSL...
No cached features at 'eval_features.pickle'... converting from examples...
Creating tokenizer...
Traceback (most recent call last):
  File "C:\Users\Administrator\CM\repos\local\cache\ad5e6da37dd24c13\inference\language\bert\run.py", line 150, in <module>
    main()
  File "C:\Users\Administrator\CM\repos\local\cache\ad5e6da37dd24c13\inference\language\bert\run.py", line 88, in main
    sut = get_onnxruntime_sut(args)
  File "C:\Users\Administrator\CM\repos\local\cache\ad5e6da37dd24c13\inference\language\bert\onnxruntime_SUT.py", line 116, in get_onnxruntime_sut
    return BERT_ONNXRuntime_SUT(args)
  File "C:\Users\Administrator\CM\repos\local\cache\ad5e6da37dd24c13\inference\language\bert\onnxruntime_SUT.py", line 57, in __init__
    self.qsl = get_squad_QSL(args.max_examples)
  File "C:\Users\Administrator\CM\repos\local\cache\ad5e6da37dd24c13\inference\language\bert\squad_QSL.py", line 96, in get_squad_QSL
    return SQuAD_v1_QSL(total_count_override, perf_count_override)
  File "C:\Users\Administrator\CM\repos\local\cache\ad5e6da37dd24c13\inference\language\bert\squad_QSL.py", line 50, in __init__
    tokenizer = BertTokenizer(vocab_file)
  File "C:\Users\Administrator\CM\repos\local\cache\36adcf167af74675\mlperf\lib\site-packages\transformers\models\bert\tokenization_bert.py", line 111, in __init__
    raise ValueError(
ValueError: Can't find a vocabulary file at path 'C:\Users\Administrator\CM\repos\local\cache\f4034a07d99647ed\vocab.txt'. To load the vocabulary from a Google pretrained model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`
Finished destroying QSL.

CM error: Portable CM script failed (name = benchmark-program, return code = 1)

cm-run-script-input.json

[arjunsuresh]

I don't think bert will run natively on Windows as some of the dependencies might be missing the run.bat files. Using --docker option can run the model via docker.

[RommelCF]

Can you please add support for windows?

[arjunsuresh]

Sorry. We don't have Windows environment to do that.

[RommelCF]

Ok, you mean you don't have the hardware available to test?

[arjunsuresh]

Windows is the problem :)

[RommelCF]

Sorry. We don't have Windows environment to do that.

If hardware is not the problem, what is? Just trying to understand. Thanks.

[arjunsuresh]

Hardly any MLPerf user uses Windows OS and those who do uses docker or WSL. Why are you trying to run MLPerf on Windows natively?

[RommelCF]

Maybe if support was there more people would use it. Kinda of a chicken and egg problem. I'll try via docker. Thanks.

[RommelCF]

Attempted to run via docker, but getting this error: CM error: Portable CM script failed (name = process-mlperf-accuracy, return code = 256) and it took a good 6 hours to fail, CPU was 100% busy during that time.

[arjunsuresh]

process-mlperf-accuracy is the last step. Can you please share the full available console output?