Make a video

[Agalakdak]

Hello. I have been trying to run at least 1 test for a long time and I constantly get errors. Please record a video or give me a link so that I can understand what a normal launch without pain should look like.

[Agalakdak]

I will donate 10$

[psyhtest]

Which test? Which errors?

[Agalakdak]

I encountered so many errors that I don't know where to start. For example, now I followed this document https://docs.mlcommons.org/inference/install/ Then I decided to go straight here https://docs.mlcommons.org/inference/benchmarks/medical_imaging/3d-unet/ And when I entered cm run script --tags=install,python-venv --name=mlperf

I got the message CM error: automation script not found!

I found this error today, up to this point the instructions worked. The log is below user@user:~~$ mkdir test user@user:~~$ cd test/ user@user:~~/test$ python3 -m venv cm user@user:~~/test$ source cm/bin/activate (cm) user@user:~~/test$ pip install cm4mlops Collecting cm4mlops Using cached cm4mlops-0.2-py3-none-any.whl Collecting cmind Using cached cmind-2.3.5.tar.gz (63 kB) Preparing metadata (setup.py) ... done Collecting pyyaml Using cached PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB) Collecting requests Using cached requests-2.32.3-py3-none-any.whl (64 kB) Collecting giturlparse Using cached giturlparse-0.12.0-py2.py3-none-any.whl (15 kB) Collecting setuptools>=60 Using cached setuptools-74.0.0-py3-none-any.whl (1.3 MB) Collecting wheel Using cached wheel-0.44.0-py3-none-any.whl (67 kB) Collecting charset-normalizer<4,>=2 Using cached charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB) Collecting idna<4,>=2.5 Using cached idna-3.8-py3-none-any.whl (66 kB) Collecting certifi>=2017.4.17 Downloading certifi-2024.8.30-py3-none-any.whl (167 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.3/167.3 KB 1.3 MB/s eta 0:00:00 Collecting urllib3<3,>=1.21.1 Using cached urllib3-2.2.2-py3-none-any.whl (121 kB) Using legacy 'setup.py install' for cmind, since package 'wheel' is not installed. Installing collected packages: wheel, urllib3, setuptools, pyyaml, idna, giturlparse, charset-normalizer, certifi, requests, cmind, cm4mlops Attempting uninstall: setuptools Found existing installation: setuptools 59.6.0 Uninstalling setuptools-59.6.0: Successfully uninstalled setuptools-59.6.0 Running setup.py install for cmind ... done Successfully installed certifi-2024.8.30 charset-normalizer-3.3.2 cm4mlops-0.2 cmind-2.3.5 giturlparse-0.12.0 idna-3.8 pyyaml-6.0.2 requests-2.32.3 setuptools-74.0.0 urllib3-2.2.2 wheel-0.44.0 (cm) user@user:~/test$ cm run script --tags=install,python-venv --name=mlperf

CM error: automation script not found! (cm) user@user:~/test$

[Agalakdak]

I ran this script

cm run --tags=run-mlperf,inference,_find-performance,_full,_r4.0 --model=3d-unet-99 --implementation=intel --framework=pytorch --category=edge - -scenario=Offline --execution_mode=test --device=cpu --quiet --test_query_count=50

There is no libffi7 package on my Ubuntu 23.04 Log below

sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libffi7 Reading package lists... Done Building dependency tree... Done Reading state information... Done E: Unable to find package libffi7

CM error: Portable CM script failed (name = get-generic-sys-util, return code = 256) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Note that it is often a portability issue of a third-party tool or a native script wrapped and unified by this CM script (automation recipe). Please re-run this script with --repro flag and report this issue with the original command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts to make existing tools and native scripts more portable, interoperable and deterministic. Thank you!

[arjunsuresh]

Hi @Agalakdak The docs page uses docker option as the default to avoid such OS dependent issues. Is there a reason you don't want to use docker?

[Agalakdak]

Hi @arjunsuresh ! Sorry for the long reply, I tried different ways to solve it. I used the command

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.0 --model=3d-unet-99 --implementation=intel --framework=pytorch --category=edge --scenario=Offline --execution_mode=test --device=cpu --docker --quiet --test_query_count=50

From here https://docs.mlcommons.org/inference/benchmarks/medical_imaging/3d-unet/

And at the last step I got an error 129.8 /home/cmuser/CM/repos/local/cache/b6acf79e843b4c1e/miniconda3/bin/conda install -y -c intel mkl-include 130.2 Collecting package metadata (current_repodata.json): ...working... failed 132.3 132.3 UnavailableInvalidChannel: HTTP 403 FORBIDDEN for channel intel https://conda.anaconda.org/intel 132.3 132.3 The channel is not accessible or is invalid. 132.3 132.3 You will need to adjust your conda configuration to proceed. 132.3 Use conda config --show channels to view your configuration's current state, 132.3 and use conda config --show-sources to view config file locations. 132.3 132.3 132.5 Detected version: 3.10.12 132.5 Detected version: 3.10.12 132.5 Detected version: 22.0.2 132.5 132.5 Extra PIP CMD: 132.5 132.5 Detected version: 3.0.0 132.5 Detected version: 24.7.1 132.5 Detected version: 3.8.0 132.5 132.5 CM error: Portable CM script failed (name = install-generic-conda-package, return code = 256)

And some more logs in the file error_with_docker.txt

I don't understand what to do with this. What information can I provide you to solve the problem?

[Agalakdak]

@arjunsuresh, I tried to run another benchmark. But there was an error there too. Please help me figure it out.

Command cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.0 \ --model=retinanet \ --implementation=intel \ --framework=pytorch \ --category=edge \ --scenario=Offline \ --execution_mode=test \ --device=cpu \ --docker --quiet \ --test_query_count=100

Error log 1762.7 environment: line 1: 51417 Killed ${CM_PYTHON_BIN_WITH_PATH} "$@" 1762.8 1762.8 CM error: Portable CM script failed (name = get-dataset-openimages, return code = 256) 1762.8 1762.8 1762.8 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1762.8 Note that it is often a portability issue of a third-party tool or a native script 1762.8 wrapped and unified by this CM script (automation recipe). Please re-run 1762.8 this script with --repro flag and report this issue with the original 1762.8 command line, cm-repro directory and full log here: 1762.8 1762.8 https://github.com/mlcommons/cm4mlops/issues 1762.8 1762.8 The CM concept is to collaboratively fix such issues inside portable CM scripts 1762.8 to make existing tools and native scripts more portable, interoperable 1762.8 and deterministic. Thank you! 1762.8 1762.8 1762.8 Using MLCommons Inference source from '/home/cmuser/CM/repos/local/cache/f93090d427b8435f/inference' 1762.8

---

1 warning found (use docker --debug to expand):

• SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "CM_GH_TOKEN") (line 14) ubuntu_22.04.Dockerfile:47

Full error log
error_with_docker2.txt

[arjunsuresh]

Hi @Agalakdak we do have problem with Intel implementation as reported here. We'll work with Intel to fix these. But even then Intel implementation is expected to work on only the latest Intel server/workstation CPUs - we'll update this in the documentation.

[Agalakdak]

Hi @arjunsuresh, Hi arjunsuresh, thanks for the prompt reply. I'll check that code on the intel XEON GOLD 6346 x2 processor a little later

[Agalakdak]

@arjunsuresh Can I clarify for the future? Are there any problems with the "Quadro RTX 5000" and "Nvidia A40" video cards?

[arjunsuresh]

@Agalakdak Nvidia doesn't officially support them for MLPerf inference. But typically we have had good success running Nvidia code on such GPUs without much difficulty. Do you have a plan of what all you are trying to benchmark?

[Agalakdak]

@arjunsuresh Yes, sure. First, I'd like to just run one of the inference benchmarks and compare them with the "reference indicators". If the launch is successful, I'll try to run a benchmark for "training" the network on several video cards using a docker container. And then use these results to find bottlenecks in the system (if there are any)

Today I'll try to run as many benchmarks as possible. And then I'll write about the results. If you need any additional information about the system, let me know.

[arjunsuresh]

@Agalakdak If you want to run as many benchmarks as possible the best option to start with will be using Nvidia implementation. Even if any issue is there, they are usually quickly resolvable. If you just want to try getting a result, reference implementation is good for smaller models like resnet50 and bert-99 as it runs on almost any CPUs.

And if you are referring to MLPerf training benchmarks - hat's very different from inference even though many of the models in inference come from MLPerf training. Currently there is no automated way to run training benchmarks and the only option is to follow the submitter READMEs is the results repository. https://github.com/mlcommons/training_results_v4.0

[Agalakdak]

@arjunsuresh Oh, thanks a lot for the help, but I'm afraid I have another question. I successfully ran "Text to Image using Stable Diffusion" cm run script --tags=run-mlperf,inference,_r4.1 \ --model=sdxl \ --implementation=reference \ --framework=pytorch \ --category=edge \ --scenario=Offline \ --execution_mode=valid \ --device=cuda \ --quiet My GPU did do some work. But... After all this I ended up inside a container and can't find any results, neither in the container nor in the logs

error_with_docker3.txt

[arjunsuresh]

@Agalakdak that's only the first step. You need to do the following command from the documentation page inside that docker container.

[Agalakdak]

Hello @arjunsuresh , unfortunately a new day and new problems On one of the systems I started SD (and the test has been running for several hours, I don't know if I need to stop it forcibly or it will handle it itself?), but on the other one it doesn't. I ran the same command

cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev \ --model=sdxl \ --implementation=nvidia \ --framework=tensorrt \ --category=edge \ --scenario=Offline \ --execution_mode=test \ --device=cuda \ --docker --quiet \ --test_query_count=50

What can I do with this error?

/usr/include/x86_64-linux-gnu/bits/mathcalls.h(110): error: identifier "_Float32" is undefined Error limit reached. 100 errors detected in the compilation of "print_cuda_devices.cu". Compilation terminated.

CM error: Portable CM script failed (name = get-cuda-devices, return code = 256) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Note that it is often a portability issue of a third- party tool or a native script wrapped and unified by this CM script (automation recipe). Please re-run this script with --repro flag and report this issue with the original command line, cm-repro directory and full log here: https://github.com/mlcommons/cm4mlops/issues The CM concept is to collaboratively fix such issues inside portable CM scripts to make existing tools and native scripts more portable, interoperable and deterministic. Thank you!

Full log with error
server_err_sd_1.log

[arjunsuresh]

@Agalakdak the problem is due to CUDA compilation not working on the host machine. This is actually not a necessity though we never had such an issue before. Let me share you the option to skip this.

@anandhu-eng are you able to share this option?

[Agalakdak]

@arjunsuresh Hi, I encountered a similar problem when I wanted to run ResNet50 I launched: cm run script --tags=run-mlperf,inference,_find-performance,_full,_r4.1-dev \ --model=resnet50 \ --implementation=reference \ --framework=onnxruntime \ --category=edge \ --scenario=Offline \ --execution_mode=test \ --device=cuda \ --docker --quiet \ --test_query_count=1000

and it was ok.

And it worked fine - I got inside the container. In the container I ran cm run script --tags=run-mlperf,inference,_r4.1-dev,_all-scenarios --model=resnet50 --implementation=reference --framework=onnxruntime --category=edge --execution_mode=valid --device=cuda --quiet

And I got (I assume) a similar error.

INFO:root: ! call /home/cmuser/CM/repos/mlcommons@cm4mlops/script/get-cuda-devices/run.sh from tmp-run.sh rm: cannot remove 'a.out': No such file or directory

Checking compiler version ...

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Tue_Feb_27_16:19:38_PST_2024 Cuda compilation tools, release 12.4, V12.4.99 Build cuda_12.4.r12.4/compiler.33961263_0

Compiling program ...

Running program ...

/home/cmuser INFO:root:======================================================== INFO:root:Print file tmp-run.out: INFO:root: INFO:root:Error: problem obtaining number of CUDA devices: 100

INFO:root:

CM error: Portable CM script failed (name = get-cuda-devices, return code = 256)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Note that it is often a portability issue of a third-party tool or a native script wrapped and unified by this CM script (automation recipe). Please re-run this script with --repro flag and report this issue with the original command line, cm-repro directory and full log here:

https://github.com/mlcommons/cm4mlops/issues

The CM concept is to collaboratively fix such issues inside portable CM scripts to make existing tools and native scripts more portable, interoperable and deterministic. Thank you!

Log with error error_resnet50_docker_log.txt

[arjunsuresh]

Hi @Agalakdak We also sometimes face the below error while using Nvidia GPUs inside a container

INFO:root:Error: problem obtaining number of CUDA devices: 100

A quick fix for this is to exit the container. Use docker ps -a to get the container ID say id. Then do docker start id && docker attach id and we should be back where we were but with working Nvidia GPUs.

We have also removed the requirement to have NVCC in the host system - please do cm pull repo and you should be able to run sdxl.