search

mlcommons / inference_results_v0.5

This repository contains the results and code for the MLPerf™ Inference v0.5 benchmark.
https://mlcommons.org/en/inference-datacenter-05/
Apache License 2.0
55 stars 43 forks source link

DellEMC: No script is provided to run ssd-small #11

Open psyhtest opened 4 years ago

psyhtest commented 4 years ago

The README for SSD-Small using OpenVINO in the DellEMC submission, says:

Run the corresponding scripts under scripts/ folder

This folder, however, contains two files:

psyhtest commented 4 years ago

If I were to guess, this scripts/ folder should have been populated similarly to that in the corresponding Intel submission:

offline_ssdm.sh
offline_ssdm_accuracy.sh
server_ssdm.sh
server_ssdm_accuracy.sh
singlestream_ssdm.sh
singlestream_ssdm_accuracy.sh
psyhtest commented 4 years ago

Guessing further, SSD-Small in the Offline scenario and Performance mode would use offline_ssdm.sh:

Release_OMP/ov_mlperf --scenario Offline --mode Performance \
--mlperf_conf_filename "mlperf.conf" --user_conf_filename "scripts/ssdmobilenet_user.conf" \
--total_sample_count 5000 --data_path "/root/CK-TOOLS/dataset-coco-2017-val/" \
--dataset coco --model_path models/ssd-mobilenet/ssd-mobilenet_int8.xml \
--model_name ssd-mobilenet --nstreams 112 --nireq 112 --nthreads 112 \
--batch_size 1 --device cpu --nwarmup_iters 1000
psyhtest commented 4 years ago

To investigate a bit further, I've also decided to take a look at the measurements/ directory for the SSD-Small benchmark. To my surprise, it only contains one Intel entry, so only covers one Intel CPU (Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz) out of two and one scenario (Offline) out of two used in the submission. Moreover, user.conf is identical to the one Intel used in their submission (Intel(R) Xeon(R) Platinum 9282 CPU @ 2.60GHz gives it away immediately).

psyhtest commented 4 years ago

Curiously, neither the user.conf under measurements/ (intended for Intel(R) Xeon(R) Platinum 9282 CPU @ 2.60GHz)

*.Server.target_qps = 5260
*.Offline.target_qps = 10000

nor the ssdmobilenet_user.conf under code/ (intended for Intel(R) Xeon(R) 8280 CPU, 28 cores x 2):

*.Server.target_qps = 3000
*.Offline.target_qps = 4800

match the Dell systems well considering their respective Offline scores:

It's probably fine for the Offline mode, but could have led to suboptimal scores in the Server mode (not reported).

psyhtest commented 4 years ago

It's also peculiar that while the Offline score (inputs/second) is better (higher) for Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz (RCP: $8719) than for Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (RCP: $3075), the SingleStream score (milliseconds/sample) is better (lower) for the cheaper one: 1.54 vs. 1.69.

psyhtest commented 4 years ago

(Aside: It's a pity that no pricing is suggested for Intel(R) Xeon(R) Platinum 9282 CPU @ 2.60GHz.)

psyhtest commented 4 years ago

(Aside again: But for the best CPU for which pricing is available, Intel® Xeon® Platinum 8280L Processor @ 2.70GHz, RCP is $17906.)

psyhtest commented 4 years ago

(Yet another aside: In the preview category, Tencent used four Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz absent from the Intel site. The closest match I can find - 24 cores per CPU - is Intel® Xeon® Platinum 8260 Processor @ 2.40 GHz with RCP of $4702.)

christ1ne commented 4 years ago

The README for SSD-Small using OpenVINO in the DellEMC submission, says:

Run the corresponding scripts under scripts/ folder

This folder, however, contains two files:

BLOCKED FILE ALERTA file has been blocked due to the 'Intel Email File Filtering Policy' rule.Context: 'ssdmobilenet_int8_offline.sh'Disallowed due to filenameTicket Number: '0ef4-5da0-0830-0001'
See your system administrator for further information. Copyright 1999-2011 McAfee, Inc.All Rights Reserved.http://www.mcafee.com
# Hardware: Intel(R) Xeon(R) 8280 CPU, 28 cores x 2

(which is neither Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz nor Intel(R) Xeon(R) Platinum 8276 CPU @ 2.20GHz that Dell EMC used in their submission.)

Please ignore all comments regarding '8280', that was a mistake in the comment. The actual CPU SKUs are reported correctly in the inference result table.

The Tencent SKU is a custom SKU so it is not reported on the Intel website.

psyhtest commented 4 years ago

Thanks @christ1ne. I had to look up SKU in Wikipedia (and donate £5 for its upkeep!)

The comment is wrong, but so is *.Offline.target_qps = 4800 for Dell EMC's CPUs?

christ1ne commented 4 years ago

My understanding is that you just need to set a number which is a reasonable for your HW. The final reported performance numbers will be from the logs.