search

mlcommons / inference_results_v2.0

This repository contains the results and code for the MLPerf™ Inference v2.0 benchmark.
https://mlcommons.org/en/inference-datacenter-20/
Apache License 2.0
9 stars 12 forks source link

DLRM preprocess failed (Nvidia) #11

Closed mahmoodn closed 1 year ago

mahmoodn commented 2 years ago

For DLRM in Nvidia folder, I get the following error while running the preprocess target.

Skip existing /disk1/scratch/data/criteo/day_22.npz
Skip existing /disk1/scratch/data/criteo/day_23.npz
Total number of samples: 4373472329
Divided into days/splits:
 [195841983, 199563535, 196792019, 181115208, 152115810, 172548507, 204846845, 200801003, 193772492, 198424372, 185778055, 153588700, 169003364, 194216520, 194081279, 187154596, 177984934, 163382602, 142061091, 156534237, 193627464, 192215183, 189747893, 178274637]
Traceback (most recent call last):
  File "code/dlrm/tensorrt/preprocess_data.py", line 100, in <module>
    main()
  File "code/dlrm/tensorrt/preprocess_data.py", line 94, in main
    preprocess_dlrm(data_dir, model_dir, preprocessed_data_dir)
  File "code/dlrm/tensorrt/preprocess_data.py", line 43, in preprocess_dlrm
    loadDataset(
  File "/work/code/dlrm/tensorrt/scripts/data_utils.py", line 1174, in loadDataset
    file = getCriteoAdData(
  File "/work/code/dlrm/tensorrt/scripts/data_utils.py", line 1108, in getCriteoAdData
    processCriteoAdData(d_path, d_file, npzfile, days, convertDicts, counts)
  File "/work/code/dlrm/tensorrt/scripts/data_utils.py", line 143, in processCriteoAdData
    X_cat_t[j, k] = convertDicts[j][x]
KeyError: 18855508
make: *** [Makefile:463: preprocess_data] Error 1

I think I have downloaded all data files in correct shape. Also, there is no memory shortage. Please see the file sizes below:

$ ls -l /disk1/scratch/data/criteo/
total 1350264428
-rw-rw-r-- 1 mahmood mahmood 49769814062 May 11 06:24 day_0
-rw-r--r-- 1 mahmood mahmood 12100793147 Jun  9 14:12 day_0.npz
-rw-rw-r-- 1 mahmood mahmood 50706674331 May 11 08:15 day_1
-rw-rw-r-- 1 mahmood mahmood 47134635824 May 11 11:18 day_10
-rw-r--r-- 1 mahmood mahmood 11540422179 Jun 13 22:18 day_10.npz
-rw-rw-r-- 1 mahmood mahmood 38895137658 May 11 11:45 day_11
-rw-r--r-- 1 mahmood mahmood  9649040863 Jun 14 00:08 day_11.npz
-rw-rw-r-- 1 mahmood mahmood 42836573400 May 11 11:57 day_12
-rw-r--r-- 1 mahmood mahmood 10659035621 Jun 14 02:14 day_12.npz
-rw-rw-r-- 1 mahmood mahmood 49264558035 May 11 12:20 day_13
-rw-r--r-- 1 mahmood mahmood 12109498929 Jun 14 04:39 day_13.npz
-rw-rw-r-- 1 mahmood mahmood 49212487346 May 11 12:33 day_14
-rw-r--r-- 1 mahmood mahmood 12068935722 Jun 14 07:03 day_14.npz
-rw-rw-r-- 1 mahmood mahmood 47496840021 May 11 12:49 day_15
-rw-r--r-- 1 mahmood mahmood 11687473742 Jun 15 14:21 day_15.npz
-rw-rw-r-- 1 mahmood mahmood 45152686382 May 11 13:20 day_16
-rw-r--r-- 1 mahmood mahmood 11094311501 Jun 15 16:34 day_16.npz
-rw-rw-r-- 1 mahmood mahmood 41343694115 May 11 13:49 day_17
-rw-r--r-- 1 mahmood mahmood 10236861739 Jun 15 18:37 day_17.npz
-rw-rw-r-- 1 mahmood mahmood 35879970929 May 11 13:59 day_18
-rw-r--r-- 1 mahmood mahmood  8963788330 Jun 15 20:23 day_18.npz
-rw-rw-r-- 1 mahmood mahmood 39590452666 May 11 14:27 day_19
-rw-r--r-- 1 mahmood mahmood  9917450120 Jun 15 22:20 day_19.npz
-rw-r--r-- 1 mahmood mahmood 12373244857 Jun  9 16:41 day_1.npz
-rw-rw-r-- 1 mahmood mahmood 49992139833 May 11 08:44 day_2
-rw-rw-r-- 1 mahmood mahmood 49132845366 May 11 15:14 day_20
-rw-r--r-- 1 mahmood mahmood 12151623839 Jun 16 00:45 day_20.npz
-rw-rw-r-- 1 mahmood mahmood 48794401031 May 11 15:44 day_21
-rw-r--r-- 1 mahmood mahmood 11921852661 Jun 16 03:11 day_21.npz
-rw-rw-r-- 1 mahmood mahmood 48166421859 May 11 16:14 day_22
-rw-r--r-- 1 mahmood mahmood 11747677556 Jul 26 17:33 day_22.npz
-rw-rw-r-- 1 mahmood mahmood 45211913694 May 11 16:52 day_23
-rw-r--r-- 1 mahmood mahmood 11088251321 Jul 26 19:41 day_23.npz
-rw-r--r-- 1 mahmood mahmood 12171731795 Jun  9 19:07 day_2.npz
-rw-rw-r-- 1 mahmood mahmood 45940271708 May 11 09:06 day_3
-rw-r--r-- 1 mahmood mahmood 11226598588 Jun  9 21:21 day_3.npz
-rw-rw-r-- 1 mahmood mahmood 38485062888 May 11 09:17 day_4
-rw-r--r-- 1 mahmood mahmood  9511849051 Jun  9 23:08 day_4.npz
-rw-rw-r-- 1 mahmood mahmood 43716433766 May 11 09:31 day_5
-rw-r--r-- 1 mahmood mahmood 10774894536 Jun 10 01:18 day_5.npz
-rw-rw-r-- 1 mahmood mahmood 52077746076 May 11 09:46 day_6
-rw-r--r-- 1 mahmood mahmood 12718765604 Jun 10 04:05 day_6.npz
-rw-rw-r-- 1 mahmood mahmood 51071297573 May 11 10:02 day_7
-rw-r--r-- 1 mahmood mahmood 12503924444 Jun 10 06:42 day_7.npz
-rw-rw-r-- 1 mahmood mahmood 49236069361 May 11 10:23 day_8
-rw-r--r-- 1 mahmood mahmood 12034160774 Jun 10 09:12 day_8.npz
-rw-rw-r-- 1 mahmood mahmood 50434915625 May 11 10:58 day_9
-rw-r--r-- 1 mahmood mahmood 12292275207 Jun 13 20:02 day_9.npz
-rw-r--r-- 1 mahmood mahmood         348 Jun 16 07:52 day_day_count.npz
-rw-r--r-- 1 mahmood mahmood         303 Jun 16 07:55 day_fea_count.npz
-rw-r--r-- 1 mahmood mahmood   133873916 Jun 16 07:54 day_fea_dict_0.npz
-rw-r--r-- 1 mahmood mahmood     6994449 Jun 16 07:54 day_fea_dict_10.npz
-rw-r--r-- 1 mahmood mahmood     1178185 Jun 16 07:54 day_fea_dict_11.npz
-rw-r--r-- 1 mahmood mahmood         249 Jun 16 07:54 day_fea_dict_12.npz
-rw-r--r-- 1 mahmood mahmood        8159 Jun 16 07:54 day_fea_dict_13.npz
-rw-r--r-- 1 mahmood mahmood       40233 Jun 16 07:54 day_fea_dict_14.npz
-rw-r--r-- 1 mahmood mahmood         796 Jun 16 07:54 day_fea_dict_15.npz
-rw-r--r-- 1 mahmood mahmood         220 Jun 16 07:54 day_fea_dict_16.npz
-rw-r--r-- 1 mahmood mahmood        3736 Jun 16 07:54 day_fea_dict_17.npz
-rw-r--r-- 1 mahmood mahmood         266 Jun 16 07:54 day_fea_dict_18.npz
-rw-r--r-- 1 mahmood mahmood   138591150 Jun 16 07:55 day_fea_dict_19.npz
-rw-r--r-- 1 mahmood mahmood      128601 Jun 16 07:54 day_fea_dict_1.npz
-rw-r--r-- 1 mahmood mahmood    58441045 Jun 16 07:55 day_fea_dict_20.npz
-rw-r--r-- 1 mahmood mahmood   128109345 Jun 16 07:55 day_fea_dict_21.npz
-rw-r--r-- 1 mahmood mahmood     1735108 Jun 16 07:55 day_fea_dict_22.npz
-rw-r--r-- 1 mahmood mahmood       44001 Jun 16 07:55 day_fea_dict_23.npz
-rw-r--r-- 1 mahmood mahmood         613 Jun 16 07:55 day_fea_dict_24.npz
-rw-r--r-- 1 mahmood mahmood         353 Jun 16 07:55 day_fea_dict_25.npz
-rw-r--r-- 1 mahmood mahmood       57644 Jun 16 07:54 day_fea_dict_2.npz
-rw-r--r-- 1 mahmood mahmood       26785 Jun 16 07:54 day_fea_dict_3.npz
-rw-r--r-- 1 mahmood mahmood       72723 Jun 16 07:54 day_fea_dict_4.npz
-rw-r--r-- 1 mahmood mahmood         219 Jun 16 07:54 day_fea_dict_5.npz
-rw-r--r-- 1 mahmood mahmood       25262 Jun 16 07:54 day_fea_dict_6.npz
-rw-r--r-- 1 mahmood mahmood        5482 Jun 16 07:54 day_fea_dict_7.npz
-rw-r--r-- 1 mahmood mahmood         466 Jun 16 07:54 day_fea_dict_8.npz
-rw-r--r-- 1 mahmood mahmood   113479498 Jun 16 07:54 day_fea_dict_9.npz

I appreciate any feedback.

nv-ananjappa commented 1 year ago

@mahmoodn Since 2.1 code is now available, why not give that a shot? https://github.com/mlcommons/inference_results_v2.1