Closed mahmoodn closed 1 year ago
For DLRM in Nvidia folder, I get the following error while running the preprocess target.
Skip existing /disk1/scratch/data/criteo/day_22.npz Skip existing /disk1/scratch/data/criteo/day_23.npz Total number of samples: 4373472329 Divided into days/splits: [195841983, 199563535, 196792019, 181115208, 152115810, 172548507, 204846845, 200801003, 193772492, 198424372, 185778055, 153588700, 169003364, 194216520, 194081279, 187154596, 177984934, 163382602, 142061091, 156534237, 193627464, 192215183, 189747893, 178274637] Traceback (most recent call last): File "code/dlrm/tensorrt/preprocess_data.py", line 100, in <module> main() File "code/dlrm/tensorrt/preprocess_data.py", line 94, in main preprocess_dlrm(data_dir, model_dir, preprocessed_data_dir) File "code/dlrm/tensorrt/preprocess_data.py", line 43, in preprocess_dlrm loadDataset( File "/work/code/dlrm/tensorrt/scripts/data_utils.py", line 1174, in loadDataset file = getCriteoAdData( File "/work/code/dlrm/tensorrt/scripts/data_utils.py", line 1108, in getCriteoAdData processCriteoAdData(d_path, d_file, npzfile, days, convertDicts, counts) File "/work/code/dlrm/tensorrt/scripts/data_utils.py", line 143, in processCriteoAdData X_cat_t[j, k] = convertDicts[j][x] KeyError: 18855508 make: *** [Makefile:463: preprocess_data] Error 1
I think I have downloaded all data files in correct shape. Also, there is no memory shortage. Please see the file sizes below:
$ ls -l /disk1/scratch/data/criteo/ total 1350264428 -rw-rw-r-- 1 mahmood mahmood 49769814062 May 11 06:24 day_0 -rw-r--r-- 1 mahmood mahmood 12100793147 Jun 9 14:12 day_0.npz -rw-rw-r-- 1 mahmood mahmood 50706674331 May 11 08:15 day_1 -rw-rw-r-- 1 mahmood mahmood 47134635824 May 11 11:18 day_10 -rw-r--r-- 1 mahmood mahmood 11540422179 Jun 13 22:18 day_10.npz -rw-rw-r-- 1 mahmood mahmood 38895137658 May 11 11:45 day_11 -rw-r--r-- 1 mahmood mahmood 9649040863 Jun 14 00:08 day_11.npz -rw-rw-r-- 1 mahmood mahmood 42836573400 May 11 11:57 day_12 -rw-r--r-- 1 mahmood mahmood 10659035621 Jun 14 02:14 day_12.npz -rw-rw-r-- 1 mahmood mahmood 49264558035 May 11 12:20 day_13 -rw-r--r-- 1 mahmood mahmood 12109498929 Jun 14 04:39 day_13.npz -rw-rw-r-- 1 mahmood mahmood 49212487346 May 11 12:33 day_14 -rw-r--r-- 1 mahmood mahmood 12068935722 Jun 14 07:03 day_14.npz -rw-rw-r-- 1 mahmood mahmood 47496840021 May 11 12:49 day_15 -rw-r--r-- 1 mahmood mahmood 11687473742 Jun 15 14:21 day_15.npz -rw-rw-r-- 1 mahmood mahmood 45152686382 May 11 13:20 day_16 -rw-r--r-- 1 mahmood mahmood 11094311501 Jun 15 16:34 day_16.npz -rw-rw-r-- 1 mahmood mahmood 41343694115 May 11 13:49 day_17 -rw-r--r-- 1 mahmood mahmood 10236861739 Jun 15 18:37 day_17.npz -rw-rw-r-- 1 mahmood mahmood 35879970929 May 11 13:59 day_18 -rw-r--r-- 1 mahmood mahmood 8963788330 Jun 15 20:23 day_18.npz -rw-rw-r-- 1 mahmood mahmood 39590452666 May 11 14:27 day_19 -rw-r--r-- 1 mahmood mahmood 9917450120 Jun 15 22:20 day_19.npz -rw-r--r-- 1 mahmood mahmood 12373244857 Jun 9 16:41 day_1.npz -rw-rw-r-- 1 mahmood mahmood 49992139833 May 11 08:44 day_2 -rw-rw-r-- 1 mahmood mahmood 49132845366 May 11 15:14 day_20 -rw-r--r-- 1 mahmood mahmood 12151623839 Jun 16 00:45 day_20.npz -rw-rw-r-- 1 mahmood mahmood 48794401031 May 11 15:44 day_21 -rw-r--r-- 1 mahmood mahmood 11921852661 Jun 16 03:11 day_21.npz -rw-rw-r-- 1 mahmood mahmood 48166421859 May 11 16:14 day_22 -rw-r--r-- 1 mahmood mahmood 11747677556 Jul 26 17:33 day_22.npz -rw-rw-r-- 1 mahmood mahmood 45211913694 May 11 16:52 day_23 -rw-r--r-- 1 mahmood mahmood 11088251321 Jul 26 19:41 day_23.npz -rw-r--r-- 1 mahmood mahmood 12171731795 Jun 9 19:07 day_2.npz -rw-rw-r-- 1 mahmood mahmood 45940271708 May 11 09:06 day_3 -rw-r--r-- 1 mahmood mahmood 11226598588 Jun 9 21:21 day_3.npz -rw-rw-r-- 1 mahmood mahmood 38485062888 May 11 09:17 day_4 -rw-r--r-- 1 mahmood mahmood 9511849051 Jun 9 23:08 day_4.npz -rw-rw-r-- 1 mahmood mahmood 43716433766 May 11 09:31 day_5 -rw-r--r-- 1 mahmood mahmood 10774894536 Jun 10 01:18 day_5.npz -rw-rw-r-- 1 mahmood mahmood 52077746076 May 11 09:46 day_6 -rw-r--r-- 1 mahmood mahmood 12718765604 Jun 10 04:05 day_6.npz -rw-rw-r-- 1 mahmood mahmood 51071297573 May 11 10:02 day_7 -rw-r--r-- 1 mahmood mahmood 12503924444 Jun 10 06:42 day_7.npz -rw-rw-r-- 1 mahmood mahmood 49236069361 May 11 10:23 day_8 -rw-r--r-- 1 mahmood mahmood 12034160774 Jun 10 09:12 day_8.npz -rw-rw-r-- 1 mahmood mahmood 50434915625 May 11 10:58 day_9 -rw-r--r-- 1 mahmood mahmood 12292275207 Jun 13 20:02 day_9.npz -rw-r--r-- 1 mahmood mahmood 348 Jun 16 07:52 day_day_count.npz -rw-r--r-- 1 mahmood mahmood 303 Jun 16 07:55 day_fea_count.npz -rw-r--r-- 1 mahmood mahmood 133873916 Jun 16 07:54 day_fea_dict_0.npz -rw-r--r-- 1 mahmood mahmood 6994449 Jun 16 07:54 day_fea_dict_10.npz -rw-r--r-- 1 mahmood mahmood 1178185 Jun 16 07:54 day_fea_dict_11.npz -rw-r--r-- 1 mahmood mahmood 249 Jun 16 07:54 day_fea_dict_12.npz -rw-r--r-- 1 mahmood mahmood 8159 Jun 16 07:54 day_fea_dict_13.npz -rw-r--r-- 1 mahmood mahmood 40233 Jun 16 07:54 day_fea_dict_14.npz -rw-r--r-- 1 mahmood mahmood 796 Jun 16 07:54 day_fea_dict_15.npz -rw-r--r-- 1 mahmood mahmood 220 Jun 16 07:54 day_fea_dict_16.npz -rw-r--r-- 1 mahmood mahmood 3736 Jun 16 07:54 day_fea_dict_17.npz -rw-r--r-- 1 mahmood mahmood 266 Jun 16 07:54 day_fea_dict_18.npz -rw-r--r-- 1 mahmood mahmood 138591150 Jun 16 07:55 day_fea_dict_19.npz -rw-r--r-- 1 mahmood mahmood 128601 Jun 16 07:54 day_fea_dict_1.npz -rw-r--r-- 1 mahmood mahmood 58441045 Jun 16 07:55 day_fea_dict_20.npz -rw-r--r-- 1 mahmood mahmood 128109345 Jun 16 07:55 day_fea_dict_21.npz -rw-r--r-- 1 mahmood mahmood 1735108 Jun 16 07:55 day_fea_dict_22.npz -rw-r--r-- 1 mahmood mahmood 44001 Jun 16 07:55 day_fea_dict_23.npz -rw-r--r-- 1 mahmood mahmood 613 Jun 16 07:55 day_fea_dict_24.npz -rw-r--r-- 1 mahmood mahmood 353 Jun 16 07:55 day_fea_dict_25.npz -rw-r--r-- 1 mahmood mahmood 57644 Jun 16 07:54 day_fea_dict_2.npz -rw-r--r-- 1 mahmood mahmood 26785 Jun 16 07:54 day_fea_dict_3.npz -rw-r--r-- 1 mahmood mahmood 72723 Jun 16 07:54 day_fea_dict_4.npz -rw-r--r-- 1 mahmood mahmood 219 Jun 16 07:54 day_fea_dict_5.npz -rw-r--r-- 1 mahmood mahmood 25262 Jun 16 07:54 day_fea_dict_6.npz -rw-r--r-- 1 mahmood mahmood 5482 Jun 16 07:54 day_fea_dict_7.npz -rw-r--r-- 1 mahmood mahmood 466 Jun 16 07:54 day_fea_dict_8.npz -rw-r--r-- 1 mahmood mahmood 113479498 Jun 16 07:54 day_fea_dict_9.npz
I appreciate any feedback.
@mahmoodn Since 2.1 code is now available, why not give that a shot? https://github.com/mlcommons/inference_results_v2.1
For DLRM in Nvidia folder, I get the following error while running the preprocess target.
I think I have downloaded all data files in correct shape. Also, there is no memory shortage. Please see the file sizes below:
I appreciate any feedback.