Encounter "Memory Error" when converting imagenet dataset

[jeff830107]

Hi, When I was trying to using the Alexnet model, I first of all tried to follow your instruction to download val224_compressed.pkl and executed the command "python convert.py" But when I was converting, it always come to the error message "Memory Error". I am curious about how to deal with this issue, since I think the memory of the machine I used is big enough, which is 64 GB. Thanks !

[gdjmck]

I ran into the same problem as well, the 244244 file was dumped okay with 7.5G and the 299299 pkl file was empty with 0B

[jnorwood]

I saw the same issue. I separated the 224 and 299 dump processing loops and cleared variables that were no longer used. Still it dies in dump_pickle, which must be making another copy. So, I looked around and found that scikit learn has a joblib.dump that can replace pkl.dump in dump_pickle, and it doesn't use as much memory while writing out the files. I think you'll still need to separate the 224 and 299 processing, as mine was running out of 32G memory while doing a transpose.... too many copies of the same data going on. With joblib, memory use goes up to 27G, and no error. This could probably use a db, instead of all this image info in a dict.

[VShawn]

same problem on 24G RAM windows PC with python 3.6.6 and torch 1.1.0

===

I finished my job by following convert.py, thx to @jnorwood

this new convert.py will takes about 16Gb memory.

code

`

import os import numpy as np import tqdm from utee import misc import argparse import cv2 import joblib

imagenet_urls = [ 'http://ml.cs.tsinghua.edu.cn/~chenxi/dataset/val224_compressed.pkl' ] parser = argparse.ArgumentParser(description='Extract the ILSVRC2012 val dataset') parser.add_argument('--in_file', default='val224_compressed.pkl', help='input file path') parser.add_argument('--out_root', default='/tmp/public_dataset/pytorch/imagenet-data/', help='output file path') args = parser.parse_args()

d = misc.load_pickle(args.in_file) assert len(d['data']) == 50000, len(d['data']) assert len(d['target']) == 50000, len(d['target'])

''' conver val224.pkl ''' data = [] for img, target in tqdm.tqdm(zip(d['data'], d['target']), total=50000): img224 = misc.str2img(img) data.append(img224) data_dict = dict( data = np.array(data).transpose(0, 3, 1, 2), target = d['target'] ) if not os.path.exists(args.out_root): os.makedirs(args.out_root) ''' misc.dump_pickle(data_dict, os.path.join(args.out_root, 'val224.pkl'))''' joblib.dump(data_dict, os.path.join(args.out_root, 'val224.pkl')) data_dict.clear() data.clear() print('val224.pkl done.')

''' conver val229.pkl ''' data = [] for img, target in tqdm.tqdm(zip(d['data'], d['target']), total=50000): img224 = misc.str2img(img) img299 = cv2.resize(img224, (299, 299)) data.append(img299) data_dict = dict( data = np.array(data).transpose(0, 3, 1, 2), target = d['target'] )

if not os.path.exists(args.out_root): os.makedirs(args.out_root) ''' misc.dump_pickle(data_dict, os.path.join(args.out_root, 'val299.pkl')) ''' joblib.dump(data_dict, os.path.join(args.out_root, 'val299.pkl')) data_dict.clear() data.clear() print('val299.pkl done.')

`

result

Loading pickle object from val224_compressed.pkl => Done (1.0991 s) 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50000/50000 [01:02<00:00, 798.99it/s] val299.pkl done.

[aaron-xichen]

thanks @jnorwood, fixed, please check