KeyError: 'datadirectories'

[ashishcse0031]

`data_dir = "/home/cse31/MalReserach/data/ember/"

ember.create_vectorized_features(datadir) = ember.create_metadata(data_dir) ` Vectorizing training set 0%| | 0/900000 [00:00<?, ?it/s]

RemoteTraceback Traceback (most recent call last) RemoteTraceback: """ Traceback (most recent call last): File "/home/cse31/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, *kwds)) File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/init.py", line 44, in vectorize_unpack return vectorize(args) File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/init.py", line 31, in vectorize feature_vector = extractor.process_raw_features(raw_features) File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/features.py", line 522, in process_raw_features feature_vectors = [fe.process_raw_features(raw_obj[fe.name]) for fe in self.features] File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/features.py", line 522, in feature_vectors = [fe.process_raw_features(raw_obj[fe.name]) for fe in self.features] KeyError: 'datadirectories' """

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)

in 1 data_dir = "/home/cse31/MalReserach/data/ember/" # change this to where you unzipped the download 2 ----> 3 ember.create_vectorized_features(data_dir) 4 _ = ember.create_metadata(data_dir) ~/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/__init__.py in create_vectorized_features(data_dir, feature_version) 73 raw_feature_paths = [os.path.join(data_dir, "train_features_{}.jsonl".format(i)) for i in range(6)] 74 nrows = sum([1 for fp in raw_feature_paths for line in open(fp)]) ---> 75 vectorize_subset(X_path, y_path, raw_feature_paths, extractor, nrows) 76 77 print("Vectorizing test set") ~/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/__init__.py in vectorize_subset(X_path, y_path, raw_feature_paths, extractor, nrows) 58 argument_iterator = ((irow, raw_features_string, X_path, y_path, extractor, nrows) 59 for irow, raw_features_string in enumerate(raw_feature_iterator(raw_feature_paths))) ---> 60 for _ in tqdm.tqdm(pool.imap_unordered(vectorize_unpack, argument_iterator), total=nrows): 61 pass 62 ~/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py in __iter__(self) 1003 """), fp_write=getattr(self.fp, 'write', sys.stderr.write)) 1004 -> 1005 for obj in iterable: 1006 yield obj 1007 # Update and possibly print the progressbar. ~/anaconda3/lib/python3.7/multiprocessing/pool.py in next(self, timeout) 746 if success: 747 return value --> 748 raise value 749 750 __next__ = next # XXX KeyError: 'datadirectories'

[mrphilroth]

Are you attempting to run on the original dataset release containing samples from 2017? If so, then you'll need to specify the feature version number (in this case 1) when you vectorize the features:

ember.create_vectorized_features(data_dir, 1)

If you're working on the most recent dataset release, then there's definitely a bug I'll have to track down.

[ashishcse0031]

I was working with 2017 dataset, and it solved the issue.