owenzhang / kaggle-avazu

2nd place solution for Avazu click-through rate prediction competition
Other
437 stars 254 forks source link

out of mem when run 'python _1_encode_cat_features.py' on 64GB machine #2

Open SeekPoint opened 7 years ago

SeekPoint commented 7 years ago

site_model site_model 24 3335302 3335302 site_model site_model 25 3363122 3363122 site_model site_model 26 3835892 3835892 site_model site_model 27 3225010 3225010 site_model site_model 28 5287222 5287222 site_model site_model 29 3832608 3832608 site_model site_model 30 4218938 4218938 site_model site_model 31 4577464 4577464 app_model app_model 22 5337126 5337126 app_model app_model 23 3870752 3870752 app_model app_model 24 3335302 3335302 app_model app_model 25 3363122 3363122 app_model app_model 26 3835892 3835892 app_model app_model 27 3225010 3225010 app_model app_model 28 5287222 5287222 app_model app_model 29 3832608 3832608 app_model app_model 30 4218938 4218938 app_model app_model 31 4577464 4577464 dev_id_ip dev_id_ip 22 5337126 5337126 Traceback (most recent call last): File "_1_encode_cat_features.py", line 56, in calc_exptv(t0, exptv_vn_list) File "/media/rzai/ai_data/prj/kaggle-avazu-2nd/utils.py", line 499, in calc_exptv t0.loc[t0.day.values == day_v, vn_exp]=day_exps[day_v][vn_key]['exp'] File "/usr/local/lib/python2.7/dist-packages/pandas/core/indexing.py", line 118, in setitem self._setitem_with_indexer(indexer, value) File "/usr/local/lib/python2.7/dist-packages/pandas/core/indexing.py", line 210, in _setitem_with_indexer take_split_path = self.obj._is_mixed_type File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 2054, in _is_mixed_type return self._protect_consolidate(f) File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 2020, in _protect_consolidate result = f() File "/usr/local/lib/python2.7/dist-packages/pandas/core/generic.py", line 2053, in f = lambda: self._data.is_mixed_type File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 2568, in is_mixed_type self._consolidate_inplace() File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 2830, in _consolidate_inplace self.blocks = tuple(_consolidate(self.blocks)) File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 3799, in _consolidate _can_consolidate=_can_consolidate) File "/usr/local/lib/python2.7/dist-packages/pandas/core/internals.py", line 3825, in _merge_blocks new_values = new_values[argsort] MemoryError