I tried to match string with 1M rows dataset against 10M rows dataset. N Block I set 10:2000.
Error occured with message:
File \"\", line 164, in match_strings\n File \"\", line 623, in fit\n File \"\", line 461, in _fit_blockwise_manual\n File \"c:\program files\python38\lib\site-packages\topn\awesome_topn.py\", line 88, in awesome_hstack_topn\n r = np.concatenate([b.indptr for b in blocks])\n File \"<__array_function__ internals>\", line 180, in concatenate\nnumpy.core._exceptions._ArrayMemoryError: Unable to allocate 8.41 GiB for an array with shape (2258174000,) and data type int32\n<traceback object at 0x0000028A1BC113C0>.
I tried to match string with 1M rows dataset against 10M rows dataset. N Block I set 10:2000. Error occured with message: File \"\", line 164, in match_strings\n File \"\", line 623, in fit\n File \"\", line 461, in _fit_blockwise_manual\n File \"c:\program files\python38\lib\site-packages\topn\awesome_topn.py\", line 88, in awesome_hstack_topn\n r = np.concatenate([b.indptr for b in blocks])\n File \"<__array_function__ internals>\", line 180, in concatenate\nnumpy.core._exceptions._ArrayMemoryError: Unable to allocate 8.41 GiB for an array with shape (2258174000,) and data type int32\n<traceback object at 0x0000028A1BC113C0>.
Any solutions?