search

dice-group / WHALE

0 stars 0 forks source link

Use dictionary to create indexing and save after a memory threshold #8

Closed sshivam95 closed 3 months ago

sshivam95 commented 3 months ago

An extension to the concept used in #7. Instead of saving the dictionaries to memory mapped files one line after the other like in issue #5, Save the dictionary once it reaches a threshold of, e.g., 90% of main memory allocated.

sshivam95 commented 3 months ago

One implementation can be found here

sshivam95 commented 3 months ago

Update: Closing this because of issue #9, reopen if needed for a bigger dataset.