Builder & mmap - Githubissues

alecthomas / mph

Minimal Perfect Hashing for Go

BSD 3-Clause "New" or "Revised" License

172 stars 23 forks source link

Builder & mmap #3

Open raichu opened 10 years ago

raichu commented 10 years ago

A major motivation for using mmap'ed files is whe the data-set does not fit in the memory. Currently, the builder uses a lot of memory.

It would be great to have a Builder that will work under these conditions.

alecthomas commented 10 years ago

You should be able to pass mmapped slices as the inputs to the builder. Does that help your use case? Not that there aren't other inefficiencies though.

gofrailo commented 9 years ago

@alecthomas - sorry it is not so clear to me what you mean. Could you give a small example? I have the same problem as @raichu where I run out of memory.

ceving commented 5 months ago

Would be nice to see how to build an index for the 35 GB of the Have I been Pwned database.