search

spotify / annoy-java

Approximate nearest neighbors in Java
Apache License 2.0
138 stars 46 forks source link

Only supports files up to 2GB #1

Open a1k0n opened 9 years ago

a1k0n commented 9 years ago

Because the API for MappedByteBuffer, or whatever it's called, uses Java int for offsets, you cannot address past 2GB. To work around this, annoy-java would have to open multiple windows into the file as separate buffers 2gb apart.

If there is a need for this, please comment here.

joprice commented 8 years ago

This would be nice to have.

flylo commented 7 years ago

+1 that this would be nice to have

Yuanrui1994 commented 7 years ago

Hi! I am using this library now. I am curious whether files larger than 2GB can be supported now since my index file is around 100GB. Thanks.

julsal commented 7 years ago

Dear @Yuanrui1994, I've just submitted a PR with a solution to this restriction. It ran successfully in my index (3GB). It would be nice if you also execute some tests in your index of 100GB.