search

ndmitchell / hoogle

Haskell API search engine
http://hoogle.haskell.org/
Other
753 stars 137 forks source link

Hoogle combine eats all my memory #2

Open nh2 opened 12 years ago

nh2 commented 12 years ago

I used the combine bash script to combine all .hoo files into one, but it doesn't complete because hoogle combine on those 3000 files loads them all strictly into memory, and 8GB is not enough for it.

Is it necessary for combine to load them all into memory or could we do it with lower/constant memory usage?

This would be cool to finally get to a point where hoogle can index all of Hackage, or at least all of my locally installed packages.

(I also think that it should be made clear somewhere in the docs that this doesn't work yet.)

jwiegley commented 11 years ago

@nh2 I wrote the "rehoo" utility to solve exactly this problem: http://hackage.haskell.org/package/rehoo

After installing, run: rehoo -j4 -c64 .

It must be run within Hoogle's databases directory (the same directory which contains default.hoo).

ndmitchell commented 9 years ago

Fixed in the dev version, which can combine all packages in little memory and very quickly.