search

shriphani / pegasus

:racehorse:✈️ Pegasus is a scalable, modular, polite web-crawler for Clojure
http://getpegasus.io
Eclipse Public License 1.0
262 stars 17 forks source link

MDB overflows - set up data-structures based on data-size #27

Closed shriphani closed 8 years ago

shriphani commented 8 years ago

Set up LMDB with a large-enough mmap and then expand it when needed mid-crawl.

shriphani commented 8 years ago

or just make it really large.

shriphani commented 8 years ago

fixed in # #28