search

kermitt2 / biblio-glutton

A high performance bibliographic information service: https://biblio-glutton.readthedocs.io
117 stars 15 forks source link

Increased lmdb max size to 1TB #86

Closed karatekaneen closed 1 year ago

karatekaneen commented 1 year ago

This solved issue #85 for me and stopped the LMDB to cap out at 300gb. Don't know if this has any side effects when using on smaller disks or if there's any performance penalty

kermitt2 commented 1 year ago

Hi @karatekaneen !

Sorry for being very slow with biblio-glutton currently - I am planning an update and improvements since quite a while, but could not find enough time to focus again on it in the last months.

In branch https://github.com/kermitt2/biblio-glutton/tree/version-0.3 - which is the most advanced, the lmdb max size has indeed already been increased to 1TB, which is necessary for latest Crossref dump versions.

https://github.com/kermitt2/biblio-glutton/blob/version-0.3/lookup/src/main/java/com/scienceminer/lookup/storage/StorageEnvFactory.java#L32

karatekaneen commented 1 year ago

No need to apologize. Thank you for the work you put in!

Nice that you found the same issue. Feel free to merge or close this as you see fit. For me it's not that important since my fork is working for us, but it might help someone else while you're working on 0.3

kermitt2 commented 1 year ago

You're right, let's merge it. Thanks for the PR in any cases !