search

opensemanticsearch / open-semantic-search

Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)
https://opensemanticsearch.org
GNU General Public License v3.0
979 stars 169 forks source link

open-semantic-search-vm_21.01.17.ova limits.conf #441

Closed NetwarSystem closed 2 years ago

NetwarSystem commented 2 years ago

I have a small herd of OSS appliances that are running under VirtualBox. I imported the OVA once, doctored it to have current software, and then I use Linked Clones, so each instance only has its unique data, not the full system.

I have long had an issue with Solr starting slowly. I had one get stuck the other day, went to manually start it, and it complained about the 1k file handles limit. Boosting that limit seems to make it start more smoothly.

I created one VM with the default /etc/security/limits.conf, which is full of comments but has no configuration at all, and another with the generous file handle and process limits I use for Elasticsearch.

I have a directory with 40k+ files in it, launched an index operation on the default machine and the one with higher limits. Load average, memory usage, and progress are essentially identical. The only benefit thus far has been a smoother start of Solr. Perhaps there would be notable differences at larger scale, but I'm not going to keep the test running.

NetwarSystem commented 2 years ago

No need to leave this open, just wanted to share the experience.