search

dice-group / AGDISTIS

AGDISTIS - Agnostic Named Entity Disambiguation
http://aksw.org/Projects/AGDISTIS.html
GNU Affero General Public License v3.0
141 stars 37 forks source link

Each request allocates two additional threads #78

Closed danielsteinigen closed 3 years ago

danielsteinigen commented 3 years ago

Hi, we use AGDISTIS to process a series of requests and we see the behavior, that with each request two new threads are created, which are not closed after the request has been processed. This leads to a lot of resource allocations, so that we need to restart our application regulary. Has anyone ever noticed that behavior or has an idea where it might come from? Thanks.

RicardoUsbeck commented 3 years ago

I guess it happens in https://github.com/dice-group/AGDISTIS/blob/master/src/main/java/org/aksw/agdistis/webapp/GetDisambiguation.java For example, the input streams are never closed, but cannot investigate right now.

vdanielupb commented 3 years ago

I had a quick look over the code and I guess the problem might be, that the connection to the Lucene Index is newly loaded at each request. It is possible that the connection is not closed after the request. I have to further investigate this. The better solution is to use a singleton for the Lucene Index here.

DiegoMoussallem commented 3 years ago

Hi guys, Sabine Weber from The University of Edinburgh noticed reported a similar issue some time ago, it was affecting her algorithm by achieving a high memory consumption (the error message that it was unable to assign memory). However, I alleviated the problem by simply closing the Lucene connection, look at here https://github.com/dice-group/AGDISTIS/commit/2685cbef5de572e8dca1ba587e954e66d4f47bb9 . At that time, the problem was while making requests, AGDISTIS did not close the index from the previous request and always open a new instance.

Maybe it is related, but I also don't have time to investigate now

vdanielupb commented 3 years ago

Ah, yes this solves the problem partly. However you still have the problem in case of an exception. Because the index is not closed then and the method just throws the exception. https://github.com/dice-group/AGDISTIS/blob/4091bcc6bc8f9407740ea53bb95f7ea62673e7a2/src/main/java/org/aksw/agdistis/webapp/GetDisambiguation.java#L182 I don't know, if that is the only problem, but I will check it.

RicardoUsbeck commented 3 years ago

good catch!

danielsteinigen commented 3 years ago

I did some further investigations and it seems that the problem was partly caused by our own adaptions to the code, changing from Lucene to Elasticsearch Index, and partly in https://github.com/dice-group/AGDISTIS/blob/master/src/main/java/org/aksw/agdistis/webapp/GetDisambiguation.java, where "agdistis" is not closed for all cases, e.g. if text==null or type is not in [null, "agdistis", "nif", "candidates"]. Thanks for the support

DiegoMoussallem commented 3 years ago

Hi @DanielSteinigen, we have one Elasticsearch branch, you can make use of it, see https://github.com/dice-group/AGDISTIS/tree/elasticsearch_development.