search

vncorenlp / VnCoreNLP

A Vietnamese natural language processing toolkit (NAACL 2018)
Other
587 stars 145 forks source link

Does this library support thread-safety? #47

Open ducanh997 opened 1 year ago

ducanh997 commented 1 year ago

When calling the word_segment function from multiple threads, my service sometimes encounters the following exception:

"/usr/local/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator                                                                                                                     ││     yield fs.pop().result()                                                                                                                                                                                     ││   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 446, in result                                                                                                                              ││     return self.__get_result()                                                                                                                                                                                  ││   File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result                                                                                                                        ││     raise self._exception                                                                                                                                                                                       ││   File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run                                                                                                                                 │
│     result = self.fn(*self.args, **self.kwargs)                                                                                                                                                                 │
│   File "/usr/local/lib/python3.9/site-packages/py_vncorenlp/vncorenlp.py", line 95, in word_segment                                                                                                             │
│     self.model.annotate(annotation)                                                                                                                                                                             │
│   File "jnius/jnius_export_class.pxi", line 878, in jnius.JavaMethod.__call__                                                                                                                                   │
│   File "jnius/jnius_export_class.pxi", line 972, in jnius.JavaMethod.call_method                                                                                                                                │
│   File "jnius/jnius_utils.pxi", line 79, in jnius.check_exception                                                                                                                                               │
│ jnius.JavaException: JVM exception occurred: java.util.ConcurrentModificationException

Here is my code to call the word_segment function:

thread_pool_executor = ThreadPoolExecutor(max_workers=4)
rdrsegmenter = py_vncorenlp.VnCoreNLP(annotators=["wseg"])
segmented_contents = list(thread_pool_executor.map(rdrsegmenter.word_segment, contents))

So my question is whether this library is thread-safe or not. Note that this exception does not always occur, only occasionally.