google-code-export / dkpro-core-asl

Automatically exported from code.google.com/p/dkpro-core-asl
0 stars 0 forks source link

Proper sharing of models in a multi-threaded environment #557

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
The original use case was to share the model between multiple instances of a 
parse in the same pipeline, but using a single thread - that works.

In your multi-threaded environment, the sharing doesn't kick in, because the 
threads start loading the model in parallel.
Say you have 2 threads.

time.1) thread 1 looks if a cached model is present
time.1) thread 2 looks if a cached model is present
time.2) thread 1 starts loading the model because it did not find a cached 
version
time.2) thread 2 starts loading the model because it did not find a cached 
version
time.3) thread 1 completes loading the model and adds it to the cache
time.3) thread 2 completes loading the model and adds it to the cache

In order for this to work as expected in a multi-threaded environment, thread 1
would have to set a lock on the model until it completes loading it. Thread 2
would then (hopefully) see that lock and wait until the lock is released. Then
it would check the cache and find the already loaded model.

This goes beyond the original use-case and isn't implemented yet.

Original discussion including sample code for testing: 
https://groups.google.com/d/topic/dkpro-core-user/lHvIN4VfUKk/discussion

Original issue reported on code.google.com by richard.eckart on 7 Dec 2014 at 4:08