gerritjvv / bigstreams

bigstreams big data kafka hadoop and file based imports
Eclipse Public License 1.0
3 stars 3 forks source link

NIOClientSocketChannelFactory must be resused - Collector Coordination service and Agent #19

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago
The NIOClientSocketChannelFactory must be a singleton and should be reused.
Not doing so would would lead to Direct Memory Errors.

The NIOClientSocketChannelFactory will pre create as many NIOWorker instances 
as processors * a constant. Each NIOWorker pre allocates 64KB of direct memory. 
On a 16 processor machine this means 64K * 16  = 256K per instance of 
NIOClientSocketChannelFactory.

Creating an instance on each client connect for the CoordinationService means 
512K per send lock and send unlock combination. 

The ClientConnectionResource class in the commons and the agent client 
connection will be changed to accept an instance of 
NIOClientSocketChannelFactory in the Constructor.

Original issue reported on code.google.com by gerritjvv@gmail.com on 4 Oct 2010 at 8:19

GoogleCodeExporter commented 8 years ago
The CoordinationServiceClientImpl now creates 3 single instances of
-> ExecutorService (Cached Thread Pool Service ) 1 for the worker boss
-> ExecutorService (Cached Thread Pool Service ) 1 for the workers
-> NioClientSocketChannelFactory using the above Executor(s)

Original comment by gerritjvv@gmail.com on 4 Oct 2010 at 8:28

GoogleCodeExporter commented 8 years ago

Original comment by gerritjvv@gmail.com on 4 Oct 2010 at 11:35