cylondata / twister2

A composable framework for fast and scalable data analytics
https://twister2.org
Apache License 2.0
57 stars 32 forks source link

Too many buffers being created in TCP mode #873

Open pulasthi opened 4 years ago

pulasthi commented 4 years ago

In the TCP mode when allocating send buffers we use the ByteBuffer.allocate method. When the graph is very large the amount of memory is being allocated becomes very large because buffer allocation is done for each communication operation in the graph . This leads to out of memory errors since heap space runs out.