Optimize thread pools to spare resource consumption

[amitjoy]

Is your feature request related to a problem? Please describe. A leak has been encountered in which a scheduled thread pool comprises > 400.000 tasks and > 300.000 tasks were in cancelled state but still remained in the thread pool. After analyzing the heapdump, it is recommended to optimize the thread pools efficiently by introducing several thread pool executor flags.

Describe the solution you'd like The following flags should be set to better optimize an executor:

• corePoolSize - to be set to a minimum number of threads which will be always available even when there are no tasks assigned to the thread pool (for example, 1 as core pool size would also work efficiently)
• maxPoolSize - to be set to the maximum number of threads to be created on demand when tasks need to be executed
• keepAliveTime - to be set to enable coreThreadTimeOut for terminating threads if no tasks arrive within this keep-alive time
• removeOnCancelPolicy - to be set for removing tasks from ScheduledThreadPoolExecutor to remove the allocated threads when they are cancelled

Describe alternatives you've considered Usual way of creating threads out of a thread pool is not optimized for memory constrained devices, hence, there is no other alternative than configuring the aforementioned options for thread pools

Additional context The changes are required to be introduced in ZigBeeExecutors

[amitjoy]

FYI @cdjackson @TomTravaglino

[ViToni]

Thanks @amitjoy for raising this issue. After a bit of research I found one potential cause, which might be resulting in the issue observed. The ScheduledExecutors created by ZigBeeExecutors use the default ScheduledExecutor which does not remove on cancel by default, see here. Maybe setting this policy would already solve the issue.

[amitjoy]

@ViToni Yeah, I have already specified this policy in the issue description as well. This will certainly be useful to spare memory consumption to a greater extent.

[ViToni]

I have a theorie what happened:

1. Before #1381 there was a somehow unconditinal creation of a future task in https://github.com/zsmartsystems/com.zsmartsystems.zigbee/blob/ef09ca1b772bf9b4f3154de534ab3c1c7290ea8c/com.zsmartsystems.zigbee/src/main/java/com/zsmartsystems/zigbee/transaction/ZigBeeTransactionManager.java#L696-L707 when the queues were empty.
2. This future task was scheduled with a delay of Long.MAX_VALUE millis.
3. On the next send the task would have been cancelled and a new one created.
4. Go to 1.

As ScheduledThreadPoolExecutor retains cancelled tasks until they would be due to avoid potential expensive clean up operations of the backlog, the cancelled task would linger there "forever".

The change #1381 avoids creating tasks when all transaction queue are empty, but the ScheduledThreadPoolExecutors still retain cancelled tasks. Hopefully that's fixed by #1393.

[cdjackson]

Thanks @amitjoy and also @ViToni for the PR. Hopefully that will fix this issue...