Open rju opened 3 days ago
author André van Hoorn -- Fri, 3 Jul 2015 14:26:39 +0200
The following code is included in the AbstractAsyncWriter's newMonitoringRecord method:
case 1: // blocks when queue full for (int i = 0; i < 10; i++) { // drop out if more than 10 times interrupted try { this.blockingQueue.put(monitoringRecord); return true; } catch (final InterruptedException ignore) { LOG.warn("Interupted when adding new monitoring record to queue. Try: " + i); Thread.currentThread().interrupt(); // propagate interrupt } } LOG.error("Failed to add new monitoring record to queue (Finally interruped while blocked)."); return false;
Assumption: this way, the interrupt flag is not reset, leading to the continuous interruption of subsequent calls to put.
Teerat will try whether the following change resolves the issue:
case 1: // blocks when queue full for (int i = 0; i < 10; i++) { // drop out if more than 10 times interrupted try { this.blockingQueue.put(monitoringRecord); return true; } catch (final InterruptedException ignore) { if(Thread.currentThread().interrupted()) { // test and reset thread's interrupted status LOG.warn("Interupted when adding new monitoring record to queue. Try: " + i); } } } LOG.error("Failed to add new monitoring record to queue (Finally interruped while blocked)."); return false;
See http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.html#interrupted%28%29
author teeratpitakrat -- Sun, 5 Jul 2015 19:02:28 +0200
Replying to [avh|comment:1]:
> The following code is included in the AbstractAsyncWriter's newMonitoringRecord method:
Moved to https://kieker-monitoring.atlassian.net/ticket/1662 as it is a separate issue.
author teeratpitakrat -- Mon, 6 Jul 2015 14:50:12 +0200
One reason for the writer not shutting down is because the interrupt flag of AbstractAsyncWriter is not reset https://kieker-monitoring.atlassian.net/ticket/1662. This causes an InterruptedException in the put method of AbstractAsyncThread's initShutdown while adding end-marker to the queue. As a consequence, the AbstractAsyncThread never receives the end-marker and does not shutdown. Fixing https://kieker-monitoring.atlassian.net/ticket/1662 should prevent this case from occurring.
However, there might be another rare circumstance, in which an interrupt occurs while adding an end-marker to the queue in AbstractAsyncThread's initShutdown. This may lead to the same problem that the writer does not receive the end-marker and does not terminate.
One possible fix for this would be to periodically check for the shutdown signal (i.e. shutdownLatch) in AbstractAsyncThread's run. For example, changing the following code:
Override public final void run() { if (LOG.isDebugEnabled()) { LOG.debug(this.getClass().getName() + " running"); } try { // making it a local variable for faster access final BlockingQueue<IMonitoringRecord> writeQueueLocal = this.writeQueue; while (true) { try { IMonitoringRecord monitoringRecord = writeQueueLocal.take(); if (monitoringRecord == END_OF_MONITORING_MARKER) { // NOPMD (CompareObjectsWithEquals ...
to
Override public final void run() { if (LOG.isDebugEnabled()) { LOG.debug(this.getClass().getName() + " running"); } try { // making it a local variable for faster access final BlockingQueue<IMonitoringRecord> writeQueueLocal = this.writeQueue; while (true) { try { IMonitoringRecord monitoringRecord = writeQueueLocal.poll(5, TimeUnit.SECONDS); if (monitoringRecord != null) { if (monitoringRecord == END_OF_MONITORING_MARKER) { // NOPMD (CompareObjectsWithEquals ... } else { // Check for shutdown signal }
JIRA Issue: KIEKER-1367 (Async FS) writer in blocking mode tries to shutdown but fails Original Reporter: Andre van Hoorn
Writer configured to use Async FS with blocking queue full behavior (mode 1) and a queue size of 100000. During monitoring, the writer tries to re-add records to the queue 10 times and then initiates the shutdown. However, monitoring just continues.
Some excerpts from the log. The full log is attached. Thanks to Teerat for reporting this issue and for providing the log.