Task appAfterIntegrationTest never finish the servlet (jetty9 or tomcat9), requesting to cassandra

[jonatanloya]

After execute cucumber bdds, the gradle task hangs

20 Scenarios (20 passed) 166 Steps (166 passed) 0m21.165s

:tourop-services:cucumber (Thread[main,5,main]) completed. Took 23.965 secs. :tourop-services:cucumberTouropIntegrationTest (Thread[main,5,main]) started. :tourop-services:cucumberTouropIntegrationTest Skipping task ':tourop-services:cucumberTouropIntegrationTest' as it has no actions. :tourop-services:cucumberTouropIntegrationTest (Thread[main,5,main]) completed. Took 0.0 secs. :tourop-services:appAfterIntegrationTest (Thread[main,5,main]) started. :tourop-services:appAfterIntegrationTest Putting task artifact state for task ':tourop-services:appAfterIntegrationTest' into context took 0.0 secs. Executing task ':tourop-services:appAfterIntegrationTest' (up-to-date check took 0.0 secs) due to: Task has not declared any outputs. 16:34:11 INFO Closing Spring root WebApplicationContext

Building 56% > :tourop-services:appAfterIntegrationTest

I will atach the build script,

main build script [ apply plugin: 'java' apply plugin: 'maven'

apply plugin: 'org.akhikhl.gretty' apply plugin: 'war' apply plugin: 'jacoco'

import org.akhikhl.gretty.AppBeforeIntegrationTestTask import org.akhikhl.gretty.AppAfterIntegrationTestTask

dependencies {

//compile 'com.fasterxml.jackson.core:jackson-core:2.1.1'
//compile 'com.fasterxml.jackson.core:jackson-databind:2.1.1'
//compile 'com.fasterxml.jackson.core:jackson-annotations:2.1.1'
compile 'com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:2.1.1'

compile 'javax.inject:javax.inject:1'
compile 'com.sun.jersey:jersey-server:1.15'
compile 'com.sun.jersey.contribs:jersey-spring:1.15'
compile 'com.google.guava:guava:18.0'
compile 'joda-time:joda-time:1.5.1'
compile 'com.datastax.cassandra:cassandra-driver-mapping:2.1.3'
compile 'org.projectlombok:lombok:1.12.6'
compile 'org.springframework:spring-web:4.2.4.RELEASE'
compile 'net.sf.opencsv:opencsv:2.3'

}

artifacts { compile zipDeploy archives zipDeploy }

buildscript { repositories { mavenCentral() } dependencies { // classpath 'org.akhikhl.gretty:gretty:+'

    classpath 'org.akhikhl.gretty:gretty:1.4.1'
    classpath 'org.akhikhl.gretty:gretty-runner-tomcat8:1.4.1'
    //classpath 'org.akhikhl.gretty:gretty-runner-jetty9:1.4.1'
}

}

repositories { mavenCentral() }

task ('cucumberTouropIntegrationTest'){ println 'Copying web app from tourop-services/src/test/webapp to tourop-services/src/main/webapp.' copy{ from 'src/test/webapp' into 'src/main/webapp' } }

gretty {

  //println 'RANDOM PORT: ' + bddPort
  httpPort = bddPort
  contextPath = '/'
  servletContainer = 'tomcat8' //tomcat7 or tomcat8
  logDir='build/log'
  logFileName='grettyLog.log'
  inplaceMode="hard"
  integrationTestTask = 'cucumberTouropIntegrationTest'

  jvmArgs= [
            "-Dprosrm.install.home=${buildDir.toString()}",
            "-Doandd.cache.host=${project.property('oandd.cache.host')}",
            "-Doandd.cache.cqlport=${project.property('oandd.cache.cqlport')}"
    ]

afterEvaluate {

tasks.appBeforeIntegrationTest.jacoco {
  append = false
  destinationFile = new File(project.buildDir, 'jacoco/integrationTest_server.exec')
}
tasks.appBeforeIntegrationTest.finalizedBy tasks.cucumber
tasks.appAfterIntegrationTest.finalizedBy tasks.integrationTestServerReport

} }

task('integrationTestServerReport', type: JacocoReport) {

executionData { tasks.appBeforeIntegrationTest.jacoco.destinationFile }

sourceDirectories = project.files(project.sourceSets.main.allSource.srcDirs) classDirectories = project.sourceSets.main.output

def reportDir = project.reporting.file("jacoco/integrationTest_server/html") reports { html.destination = reportDir } doLast { println "Jacoco report for server created: file://${reportDir.toURI().path}" } }

]

cucumber task

task cucumber {

doLast {
    System.setProperty('prosrm.install.home', buildDir.toString())
    javaexec {

        systemProperties = ['test.server.http.port': bddPort]
        if (project.hasProperty('oandd.cache.host')) {
            systemProperties.put('oandd.cache.host', project.property('oandd.cache.host'));
        }
        if (project.hasProperty('oandd.cache.cqlport')) {
            systemProperties.put('oandd.cache.cqlport', project.property('oandd.cache.cqlport'));
        }
        main = 'cucumber.api.cli.Main'
        classpath = sourceSets.main.output +
                    sourceSets.test.output +
                    configurations.testRuntime
        args = cucumberArgs()
    }
}

}

List cucumberArgs() { // Boilerplate args def args = [ '--format', 'junit:build/cucumber-reports/junit/report.xml', '--format', 'html:build/reports/cucumber', '-f', 'pretty', '--glue', 'com.prosrm.rms.bdd']

// Add the tags arguments if we have any.
if (project.hasProperty('tags')) {
    def tagLists = tags.split(':')
    for (tagList in tagLists) {
        args.add('--tags')
        args.add(tagList)
    }
}

// Feature locations
args.add('src/test/resources/features')

return args

}

[dpranantha]

I have the same problem with running postman integration test within gretty integration test. It is stuck right after closing spring root webapplicationcontext when running appAfterIntegrationTest.

myService-postman:appAfterIntegrationTest 10:32:51 INFO Closing Spring root WebApplicationContext

Building 99% > :myService-postman:appAfterIntegrationTest

my build.gradle is as follows.

apply from: 'docker.gradle' apply from: 'postman.gradle' apply plugin: 'java' apply plugin: 'war' apply plugin: 'org.akhikhl.gretty'

configurations { all.exclude group: 'javaee' all.exclude group:'javassist', module: 'javassist' all.exclude group: 'org.mortbay.jetty', module: 'servlet-api' all.exclude group: 'org.mortbay.jetty', module: 'jetty' all*.exclude group: 'org.mortbay.jetty', module: 'jetty-util' }

dependencies { compile 'org.springframework:spring-core' compile 'org.springframework:spring-context' compile 'org.springframework:spring-beans' compile 'org.springframework:spring-web'

compile 'net.minidev:json-smart:2.2.1'
compile 'commons-io:commons-io:2.5'

compile 'com.philemonworks:selfdiagnose'

testCompile 'org.springframework:spring-test'
testCompile 'net.serenity-bdd:serenity-core'
testCompile 'net.serenity-bdd:serenity-junit'
testCompile 'net.serenity-bdd:serenity-spring'

testCompile 'com.moowork.gradle:gradle-node-plugin'
testCompile 'gradle.plugin.de.infonautika.postman:postman-runner'

testCompile 'cglib:cglib-nodep'
testCompile 'com.google.guava:guava'

testRuntime 'ch.qos.logback:logback-classic'

}

task runPostman(type: PostmanTask) { println 'running postman' }

gretty { integrationTestTask = 'integrationTest' contextPath = '' httpPort = 7777 servletContainer = 'jetty9' }

task integrationTest << { }

project.afterEvaluate { tasks.integrationTest.dependsOn runPostman }

[jonatanloya]

Can you change the servlet to tomcat9? This is because tomcat shows the threads that servlet creates and I think that is the main problem of the task afterIntegrationTask. Some threads stay alive and gretty is unable to stop them.

Please share your entire.configuration.

[dpranantha]

good one @jonatanloya, I changed it to tomcat8, it printed out all threads where gretty unable to stop. First, it failed to stop hikari connection pool thread, and the PG jdbc thread

14:24:28 WARN The web application [ROOT] appears to have started a thread named [Hikari housekeeper (pool HikariCP)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) 14:24:28 WARN The web application [ROOT] appears to have started a thread named [PostgreSQL-JDBC-SharedTimer-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) 14:24:28 WARN The web application [ROOT] appears to have started a thread named [com.bol.statsd.writer.CooperativeStatsWriter-flush-timer] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) 14:24:28 WARN The web application [ROOT] appears to have started a thread named [New I/O worker #1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68) org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)

I shared configuration above, (apart from docker.gradle which I ignored for now to spin up postgres db) and also postman.gradle to run postman as follows.

`apply plugin: 'de.infonautika.postman'

postman { collections = fileTree(dir: 'src/test/resources', include: '*/.postman_collection*') environment = file('src/test/resources/TEST.postman_environment.json') stopOnError = false cliReport = true disableUnicode = false xmlReportDir = 'build/testoutput' htmlReportDir = 'build/testoutput/html' jsonReportDir = 'build/testoutput/json' }`

and these are the properties deployed

Http REST Service

main.host=localhost main.port=7777

Database

database.logschema=public database.url=jdbc:postgresql://localhost:5432/tom jpa.show_sql=false hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect

and web.xml

Archetype Created Web Application contextConfigLocation classpath:/appContext.xml org.springframework.web.context.ContextLoaderListener app-web com.sun.jersey.spi.spring.container.servlet.SpringServlet com.sun.jersey.config.property.resourceConfigClass com.sun.jersey.api.core.PackagesResourceConfig com.sun.jersey.config.property.packages com.bol.service.travelobjectmap.server.endpoint.v1.test.rest 1 app-web /*

[jonatanloya]

@dpranantha, It looks like is a gretty bug that is not able to stop threads created by the web apps over the servlet.

Previously that Jetty was working fine, this strated when I upgraded to Gretty, same web app, same cucumber bdds project.

@akhikhl do you think gretty need to improve a forced stopper?

I think this should be a good improvement for gretty.

[akhikhl]

Sorry, no, Gretty will not implement forced stopper. This is just the wrong way to do so. My recommendation for graceful shutdown:

1. Put the following code to your web.xml:

<listener>
  <listener-class>mypackage.MyContextListener</listener-class>
</listener>

you will have to substitute your package and class name, of course.

2. Create a class:

package mypackage;

import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

public class MyContextListener implements ServletContextListener {
  public void contextInitialized(ServletContextEvent servletContextEvent) {
  }
  public void contextDestroyed(ServletContextEvent servletContextEvent) {
    // set shutdown flag here
  }
}

3. Check for shutdown flag in application threads. As soon as shutdown flag is true, interrupt thread loops and exit threads.

If threads are out of your control, then please feel free implementing "force stopper" in contextDestroyed.