Constantly getting Couldn't start appServer tomcat on AuctionWeb1 in Docker container deployment

[mafzal786]

I'm getting this randomly on different container Couldn't start appServer tomcat on AuctionWeb1 sometimes Couldn't start appServer tomcat on AuctionCm1 Couldn't start appServer tomcat on AuctionCs1 or Couldn't start appServer tomcat on AuctionDriver1

After looking into logs files in /ouput/9/logs/appServer/AuctionWeb1/AuctionApp1D/Tom....log

I see the following:

12-Jul-2017 19:22:34.712 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener] org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in ServletContext resource [/WEB-INF/spring/jpa-context.xml]: Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [jdbc/auction] is not bound in this Context. Unable to find [jdbc]. at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1531) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1276) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1081) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:856) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1419) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1409) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource': Invocation of init method failed; nested exception is javax.naming.NameNotFoundException: Name [jdbc/auction] is not bound in this Context. Unable to find [jdbc]. at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) ... 24 more Caused by: javax.naming.NameNotFoundException: Name [jdbc/auction] is not bound in this Context. Unable to find [jdbc]. at org.apache.naming.NamingContext.lookup(NamingContext.java:816) at org.apache.naming.NamingContext.lookup(NamingContext.java:173) at org.apache.naming.SelectorContext.lookup(SelectorContext.java:163) at javax.naming.InitialContext.lookup(InitialContext.java:417) at org.springframework.jndi.JndiTemplate$1.doInContext(JndiTemplate.java:155) at org.springframework.jndi.JndiTemplate.execute(JndiTemplate.java:87) at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:152) at org.springframework.jndi.JndiTemplate.lookup(JndiTemplate.java:179) at org.springframework.jndi.JndiLocatorSupport.lookup(JndiLocatorSupport.java:104) at org.springframework.jndi.JndiObjectLocator.lookup(JndiObjectLocator.java:106) at org.springframework.jndi.JndiObjectFactoryBean.lookupWithFallback(JndiObjectFactoryBean.java:231) at org.springframework.jndi.JndiObjectFactoryBean.afterPropertiesSet(JndiObjectFactoryBean.java:217) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624) ... 31 more

12-Jul-2017 19:22:50.370 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext 12-Jul-2017 19:32:38.578 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Closing Spring root WebApplicationContext 113

[hrosenbe]

Can you please attach the weathervane.config file that you are using for the run. That will help me understand what is happening a little better.

[mafzal786]

{ "users" : 60, "maxUsers" : 60,

"description" : "Simple Weathervane Config.", "runLength" : "medium",

"numWorkloads" : 1, "numAppInstances" : 1, "numDrivers" : 1, "numConfigurationManagers" : 1,

"numElasticityServices" : 0,

"numCoordinationServers" : 1,

"numLbServers" : 0,

"numWebServers" : 1, "numAppServers" : 1, "numMsgServers" : 1,

"numFileServers" : 0,

"numNosqlServers" : 1,

"nosqlSharded" : false,

"nosqlReplicated" : false,

"numDbServers" : 1,

"logLevel" : 0,

"runStrategy" : "single",

"powerOnVms" : false,

"powerOffVms" : false,

"numViHosts" : 0,

"useDocker" : true, "dockerNamespace" : "xxxxx",

"dockerHostPin" : false,

"dockerHostPinMode" : "all",

"dockerCpuShares" : n/a,

"dockerCpuSetCpus" : n/a,

"dockerCpuSetMems" : n/a,

"dockerMemory" : n/a,

"dockerMemorySwap" : n/a,

"dockerNet" : "bridge",

"dockerRegistryName" : "vmc-perf-h11",

}

[hrosenbe]

Are you running everything on one VM or Docker host? I'm not sure why the hostname for which you get the error message keeps changing.

Can you create a tar-file from the output directory and attach it to the bug? It will be alot quicker than me asking for one file at a time.

The command would be:

tar cvf 9.tgz /root/weathervane/output/9

Then attach the file 9.tgz that is created.

[mafzal786]

it is docker bare-metal host. log is attached. it shows jdbc connection issues.

9.zip

[hrosenbe]

It looks as though the jdbc issue is caused by this error:

java.sql.SQLException: Unable to load class: org.postgresql.Driver

If the build of the Docker images went properly, the jar file for the driver class should be in the image in the proper place.

Can you look in the file buildDockerImages.log for errors. This file is created when running the buildDockerImages.pl script, and should be in /root/weathervane.

[mafzal786]

tried re-creating tomcat image with the Dockerfile. I'm getting the following in the end

cp: cannot create regular file '/opt/apache-tomcat-auction1/bin/': Not a directory

[mafzal786]

look like this line is failing in creating docker image

cp /opt/apache-tomcat/bin/tomcat-juli.jar /opt/apache-tomcat-auction1/bin/

[mafzal786]

On the side note, in zookeeper dockerfile

curl -s http://www.us.apache.org/dist/zookeeper/stable/zookeeper-3.4.9.tar.gz -o zookeeper-3.4.9.tar.gz && \

3.4.9 does not exist. it needs to be changed to 3.4.10. That curl link is invalid. Can you please verify.

Thanks

[hrosenbe]

You are right about the version of Zookeeper.

I have fixed the Dockerfiles and the buildDockerImages.pl script to always get the most recent available version of Tomcat and Zookeeper. The changes were pushed to the master branch in this commit: https://github.com/vmware/weathervane/commit/7819be393d748bf822dd23c046e23aaa0f312b28

If you get Weathervane by cloning the repository, you should be able to get the latest versions by doing a git pull. If you deployed a release package, you can get the changed files from that commit link, and copy them over the respective files in your deployment.

Then try build the docker images again and hopefully everything will work.

[mafzal786]

Version patch did not help!!! tomcat dockerfile failed as tar file failed to download. I changed the Dockerfile to use 8.5.16 in the url than the TOMCAT_VERSION variable, and it worked. but later it failed on the same cp command in Dockerfile as follows

cp Command: cp /opt/apache-tomcat/bin/tomcat-juli.jar /opt/apache-tomcat-auction1/bin/ && \

Error as below. it does not find the /bin directory cp: cannot create regular file '/opt/apache-tomcat-auction1/bin/': Not a directory

[hrosenbe]

Can you please post the entire buildDockerImages.log file. I am not sure why it is not working.

[hrosenbe]

Never mind. I just discovered that there is a bug in the updated script. I will fix it and let you know when I have pushed up the fix.

[hrosenbe]

There was a problem with the way that I was referencing the Tomcat version number in the dockerfile. I have fixed this in commit https://github.com/vmware/weathervane/commit/e057cf62b72707efde9cd92790983fa370dd38d6.

Can you please take the file dockerImages/tomcat/Dockerfile from that commit, use it to replace the file /root/weathervane/dockerImages/tomcat/Dockerfile on your system, and rety building the Docker images.

[mafzal786]

now download worked fine for tomcat, but the real issue is not resolved. still seeing this cp command failed.

[hrosenbe]

Can you please attach the entire buildDockerImages.log file

[mafzal786]

attached buildDockerImages.log.zip

[hrosenbe]

Can you please confirm for me that there isn't something missing from your Weathervane installation.

Under the directory /root/weathervane/dockerImages/tomcat/apache-tomcat-auction1, there should be three subdirectories:

• bin
• conf
• lib All of these directories should contain at least one file.

[mafzal786]

i see only conf and webapps folder. no other folders you mentioned.

[hrosenbe]

OK. I found the issue. These files exist on my systems, but the .gitgnore is preventing them from being added to the repository. I will fix this and create a new release.

[hrosenbe]

I have push a fix in https://github.com/vmware/weathervane/commit/ad27fdc5e009fbd2c24302b372c25c0f201b5466 that makes the missing files available.

If you grab the files in dockerImages/tomcat/apache-tomcat-auction1/bin and dockerImages/tomcat/apache-tomcat-auction1/lib and place them in the correct place on your system then hopefully everything will work.

[mafzal786]

I'm able to create tomcat image now. also weathvane.pl ran but it ended up on this

The workload driver nodes for workload 1 did not start within 4 minutes. Exiting

[mafzal786]

what could be the reason for that

[hrosenbe]

Sorry about the slow response.

Can you please attach a zip of the output directory: /root/weathervane/output/15

Failing proportion checks can happen for a few reasons:

• The application deployment cannot handle the load placed by the selected number of users.
• The run is too short, or the number of users too low, to reach the expected steady-state proportions. In short or low-load runs, random variation can cause these checks to fail.
• There was some sort of error during the run that was not caught by the run harness.

I suspect some sort of error in this case, as the failing operations are all those that a user performs when it initially starts. This pattern of proportion check failures sometime occurs if the auctions did not start for some reason. I need to see the logs in order to help debug.

[mafzal786]

it finally worked after reducing number of users and making some changes in weathervane.forward.zone file.

Is there any option to use overlay network with weathervane.config file? I see it only support bridge and host network.

Does it work with docker in swarm mode?

Thanks

[hrosenbe]

Weathervane does not currently support Docker swarm.

Adding support for Kubernetes and Docker swarm is high on our priority list, but probably won't happen for a month or more.

If all that you need from overlay networks is to have a custom network name in the --network setting, then that works already. Just set the parameter "dockerNet" to the name of your network. The documentation claims that only "host" and "bridge" are supported, but that is out of date.