Exception = java.io.IOException Source = com.ibm.ws.app.manager.war.internal.WARDeployedAppInfoFactoryImpl

[sarath-s]

Ibm liberty app redeployment of war with mbeans fail with below error, war fully copied to dropin but due this error, war not exploding at all. could see few jars from previous installment in exploded/appname/web-inf/lib folder.

Enabled features are:

  <featureManager>
 <feature>webProfile-8.0</feature>
<feature>restConnector-2.0</feature>
 </featureManager> 

Error is:

-----Start of DE processing----- = [2/10/21, 23:49:58:526 IST]
Exception = java.io.IOException
Source = com.ibm.ws.app.manager.war.internal.WARDeployedAppInfoFactoryImpl
probeid = 113
Stack Dump = java.io.IOException: Failed to delete [ C:\IBM\WebSphere\Liberty\usr\servers\Libertyhost\apps\expanded\test.war ] because [ C:\IBM\WebSphere\Liberty\usr\servers\Libertyhost\apps\expanded\test.war\WEB-INF\lib\com.us.tracking-21.2.0.1.jar ] could not be deleted.
at com.ibm.ws.app.manager.war.internal.WARDeployedAppInfoFactoryImpl.expand(WARDeployedAppInfoFactoryImpl.java:71)
at com.ibm.ws.app.manager.war.internal.WARDeployedAppInfoFactoryImpl.createDeployedAppInfo(WARDeployedAppInfoFactoryImpl.java:107)
at com.ibm.ws.app.manager.war.internal.WARDeployedAppInfoFactoryImpl.createDeployedAppInfo(WARDeployedAppInfoFactoryImpl.java:35)
at com.ibm.ws.app.manager.war.internal.WARApplicationHandlerImpl.setUpApplicationMonitoring(WARApplicationHandlerImpl.java:50)
at com.ibm.ws.app.manager.internal.statemachine.StartAction.execute(StartAction.java:139

looks like few jars are locked by the server and redeployment fails. if server is restarted and delete expanded\test.war then redeployment pass.

though app not exploding, shows app as started

This issue is with both windows and linux.

Reference : https://github.com/OpenLiberty/open-liberty/blob/6c0a35b35e21da1d2babdfe42f1748a9cfd5ce0f/dev/com.ibm.ws.app.manager.war/src/com/ibm/ws/app/manager/war/internal/WARDeployedAppInfoFactoryImpl.java#L85

[jhanders34]

What version of Liberty are you using?

[sarath-s]

What version of Liberty are you using?

product = WebSphere Application Server 20.0.0.12 (wlp-1.0.47.cl201220201111-0736)

[andymc12]

I have a few thoughts here: (1) if something in the server is holding a reference to a class in the application, then you've most likely got a memory leak. Something that is outside of the application (a common shared library, something in the server runtime, etc.) is holding a reference to a class in the application. You may need to get a heap or core dump and use a tool like MAT to find out what is holding the reference. (2) iirc, Liberty uses a wsjar URL handler for loading class files by default. The wsjar protocol is intended to avoid holding a file lock on the JAR file even when it is accessed by the classloader. According to this APAR doc, you can switch from wsjar to jar protocol by adding config like this to your server config: <classloading useJarUrls="true"/>

If you have useJarUrls set to true in your config, you might be able to work around the problem by setting it false. But keep in mind that that is just side-stepping the potentially larger problem of a memory leak. Assuming there is a leak, it is possible that the heap will grow every time you update the application.

[sarath-s]

classloading useJarUrls="true" is not set in the server.xml, tried with classloading useJarUrls="false" but still the issue is reproducible. Its a spring mvc application. Does the feature "webProfile-8.0" has any influence in this issue?

working on andymc12 tought (1) and give the analysis here.

Same application works in other server tomcat and weblogic,jboss as expected and redeployment didn't created any issues.

In tomcat, same jars getting locked in tomcat temp\appname but the "weppapps\appname" folder not locking the jars and other files and redeployment don't have any issues.

[sarath-s]

@andymc12 could you suggest steps for use a tool like MAT to find out what is holding the reference. not able to search WS liberty related classes here .

[andymc12]

@sarath-s Here's my normal process for finding classloader leaks: 1) Start the server and invoke something on the application. The main thing here is to reproduce whatever may be causing the leak - this could just be starting up the app, but might include invoking a servlet, web service, etc. Make sure you know the class name of at least one class in the app that will be loaded. For example, let's say that you invoke a servlet called org.mypkg.MyServlet - remember that classname. 2) Stop the application. It might be good to perform the JMX operation that you used to restart the app - just so that the full problem is reproduced. 3) Acquire the heap and/or system core dump. A Liberty server dump might also be helpful (wlp/bin/server dump <serverName> --include=heap,system will provide a zip file that includes Liberty server diagnostics as well as both a heap dump and core dump). 4) Open the core dump (the core dump is preferred over the heap dump because it contains more details like symbol names, etc. - but sometimes MAT cannot open core dumps from some JVMs - in that case, open the heap dump). 5) Search for the class identified in step 1 (i.e. org.mypkg.MyServlet) - you can search by clicking the blue disk icon and selecting "List objects -> with incoming references" - make sure to include class instances. If that class is not found in the dump, that means that it either wasn't loaded or was garbage collected - if it was never loaded that means you may need to run your test again. If it was garbage collected, that means you likely don't have a memory leak. If you are able to find it, that means that the class was not garbage collected after the app was stopped - most likely a leak. From there, right click on the class and select "Path to GC Roots -> exclude all phantom/weak/soft etc. references". This will give you the path to the root of the heap - something in that path is holding a reference to your class (or the classloader that loaded it) and shouldn't be.

Here's and example of a memory leak I looked at a long time ago: I don't remember the details of this offhand, but I can tell that this servlet is kept on the heap because the web container has a reference to it. And the session store has a reference to some of the web container classes - so it is possible that the app stored something in the in-memory session store that referenced an app class and then did not remove it from the store when it shut down.

[sarath-s]

able to take only heap dump as its shows the below error for system CWWKE0069W: Java dump type system is not supported.

[sarath-s]

hi @andymc12

When i preproduce the issue, able to see springframework related objects in dominator_tree , able to see this tough app is not exploded properly when redeployed. Able to see these objects app deployed and logged first time and when app is stopped as well. liberty is running with oracle jdk 11

how can we resolve this issue? the application is spring mvn application.

thanks sarath

[sarath-s]

list objects with incoming references:

[sarath-s]

hi @andymc12

Thread dump shows let me know if i have to share the thread dump