MemoryLeak in OpenWebStart

After starting our application with OpenWebStart 1.4 a customer pc stops working during the day due to a high cpu usage.

I was able to check the PC and have seen the the garbage collector was running because of low memory (application was started with 1GB) After that I've created a heap dump that shows that JavaConsole and KeystorePasswordAttempter are the biggest Objects in the dump -> not sure if it's really related to OWS or maybe more to ITW

Logging options are set to

There is also a proxy.pac configured in ows-proxy-settings (proxy is setup with authentication and therefore OWS asks for credentials on startup)

With enabled console 've seen a lot of outputs like this for EACH connection to a https-url.

so it seems at least that the JavaConsole is not limited and uses more and more memory over the day.

Same for KeystorePasswordAttempter but I have no idea where this comes from.

I haven't found any OutOfMemory-Issues so I've created this one. I'll try to test this with the latest OWS-Version as soon as possible on the customer-PC.

[edit] same issue with OWS 1.7 setting the console to "deactivated" instead of "hidden" seems to ~~fix~~ workaround the leak but KeystorePasswordAttempter still wastes a lot of memory.

[/edit]

"-> not sure if it's really related to OWS or maybe more to ITW"

Keystores are handled by ITW.

Can you please share your OWS stage1 and stage2 log files and deployment.properties? Either you can post on the OWS forum or email zip at openwebstart@karakun.com.

You say " JavaConsole and KeystorePasswordAttempter are the biggest Objects in the dump".

Console: is the app started with console on and kept running through the day?

KeystorePasswordAttempter: you say that there is Keystore related log "for EACH connection to a https-url.". Is your application making https-url requests? Is it making these requests to the same server from where jnlp and jars were downloaded or to different servers?

OWS sets up the keystores before downloading resources like jnlp and jars. These keystores are loaded from UserHome/.config/icedtea-web/security dir and from the security dir of the jvm (bundled jvm in stage 1 and specified jvm in jnlp for stage-2).

I tried creating a scenario where the jnlp and jars are accessed by OWS with https-url behind a proxy and the app after starting makes https-url requests to the same server. The app uses the same proxy. I do not see additional Keystores being created per https-url request.

Logs: I'll try to create some next week.

JavaConsole: the default settings "hide" are used. An yes, the app runs the whole day.

Requests: we are calling out services over https from out webstart-application. So yes, it's always the same server but functionality comes from some (signed) jars. Next week I can check what happens if I use plain url-connections.

I was also not able to reproduce the behavior of the KeystorePasswordAttempter locally but it reproducible everytime on the customer system.

I was able to debug the problem on the customer system on each request (with apache cxf) I have this stacktrace

    Thread [Worker-5: Session keep alive] (Suspended (breakpoint at line 161 in net.sourceforge.jnlp.security.KeystorePasswordAttempter))   
        owns: java.util.Collections$SynchronizedRandomAccessList<E>  (id=735)   
        owns: net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader  (id=736)    
        owns: net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage  (id=1632) 
        owns: net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.LazilyLoadedCtor  (id=1633) 
        owns: sun.net.www.protocol.https.DelegateHttpsURLConnection  (id=1634)  
        owns: sun.net.www.protocol.https.HttpsURLConnectionImpl  (id=1635)  
        owns: org.apache.cxf.phase.PhaseInterceptorChain  (id=1636) 
        net.sourceforge.jnlp.security.KeystorePasswordAttempter.unlockKeystore(net.sourceforge.jnlp.security.KeystorePasswordAttempter$KeystoreOperation) line: 161 
        net.sourceforge.jnlp.security.SecurityUtil.loadKeyStore(java.security.KeyStore, java.io.File) line: 358 
        net.sourceforge.jnlp.security.KeyStores.createKeyStoreFromFile(java.io.File, boolean) line: 328 
        net.sourceforge.jnlp.security.KeyStores.getKeyStore(net.sourceforge.jnlp.security.KeyStores$Level, net.sourceforge.jnlp.security.KeyStores$Type) line: 118  
        net.sourceforge.jnlp.security.KeyStores.getCertKeyStores() line: 146    
        net.sourceforge.jnlp.tools.JarCertVerifier.checkTrustedCerts(java.security.cert.CertPath) line: 486 
        net.sourceforge.jnlp.tools.JarCertVerifier.verifyJars(java.util.List<net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc>, net.adoptopenjdk.icedteaweb.resources.ResourceTracker) line: 262   
        net.sourceforge.jnlp.tools.JarCertVerifier.add(net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc, net.adoptopenjdk.icedteaweb.resources.ResourceTracker) line: 214  
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.lambda$addNewJar$11(net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc, java.net.URL) line: 1562    
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$618.1655408146.run() line: not available   
        java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction<T>) line: not available [native method] 
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.addNewJar(net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc, net.adoptopenjdk.icedteaweb.resources.UpdatePolicy) line: 1561    
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.addNewJar(net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc) line: 1523    
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.loadFromJarIndexes(java.lang.String) line: 1499    
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.lambda$loadClass$8(java.lang.String) line: 1444    
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$431.2064300758.call() line: not available  
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$431.2064300758(net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$ExceptionalSupplier<T,E>).getResultOfCallOrNull() line: 1405   
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$432.1932561632.apply(java.lang.Object) line: not available 
        java.util.stream.ReferencePipeline$3$1.accept(P_OUT) line: 195  
        java.util.ArrayList$ArrayListSpliterator.tryAdvance(java.util.function.Consumer<? super E>) line: 1632  
        java.util.stream.ReferencePipeline$Head<E_IN,E_OUT>(java.util.stream.ReferencePipeline<P_IN,P_OUT>).forEachWithCancel(java.util.Spliterator<P_OUT>, java.util.stream.Sink<P_OUT>) line: 127 
        java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).copyIntoWithCancel(java.util.stream.Sink<P_IN>, java.util.Spliterator<P_IN>) line: 502    
        java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).copyInto(java.util.stream.Sink<P_IN>, java.util.Spliterator<P_IN>) line: 488  
        java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).wrapAndCopyInto(S, java.util.Spliterator<P_IN>) line: 474 
        java.util.stream.FindOps$FindOp<T,O>.evaluateSequential(java.util.stream.PipelineHelper<T>, java.util.Spliterator<S>) line: 150 
        java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).evaluate(java.util.stream.TerminalOp<E_OUT,R>) line: 234  
        java.util.stream.ReferencePipeline$2(java.util.stream.ReferencePipeline<P_IN,P_OUT>).findFirst() line: 543  
        net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.loadClass(java.lang.String) line: 1448 
        org.eclipse.osgi.internal.framework.ContextFinder.loadClass(java.lang.String, boolean) line: 145    
        org.eclipse.osgi.internal.framework.ContextFinder(java.lang.ClassLoader).loadClass(java.lang.String) line: 522  
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Kit.classOrNull(java.lang.ClassLoader, java.lang.String) line: 88 
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaPackage).getPkgProperty(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, boolean) line: 154   
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaPackage).get(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 105   
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage.init(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, boolean) line: 129 
        jdk.internal.reflect.GeneratedMethodAccessor6.invoke(java.lang.Object, java.lang.Object[]) line: not available  
        jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 43 
        java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 566    
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject.buildClassCtor(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Class, boolean, boolean) line: 1018   
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.LazilyLoadedCtor.buildValue() line: 110   
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.LazilyLoadedCtor.init() line: 89  
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeObject(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject).getImpl(java.lang.String, int, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 1992  
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeObject(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject).get(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 280    
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeObject(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.IdScriptableObject).get(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 385  
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject.getProperty(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String) line: 1575   
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.topScopeName(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String) line: 1748  
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.nameOrFunction(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String, boolean) line: 1715 
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.name(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String) line: 1657  
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Interpreter.interpretLoop(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Interpreter$CallFrame, java.lang.Object) line: 3413    
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Interpreter.interpret(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.InterpretedFunction, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 2484    
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.InterpretedFunction.call(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 162 
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ContextFactory.doTopCall(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Callable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 401 
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.doTopCall(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Callable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 3003 
        net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.InterpretedFunction.call(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 160 
        com.openwebstart.proxy.pac.PacFileEvaluator.lambda$getProxiesWithoutCaching$0(java.net.URI) line: 190   
        com.openwebstart.proxy.pac.PacFileEvaluator$$Lambda$258.1270431260.run() line: not available    
        java.security.AccessController.doPrivileged(java.security.PrivilegedAction<T>, java.security.AccessControlContext) line: not available [native method]  
        com.openwebstart.proxy.pac.PacFileEvaluator.getProxiesWithoutCaching(java.net.URI) line: 201    
        com.openwebstart.proxy.pac.PacFileEvaluator.getProxies(java.net.URI) line: 133  
        com.openwebstart.proxy.manual.ManualPacFileProxyProvider(com.openwebstart.proxy.pac.PacBasedProxyProvider).select(java.net.URI) line: 30    
        com.openwebstart.proxy.WebStartProxySelector.select(java.net.URI) line: 50  
        sun.net.www.protocol.https.DelegateHttpsURLConnection(sun.net.www.protocol.http.HttpURLConnection).plainConnect0() line: 1181   
        sun.net.www.protocol.http.HttpURLConnection$6.run() line: 1071  
        sun.net.www.protocol.http.HttpURLConnection$6.run() line: 1069  
        java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext) line: not available [native method] 
        java.security.AccessController.doPrivilegedWithCombiner(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext, java.security.Permission...) line: 795  
        sun.net.www.protocol.https.DelegateHttpsURLConnection(sun.net.www.protocol.http.HttpURLConnection).plainConnect() line: 1068    
        sun.net.www.protocol.https.DelegateHttpsURLConnection(sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection).connect() line: 189    
        sun.net.www.protocol.https.DelegateHttpsURLConnection(sun.net.www.protocol.http.HttpURLConnection).getOutputStream0() line: 1367    
        sun.net.www.protocol.http.HttpURLConnection$8.run() line: 1334  
        sun.net.www.protocol.http.HttpURLConnection$8.run() line: 1332  
        java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext) line: not available [native method] 
        java.security.AccessController.doPrivilegedWithCombiner(java.security.PrivilegedExceptionAction<T>, java.security.AccessControlContext, java.security.Permission...) line: 795  
        sun.net.www.protocol.https.DelegateHttpsURLConnection(sun.net.www.protocol.http.HttpURLConnection).getOutputStream() line: 1331 
        sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream() line: 246   
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$1.run() line: 267   
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream$1.run() line: 264   
        java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction<T>) line: not available [native method] 
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream() line: 264  
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream(org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream).handleHeadersTrustCaching() line: 1343   
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream(org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream).onFirstWrite() line: 1304    
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite() line: 307    
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream(org.apache.cxf.io.AbstractWrappedOutputStream).write(byte[], int, int) line: 47 
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream(org.apache.cxf.io.AbstractThresholdOutputStream).write(byte[], int, int) line: 69   
        org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream(org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream).close() line: 1356   
        org.apache.cxf.transport.http.URLConnectionHTTPConduit(org.apache.cxf.transport.AbstractConduit).close(org.apache.cxf.message.Message) line: 56 
        org.apache.cxf.transport.http.URLConnectionHTTPConduit(org.apache.cxf.transport.http.HTTPConduit).close(org.apache.cxf.message.Message) line: 671   
        org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(org.apache.cxf.message.Message) line: 63   
        org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(org.apache.cxf.message.Message) line: 308    
        org.apache.cxf.endpoint.ClientImpl.doInvoke(org.apache.cxf.endpoint.ClientCallback, org.apache.cxf.service.model.BindingOperationInfo, java.lang.Object[], java.util.Map<java.lang.String,java.lang.Object>, org.apache.cxf.message.Exchange) line: 530 
        org.apache.cxf.endpoint.ClientImpl.invoke(org.apache.cxf.service.model.BindingOperationInfo, java.lang.Object[], java.util.Map<java.lang.String,java.lang.Object>, org.apache.cxf.message.Exchange) line: 441   
        org.apache.cxf.endpoint.ClientImpl.invoke(org.apache.cxf.service.model.BindingOperationInfo, java.lang.Object[], org.apache.cxf.message.Exchange) line: 356 
        org.apache.cxf.endpoint.ClientImpl.invoke(org.apache.cxf.service.model.BindingOperationInfo, java.lang.Object...) line: 314 
        org.apache.cxf.jaxws.JaxWsClientProxy(org.apache.cxf.frontend.ClientProxy).invokeSync(java.lang.reflect.Method, org.apache.cxf.service.model.BindingOperationInfo, java.lang.Object[]) line: 96 
        org.apache.cxf.jaxws.JaxWsClientProxy.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) line: 140  
        com.sun.proxy.$Proxy110.ping(java.lang.String) line: not available  
        *****.SessionKeepaliveRunnable.run(org.eclipse.core.runtime.IProgressMonitor) line: 52  
        org.eclipse.core.internal.jobs.Worker.run() line: 63

The loaded jars are always to and the loaded "class" is always com or org

The two jars and a few others containing an INDEX.LIST and only the two cars containing entries like this:

...
org
org/apache
org/apache/groovy
org/apache/groovy/antlr
...

I'll try to remove the INDEX.LIST from the JAR files an see what happens

[edit] removing the INDEX.LIST from the affected jars seems also to solve the issue becuase I never reach the put in the KeystorePasswordAttempter. So it seems to be an issue with the file and "something" in the infrastructure (proxy with authentication, signed application, load-balancer, ...)

hopefully the stacktrace gives you the right direction. [/edit]

"An INDEX.LIST file is inserted to the META-INF directory which will enable the application class loader to download the specified jar files when it is searching for classes or resources. ". ITW's JNLPClassloader downloads the jars in index.list and verifies them for signature and that is when the Keystore related code is executed.

net.sourceforge.jnlp.security.KeystorePasswordAttempter.unlockKeystore(net.sourceforge.jnlp.security.KeystorePasswordAttempter$KeystoreOperation) line: 161 net.sourceforge.jnlp.security.SecurityUtil.loadKeyStore(java.security.KeyStore, java.io.File) line: 358 net.sourceforge.jnlp.security.KeyStores.createKeyStoreFromFile(java.io.File, boolean) line: 328 net.sourceforge.jnlp.security.KeyStores.getKeyStore(net.sourceforge.jnlp.security.KeyStores$Level, net.sourceforge.jnlp.security.KeyStores$Type) line: 118 net.sourceforge.jnlp.security.KeyStores.getCertKeyStores() line: 146 net.sourceforge.jnlp.tools.JarCertVerifier.checkTrustedCerts(java.security.cert.CertPath) line: 486 net.sourceforge.jnlp.tools.JarCertVerifier.verifyJars(java.util.List<net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc>, net.adoptopenjdk.icedteaweb.resources.ResourceTracker) line: 262 net.sourceforge.jnlp.tools.JarCertVerifier.add(net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc, net.adoptopenjdk.icedteaweb.resources.ResourceTracker) line: 214 net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.lambda$addNewJar$11(net.adoptopenjdk.icedteaweb.jnlp.element.resource.JARDesc, java.net.URL) line: 1562

So I can understand that OWS is accessing Keystores while downloading jar files. But the thing that I did not understand is when you said that the KeyStore is accessed for every https request made by the application that is started by OWS.

it's not every https-request - I've tried to create an URL (https://server/index.html) and open the stream. This results in an exception that the proxy needs authentication (maybe I had to setup additional proxy-informations).

Our service-calls are done with apache cxf and using the same url but they can connect to the server. and during this calls always the same two jars are loaded again and again und fillup the sucessfulPerKeystore map.

I've downloaded the proxy.pac and removed nearly all entries. With this proxy.pac I still have the problem on the system

function FindProxyForURL(url, host) {
    if (isInNet(host, "127.0.0.0", "255.0.0.0")) return "DIRECT";

    return "DIRECT";
}

with this not

function FindProxyForURL(url, host) {
    return "DIRECT";
}

This is strange because both are returning always DIRECT 🤔

A workaround for this issue could be to simply not cache default-passwords in KeystorePasswordAttempter by changing

                successfulPerKeystore.put(operation.ks, pass);

                if (!Arrays.equals(pass.pass, DEFAULT_PASSWORD)) // maybe additional checks if localPasses.size() == 1 and/or operation.f == null
                  successfulPerKeystore.put(operation.ks, pass);

This does not fix the problem but results (at least on the customer system) on an empty successfulPerKeystore because there are only keystores with the default password.

I created an app that makes https requests to a server other than where the jnlp and jars are hosted. That other server has a different certifcate than the server jnlp is hosted. In this case, when the request is made, OWS shows the security dialog for the new certificate and once it is accepted Keystores are not accessed. So I still don't understand how Keystores are access on every https request by your app.

You say "Our service-calls are done with apache cxf and using the same url but they can connect to the server.and during this calls always the same two jars are loaded again and again und fillup". ? From the stacktrace:

https req is made.
while making request pac proxy is accessed
LazilyLoadedCtor is called
Some classes are loaded from JarIndexe
jars from JarIndexes are being loaded and as a result OWS 's jareCertVerifier is accessing Keystores to check the certificate.

I was also not able to reproduce it in our environment the only thing I've observed is that always the classes "com" and "org" are loaded from the same two jars and therefore (I think its because that are packages and not classes) they are loaded again and again from the jars.

maybe it's because of the customers infrastructure, java-version or something like that. if you have any ideas how to reproduce this feel free to ask - I can try this on the customer-system.

But regardless of the problem, the map should not be filled to an OutOfMemory :-)

@AlBundy33 Can you please try setting deployment.proxy.pac.cache=true in deployment.properties and then try your application?

From your stack trace com.openwebstart.proxy.pac.PacFileEvaluator which results in call to JNLPClassLoader.loadClass which then tries to load a jar file which results in JarCertVerifier accessing KeyStore. My guess is that setting the above property should prevent repeated calls to PacFileEvaluator and thus the whole chain of calls. Please let me know if this works.

Sorry for late reply but I did create C:\Windows\Sun\Java\Deplyoment\deplyoment.config with deployment.proxy.pac.cache=true but this didn't help.

Daemon Thread [service-executor-service2-pool-6] (Suspended (breakpoint at line 1495 in net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader))  
    owns: java.util.Collections$SynchronizedRandomAccessList<E>  (id=115)   
    owns: net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader  (id=116)    
    owns: net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage  (id=200)  
    owns: net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.LazilyLoadedCtor  (id=201)  
    net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.loadFromJarIndexes(java.lang.String) line: 1495    
    net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.lambda$loadClass$8(java.lang.String) line: 1444    
    net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$428.1858761145.call() line: not available  
    net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$428.1858761145(net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$ExceptionalSupplier<T,E>).getResultOfCallOrNull() line: 1405   
    net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader$$Lambda$429.1442483868.apply(java.lang.Object) line: not available 
    java.util.stream.ReferencePipeline$3$1.accept(P_OUT) line: 195  
    java.util.ArrayList$ArrayListSpliterator.tryAdvance(java.util.function.Consumer<? super E>) line: 1632  
    java.util.stream.ReferencePipeline$Head<E_IN,E_OUT>(java.util.stream.ReferencePipeline<P_IN,P_OUT>).forEachWithCancel(java.util.Spliterator<P_OUT>, java.util.stream.Sink<P_OUT>) line: 127 
    java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).copyIntoWithCancel(java.util.stream.Sink<P_IN>, java.util.Spliterator<P_IN>) line: 502    
    java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).copyInto(java.util.stream.Sink<P_IN>, java.util.Spliterator<P_IN>) line: 488  
    java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).wrapAndCopyInto(S, java.util.Spliterator<P_IN>) line: 474 
    java.util.stream.FindOps$FindOp<T,O>.evaluateSequential(java.util.stream.PipelineHelper<T>, java.util.Spliterator<S>) line: 150 
    java.util.stream.ReferencePipeline$2(java.util.stream.AbstractPipeline<E_IN,E_OUT,S>).evaluate(java.util.stream.TerminalOp<E_OUT,R>) line: 234  
    java.util.stream.ReferencePipeline$2(java.util.stream.ReferencePipeline<P_IN,P_OUT>).findFirst() line: 543  
    net.sourceforge.jnlp.runtime.classloader.JNLPClassLoader.loadClass(java.lang.String) line: 1448 
    org.eclipse.osgi.internal.framework.ContextFinder.loadClass(java.lang.String, boolean) line: 145    
    org.eclipse.osgi.internal.framework.ContextFinder(java.lang.ClassLoader).loadClass(java.lang.String) line: 522  
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Kit.classOrNull(java.lang.ClassLoader, java.lang.String) line: 88 
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaPackage).getPkgProperty(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, boolean) line: 154   
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaPackage).get(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 105   
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeJavaTopPackage.init(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, boolean) line: 129 
    jdk.internal.reflect.GeneratedMethodAccessor4.invoke(java.lang.Object, java.lang.Object[]) line: not available  
    jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) line: 43 
    java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object...) line: 566    
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject.buildClassCtor(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Class, boolean, boolean) line: 1018   
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.LazilyLoadedCtor.buildValue() line: 110   
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.LazilyLoadedCtor.init() line: 89  
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeObject(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject).getImpl(java.lang.String, int, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 1992  
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeObject(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject).get(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 280    
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.NativeObject(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.IdScriptableObject).get(java.lang.String, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable) line: 385  
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptableObject.getProperty(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String) line: 1575   
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.topScopeName(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String) line: 1748  
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.nameOrFunction(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String, boolean) line: 1715 
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.name(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.String) line: 1657  
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Interpreter.interpretLoop(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Interpreter$CallFrame, java.lang.Object) line: 3413    
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Interpreter.interpret(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.InterpretedFunction, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 2484    
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.InterpretedFunction.call(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 162 
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ContextFactory.doTopCall(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Callable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 401 
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.ScriptRuntime.doTopCall(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Callable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 3003 
    net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.InterpretedFunction.call(net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Context, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, net.adoptopenjdk.icedteaweb.shaded.mozilla.javascript.Scriptable, java.lang.Object[]) line: 160 
    com.openwebstart.proxy.pac.PacFileEvaluator.lambda$getProxiesWithoutCaching$0(java.net.URI) line: 190   
    com.openwebstart.proxy.pac.PacFileEvaluator$$Lambda$252.1712233762.run() line: not available    
    java.security.AccessController.doPrivileged(java.security.PrivilegedAction<T>, java.security.AccessControlContext) line: not available [native method]  
    com.openwebstart.proxy.pac.PacFileEvaluator.getProxiesWithoutCaching(java.net.URI) line: 201    
    com.openwebstart.proxy.pac.PacFileEvaluator.getProxies(java.net.URI) line: 133  
    com.openwebstart.proxy.manual.ManualPacFileProxyProvider(com.openwebstart.proxy.pac.PacBasedProxyProvider).select(java.net.URI) line: 30    
    com.openwebstart.proxy.WebStartProxySelector.select(java.net.URI) line: 50  
    java.net.SocksSocketImpl.connect(java.net.SocketAddress, int) line: 384 
    java.net.Socket.connect(java.net.SocketAddress, int) line: 609  
    org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(int, java.net.Socket, org.apache.http.HttpHost, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.protocol.HttpContext) line: 75   
    org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(org.apache.http.conn.ManagedHttpClientConnection, org.apache.http.HttpHost, java.net.InetSocketAddress, int, org.apache.http.config.SocketConfig, org.apache.http.protocol.HttpContext) line: 142 
    org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(org.apache.http.HttpClientConnection, org.apache.http.conn.routing.HttpRoute, int, org.apache.http.protocol.HttpContext) line: 376 
    org.apache.http.impl.execchain.MainClientExec.establishRoute(org.apache.http.auth.AuthState, org.apache.http.HttpClientConnection, org.apache.http.conn.routing.HttpRoute, org.apache.http.HttpRequest, org.apache.http.client.protocol.HttpClientContext) line: 401    
    org.apache.http.impl.execchain.MainClientExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) line: 236    
    org.apache.http.impl.execchain.ProtocolExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) line: 186  
    org.apache.http.impl.execchain.RetryExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) line: 89  
    org.apache.http.impl.execchain.RedirectExec.execute(org.apache.http.conn.routing.HttpRoute, org.apache.http.client.methods.HttpRequestWrapper, org.apache.http.client.protocol.HttpClientContext, org.apache.http.client.methods.HttpExecutionAware) line: 110  
    org.apache.http.impl.client.InternalHttpClient.doExecute(org.apache.http.HttpHost, org.apache.http.HttpRequest, org.apache.http.protocol.HttpContext) line: 185 
    org.apache.http.impl.client.InternalHttpClient(org.apache.http.impl.client.CloseableHttpClient).execute(org.apache.http.client.methods.HttpUriRequest, org.apache.http.protocol.HttpContext) line: 83   
    org.apache.http.impl.client.InternalHttpClient(org.apache.http.impl.client.CloseableHttpClient).execute(org.apache.http.client.methods.HttpUriRequest) line: 108    
    ...

I can demonstrate this console memory effect with v1.8 by providing a "hello world" example that periodically logs a message to the console (every 500ms). After three hours, we will encounter the memory limit of 1.4GB.

Sorry for late reply but I did create C:\Windows\Sun\Java\Deplyoment\deplyoment.config with deployment.proxy.pac.cache=true but this didn't help.

The above mentioned property had to be included in deployment.properties file used by OWS.

I can demonstrate this console memory effect with v1.8 by providing a "hello world" example that periodically logs a message to the console (every 500ms). After three hours, we will encounter the memory limit of 1.4GB.

Sure, the buffer of the console window will fill up if messages are logged at high frequency. Currently there is no mechanism to clear the console log. This can be considered in the future. For now, unless you are investigating a bug, please run OWS without console in case you app is open for along time and is logging a lot.

A workaround for this issue could be to simply not cache default-passwords in KeystorePasswordAttempter by changing
                successfulPerKeystore.put(operation.ks, pass);

@AlBundy33 Has the behavior w.r.t. KeyStorePasswordAttempter improved with the latestOWS v 1.8.0? Can you please try your app with OWS 1.8.0 and let me know?

@janakmulani did you changed something? With OWS 1.8.0 the keystore-memory-leak seems to be gone.

The hidden-console-memory-leak still exists.

@janakmulani did you changed something? With OWS 1.8.0 the keystore-memory-leak seems to be gone.

Instead of Keystore object we now use the keystore file as key to store successful password.

The hidden-console-memory-leak still exists. Currently there is no mechanism to clear the buffer of console log, it will keep on growing.

@janakmulani sounds good - thanks for the changes and the reminder.

If there is currently no mechanism to clear the buffer or a list with fixed size is used maybe disabling the console by default would be a workaround (console is by default set to hidden)

@janakmulani sounds good - thanks for the changes and the reminder.

If there is currently no mechanism to clear the buffer or a list with fixed size is used maybe disabling the console by default would be a workaround (console is by default set to hidden)

Ok. Thanks. will have a look.

Any news about this issue?

@janakmulani any news about this issue?

@AlBundy33 Will try to do this in the November release.

karakun / OpenWebStart

MemoryLeak in OpenWebStart #530