search

stainlessai / micronaut-jupyter

A Micronaut configuration that integrates your app with an existing Jupyter installation.
https://stainlessai.github.io/micronaut-jupyter
Apache License 2.0
9 stars 5 forks source link

MJ-22 ⁃ StackOverflowError when starting many kernels simultaneously at startup #22

Closed JoshuaCWebDeveloper closed 4 years ago

JoshuaCWebDeveloper commented 4 years ago

At one point, I encountered some sort of a resource exception (Stackoverflow?) when a large number of kernels were started at once. Investigate possible issues when starting or running a large number of kernels.

┆Issue Type: Bug ┆Priority: Medium ┆Status: Resolved ┆Fix Versions: v0.2.1 ┆Issue Number: MJ-22 ┆Epic: Kernel

stainlessbot commented 4 years ago

➤ Joshua Carter commented:

I just tested this by starting up 9 kernels one by one, and then starting up 11 more kernels in rapid succession. No errors were encountered.

All 20 notebooks were able to execute their cells. (Some one at a time, others simultaneously). After the last notebooks executed,, the first notebooks were still able to execute No errors were encountered during execution either.

I then quit jupyter, which shutdown all 20 kernels gracefully, no errors.

I then started jupyter again, and opened up lab, which start all 20 kernels simultaneously. Again no errors.

If something like this is encountered again, copy the actually error message and stacktrace.

stainlessbot commented 4 years ago

➤ Joshua Carter commented:

I am now consistently getting stackoverflow errors by repeating the last test above. No idea why it was working yesterday (confirmed am on the same branch).

The error:

Exception in thread "pool-1-thread-17" java.lang.StackOverflowError at java.io.File.(File.java:279) at java.io.FilePermission$1.run(FilePermission.java:215) at java.io.FilePermission$1.run(FilePermission.java:203) at java.security.AccessController.doPrivileged(Native Method) at java.io.FilePermission.init(FilePermission.java:203) at java.io.FilePermission.(FilePermission.java:277) at java.lang.SecurityManager.checkRead(SecurityManager.java:888) at java.io.File.exists(File.java:814) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334) at sun.misc.URLClassPath.getResource(URLClassPath.java:249) at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708) at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41) at java.lang.SecurityManager.checkRead(SecurityManager.java:888) at java.io.File.exists(File.java:814) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334) at sun.misc.URLClassPath.getResource(URLClassPath.java:249) at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708) at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41) at java.lang.SecurityManager.checkRead(SecurityManager.java:888) at java.io.File.exists(File.java:814) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334) at sun.misc.URLClassPath.getResource(URLClassPath.java:249) at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708) at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41) at java.lang.SecurityManager.checkRead(SecurityManager.java:888) at java.io.File.exists(File.java:814) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334) at sun.misc.URLClassPath.getResource(URLClassPath.java:249) at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708) at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41) at java.lang.SecurityManager.checkRead(SecurityManager.java:888) at java.io.File.exists(File.java:814) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334) at sun.misc.URLClassPath.getResource(URLClassPath.java:249) at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704) at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708) at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41) at java.lang.SecurityManager.checkRead(SecurityManager.java:888) at java.io.File.exists(File.java:814)

    ... Truncated ...

    at java.io.File.exists(File.java:814)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708)
    at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41)
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
    at java.io.File.exists(File.java:814)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708)
    at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41)
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
    at java.io.File.exists(File.java:814)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708)
    at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41)
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
    at java.io.File.exists(File.java:814)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708)
    at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41)
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
    at java.io.File.exists(File.java:814)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareNotEqual(ScriptBytecodeAdapter.java:708)
    at org.codehaus.groovy.tools.shell.util.NoExitSecurityManager.checkPermission(NoExitSecurityManager.groovy:41)
    at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
    at java.io.File.exists(File.java:814)
    at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1334)
    at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation.compareEqual(DefaultTypeTransformation.java:615)
    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.compareEqual(ScriptBytecodeAdapter.java:704)
stainlessbot commented 4 years ago

➤ Joshua Carter commented:

This error only happens if no kernels have been started on the server yet. The error is caused by setting a NoExitSecurityManager on the system twice. There was logic put in place to prevent this, but this logic does not take into account the fact that requests made to Micronaut endpoints are multithreaded (each request gets a new thread). This means that if there is no security manager currently set, then all (say 20) requests will hit this logic at a point in time when there is no security manager, and all 20 of them will create their own security managers, hence the error.

FYI, currently the security manager is never unset.

This problem is very similar to the many kernels problem with MJ-17. Again, using just one resource per application runtime, as opposed to one resource per kernel should solve the issue. Create a single security manager for the lifetime of the app (this will have not much of an effect since the security managers were never restored anyways).