fwcd / kotlin-language-server

Kotlin code completion, diagnostics and more for any editor/IDE using the Language Server Protocol
MIT License
1.65k stars 208 forks source link

Do not iterate over excluded folders while looking for dependencies #176

Open eiva opened 4 years ago

eiva commented 4 years ago

Problem: In project I have many generated folders completely unrelated to java or kotlin. These folders are really huge and to avoid problems with file tracking and searches I've excluded them using

"files.exclude": {
        "build/": true,
        "env/": true
}

With growth of this folder language server first start taking too long to start. Unfortunately after some certain size of this folder kotlin server start crashing on startup while looking for dependencies

[Info  - 8:01:16 PM] client    Searching for dependencies in workspace root /home/xxxxxx
[Info  - 8:01:19 PM] Connection to server got closed. Server will restart.

In certain conditions it is possible to see exception:

PM org.eclipse.lsp4j.jsonrpc.RemoteEndpoint fallbackResponseError
SEVERE: Internal error: java.lang.reflect.InvocationTargetException
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:67)
    at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.request(GenericEndpoint.java:120)
    at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleRequest(RemoteEndpoint.java:261)
    at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume(RemoteEndpoint.java:190)
    at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:192)
    at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
    at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:65)
    ... 11 more
Caused by: java.nio.file.FileSystemException: /home/xxx xxxxx: Too many open files
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
    at java.nio.file.Files.newDirectoryStream(Files.java:457)
    at java.nio.file.Files.list(Files.java:3451)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:25)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.folderResolvers(DefaultClassPathResolver.kt:33)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.workspaceResolvers(DefaultClassPathResolver.kt:18)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.access$workspaceResolvers(DefaultClassPathResolver.kt:1)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt$defaultClassPathResolver$1.invoke(DefaultClassPathResolver.kt:12)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt$defaultClassPathResolver$1.invoke(DefaultClassPathResolver.kt)
    at kotlin.sequences.FlatteningSequence$iterator$1.ensureItemIterator(Sequences.kt:277)
    at kotlin.sequences.FlatteningSequence$iterator$1.hasNext(Sequences.kt:265)
    at org.javacs.kt.classpath.ClassPathResolverKt.getJoined(ClassPathResolver.kt:66)
    at org.javacs.kt.classpath.DefaultClassPathResolverKt.defaultClassPathResolver(DefaultClassPathResolver.kt:12)
    at org.javacs.kt.CompilerClassPath.refresh(CompilerClassPath.kt:20)
    at org.javacs.kt.CompilerClassPath.refresh$default(CompilerClassPath.kt:18)
    at org.javacs.kt.CompilerClassPath.addWorkspaceRoot(CompilerClassPath.kt:64)
    at org.javacs.kt.KotlinLanguageServer.initialize(KotlinLanguageServer.kt:76)
    ... 16 more

I think it simplest way is not trying to iterate over excluded folders.

sys limit for this could not solve the problem, as it is already large enough (in my case 1.5M)

colorhaake commented 4 years ago

I have traced the source codes. I thought it did ignore the patterns from your project root's ".gitignore". The reason to crash could be the method folderResolvers recursively search from the project root to all of its subfolders, especially your project is getting huge. Maybe you can try to add more rules in your ".gitignore" to see the Kotlin Language Server works or not.

/** Searches the workspace for all files that could provide classpath info. */
private fun workspaceResolvers(workspaceRoot: Path): Sequence<ClassPathResolver> {
    val ignored: List<PathMatcher> = ignoredPathPatterns(workspaceRoot.resolve(".gitignore"))
    return folderResolvers(workspaceRoot, workspaceRoot, ignored).asSequence()
}

/** Searches the folder for all build-files. */
private fun folderResolvers(workspaceRoot: Path, folder: Path, ignored: List<PathMatcher>): Collection<ClassPathResolver> {
    var resolvers = mutableListOf<ClassPathResolver>()

    for (file in Files.list(folder)) {
        // Only test whether non-ignored file is a build-file
        if (ignored.none { it.matches(workspaceRoot.relativize(file)) }) {
            val resolver = asClassPathProvider(file)
            if (resolver != null) {
                resolvers.add(resolver)
                break
            } else if (Files.isDirectory(file)) {
                resolvers.addAll(folderResolvers(workspaceRoot, file, ignored))
            }
        }
    }

    return resolvers
}

https://github.com/fwcd/kotlin-language-server/blob/master/shared/src/main/kotlin/org/javacs/kt/classpath/DefaultClassPathResolver.kt#L22