search

ktorio / ktor

Framework for quickly creating connected applications in Kotlin with minimal effort
https://ktor.io
Apache License 2.0
12.72k stars 1.04k forks source link

http client (native macos_x64) v0.9.5-rc13 Memory leaks found #593

Closed luca992 closed 1 year ago

luca992 commented 5 years ago

Whenever I try and do a simple http request on macOs the program freezes and reports a memory leak:

an example where I publish the latest eap13 branch commit to localMaven which is tagged (v0.9.5-rc13):

buildscript {
    repositories {
        jcenter()
        maven { url 'https://plugins.gradle.org/m2/' }
        maven { url 'https://dl.bintray.com/jetbrains/kotlin-native-dependencies' }
        maven {url 'http://dl.bintray.com/kotlin/kotlin-dev'}
    }
    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-native-gradle-plugin:0.9.2-dev-4008"
    }
}

allprojects {

    project.ext {
        //make extras set inside buildscript available in all modules
        kotlin_version = '1.3.0-rc-51'
        kotlin_coroutines_version = '0.26.1-eap13'
    }
}

apply plugin: 'kotlin-platform-native'

repositories {
    mavenLocal()
    jcenter()
}

components.main {
    targets = ["macos_x64"]
    outputKinds = [EXECUTABLE]
    dependencies {
        implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core-native:0.26.1-eap13")

        //you need to clone ktor and publish to local maven
        implementation "io.ktor:ktor-client-ios:0.9.4-SNAPSHOT"
    }

}

example program:

import kotlinx.coroutines.runBlocking
import io.ktor.client.engine.ios.*
import io.ktor.client.*
import io.ktor.http.*
import io.ktor.client.request.*
import kotlinx.coroutines.*
import kotlin.coroutines.*
import platform.darwin.*

internal val ApplicationDispatcher: CoroutineDispatcher = NsQueueDispatcher(dispatch_get_main_queue())

fun main(args: Array<String>) = runBlocking<Unit> {
    val api = ApplicationApi()
    api.about {
        println(it)
    }
    delay(3000L)

    return@runBlocking
}

internal class NsQueueDispatcher(
        private val dispatchQueue: dispatch_queue_t
) : CoroutineDispatcher() {
    override fun dispatch(context: CoroutineContext, block: Runnable) {
        dispatch_async(dispatchQueue) {
            block.run()
        }
    }
}

class ApplicationApi {
    private val client = HttpClient()

    fun about(callback: (String) -> Unit) {
        launch(ApplicationDispatcher) {
            val result: String = client.get {
                url {
                    protocol = URLProtocol.HTTPS
                    port = 443
                    host = "tools.ietf.org"
                    encodedPath = "rfc/rfc1866.txt"
                }
            }

            callback(result)
        }
    }
}

when run:

/opt/teamcity-agent/work/4d622a065c544371/runtime/src/main/cpp/Memory.cpp:1124: runtime 
assert: Memory leaks found
Abort trap: 6
gabrielhuff commented 5 years ago

Any updates on this? This is still happening on 1.0.0-beta-3. Not sure if I'm doing something wrong, but the macOS client always blocks when requesting. The funny thing is that I can see both request and response by sniffing the traffic.

e5l commented 5 years ago

Still in progress. I'll ping in the issue when resolve.

e5l commented 5 years ago

Hi @gabrielhuff. I can't reproduce it with 1.0.1. Could you recheck?

gabrielhuff commented 5 years ago

Sure! Again, I might be doing something wrong, but this is a minimal setup:

settings.gradle

enableFeaturePreview('GRADLE_METADATA')

build.gradle

plugins { id "org.jetbrains.kotlin.native" version "1.3.11" }

repositories {
    jcenter()
}

components.main {
    targets = ["macos_x64"]
    outputKinds = [EXECUTABLE]
    dependencies {
        implementation "org.jetbrains.kotlinx:kotlinx-coroutines-core-native:1.0.1"
        implementation "io.ktor:ktor-client-ios:1.0.1"
    }
}

Main.kt

import io.ktor.client.*
import io.ktor.client.request.*
import kotlinx.coroutines.*

fun main(args: Array<String>) {
    val response = runBlocking<Unit> { HttpClient().get("http://www.google.com") }
    println(response)
}

When executing the program I'm getting the following exception stack:

Uncaught exception from Kotlin's main: kotlin.native.concurrent.InvalidMutabilityException: mutation attempt of frozen <object>@dc59598
        at 0   ktor-mac.kexe                       0x000000010ac1d786 kfun:kotlin.Exception.<init>(kotlin.String?)kotlin.Exception + 70
        at 1   ktor-mac.kexe                       0x000000010ac1d6a6 kfun:kotlin.RuntimeException.<init>(kotlin.String?)kotlin.RuntimeException + 70
        at 2   ktor-mac.kexe                       0x000000010ac1efd6 kfun:kotlin.native.concurrent.InvalidMutabilityException.<init>(kotlin.String)kotlin.native.concurrent.InvalidMutabilityException + 70
        at 3   ktor-mac.kexe                       0x000000010acb74c8 ThrowInvalidMutabilityException + 280
        at 4   ktor-mac.kexe                       0x000000010acd9c68 MutationCheck + 24
        at 5   ktor-mac.kexe                       0x000000010adb2eb8 kfun:io.ktor.client.engine.ios.IosClientEngine.object-1.<init>#internal + 136
        at 6   ktor-mac.kexe                       0x000000010adb29c8 kfun:io.ktor.client.engine.ios.IosClientEngine.execute(io.ktor.client.call.HttpClientCall;io.ktor.client.request.HttpRequestData)io.ktor.client.call.HttpEngineCall + 568
        at 7   ktor-mac.kexe                       0x000000010ada75d0 kfun:io.ktor.client.HttpClient.$<anonymous>_2$COROUTINE$0.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 784
        at 8   ktor-mac.kexe                       0x000000010ada95d6 kfun:io.ktor.client.HttpClient.$<anonymous>_2$COROUTINE$0.invoke#internal + 118
        at 9   ktor-mac.kexe                       0x000000010ad63ccf kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 495
        at 10  ktor-mac.kexe                       0x000000010ad64939 kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 185
        at 11  ktor-mac.kexe                       0x000000010ad64697 kfun:io.ktor.util.pipeline.SuspendFunctionGun.execute#internal + 199
        at 12  ktor-mac.kexe                       0x000000010ad6328e kfun:io.ktor.util.pipeline.Pipeline.execute(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)#GENERIC_kotlin.Any + 142
        at 13  ktor-mac.kexe                       0x000000010ada4307 kfun:io.ktor.client.features.HttpSend.DefaultSender.$execute$COROUTINE$20.invokeSuspend#internal + 439
        at 14  ktor-mac.kexe                       0x000000010ada3e32 kfun:io.ktor.client.features.HttpSend.DefaultSender.execute#internal + 130
        at 15  ktor-mac.kexe                       0x000000010ada3608 kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0$COROUTINE$19.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 968
        at 16  ktor-mac.kexe                       0x000000010ada4846 kfun:io.ktor.client.features.HttpSend.Feature.$install$lambda-0$COROUTINE$19.invoke#internal + 118
        at 17  ktor-mac.kexe                       0x000000010ad63ccf kfun:io.ktor.util.pipeline.SuspendFunctionGun.loop#internal + 495
        at 18  ktor-mac.kexe                       0x000000010ad64939 kfun:io.ktor.util.pipeline.SuspendFunctionGun.proceed#internal + 185
        at 19  ktor-mac.kexe                       0x000000010ad64697 kfun:io.ktor.util.pipeline.SuspendFunctionGun.execute#internal + 199
        at 20  ktor-mac.kexe                       0x000000010ad6328e kfun:io.ktor.util.pipeline.Pipeline.execute(#GENERIC_kotlin.Any;#GENERIC_kotlin.Any)#GENERIC_kotlin.Any + 142
        at 21  ktor-mac.kexe                       0x000000010adaa968 kfun:io.ktor.client.HttpClient.$execute$COROUTINE$1.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 296
        at 22  ktor-mac.kexe                       0x000000010adaaae2 kfun:io.ktor.client.HttpClient.execute(io.ktor.client.request.HttpRequestBuilder)io.ktor.client.call.HttpClientCall + 130
        at 23  ktor-mac.kexe                       0x000000010adabc56 kfun:io.ktor.client.call.$call$COROUTINE$3.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 758
        at 24  ktor-mac.kexe                       0x000000010adabe12 kfun:io.ktor.client.call.call@io.ktor.client.HttpClient.(kotlin.coroutines.SuspendFunction1<io.ktor.client.request.HttpRequestBuilder,kotlin.Unit>)io.ktor.client.call.HttpClientCall + 130
        at 25  ktor-mac.kexe                       0x000000010adac27b kfun:io.ktor.client.call.call@io.ktor.client.HttpClient.(io.ktor.client.request.HttpRequestBuilder)io.ktor.client.call.HttpClientCall + 123
        at 26  ktor-mac.kexe                       0x000000010ac1bad9 kfun:$dummyRequest$COROUTINE$1.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 1417
        at 27  ktor-mac.kexe                       0x000000010ac1b463 kfun:dummyRequest()kotlin.String + 99
        at 28  ktor-mac.kexe                       0x000000010ac1b193 kfun:$main$lambda-0$COROUTINE$0.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? + 195
/opt/teamcity-agent/work/4d622a065c544371/runtime/src/main/cpp/Memory.cpp:1150: runtime assert: Memory leaks found
Abort trap: 6

Someone mentioned here that this is a Kotlin 1.3.11 thing. However if you downgrade to 1.3.10, nothing will be printed and the program will block (although it is possible to sniff the traffic).

Some additional details:

gabrielhuff commented 5 years ago

I just took a look at the source code. Just a hypothesis, but maybe the runBlocking call is blocking the execution (when running with Kotlin 1.3.10) because the IosClientEngine is always delivering its results to the main queue.

Since runBlocking is being called from the main thread itself the execution would be trapped on a deadlock, where main is waiting for the IosClientEngine to deliver the response and the IosClientEngine is waiting for main to unblock so that the response can be dispatched.

e5l commented 5 years ago

Ios client engine can't work in default runBlocking for now: it schedules request in main NSqueue

cy6erGn0m commented 5 years ago

@e5l see repro project

ktor-ios-bug.zip

cy6erGn0m commented 5 years ago

See the following debug information

Detailed backtrace (notice thread name)

Process 31816 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00000001000a4630 untitled.kexe`ThrowInvalidMutabilityException at Internal.kt:71
Target 0: (untitled.kexe) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00000001000a4630 untitled.kexe`ThrowInvalidMutabilityException at Internal.kt:71
    frame #1: 0x00000001000c6ed8 untitled.kexe`MutationCheck + 24
    frame #2: 0x00000001001a0058 untitled.kexe`kfun:io.ktor.client.engine.ios.IosClientEngine.object-1.<init>#internal($this=0x00000001009326d8, call=0x0000000100914578, requestTime_52=0x000000010095ca58, callContext_50=0x0000000100916468, request_51=0x0000000100918458) at IosClientEngine.kt:35
    frame #3: 0x000000010019fb68 untitled.kexe`kfun:io.ktor.client.engine.ios.IosClientEngine.execute(data=0x000000010094bf78)io.ktor.client.call.HttpEngineCall at IosClientEngine.kt:36
    frame #4: 0x0000000100194840 untitled.kexe`kfun:io.ktor.client.HttpClient.$<anonymous>_2$COROUTINE$0.invokeSuspend(kotlin.Result<kotlin.Any?>)kotlin.Any? at HttpClient.kt:86
    frame #5: 0x0000000100196846 untitled.kexe`kfun:io.ktor.client.HttpClient.$<anonymous>_2$COROUTINE$0.invoke#internal(content=0x0000000100924868) at HttpClient.kt:77
...

disassembled (note that we are crashing at the first mutation check just after EnterFrame invocation)

(lldb) f 2
frame #2: 0x00000001001a0058 untitled.kexe`kfun:io.ktor.client.engine.ios.IosClientEngine.object-1.<init>#internal($this=0x00000001009326d8, call=0x0000000100914578, requestTime_52=0x000000010095ca58, callContext_50=0x0000000100916468, request_51=0x0000000100918458) at IosClientEngine.kt:35
(lldb)
(lldb) dis 
untitled.kexe`kfun:io.ktor.client.engine.ios.IosClientEngine.object-1.<init>#internal:
    0x10019ffd0 <+0>:   pushq  %rbp
    0x10019ffd1 <+1>:   movq   %rsp, %rbp
    0x10019ffd4 <+4>:   pushq  %r15
    0x10019ffd6 <+6>:   pushq  %r14
    0x10019ffd8 <+8>:   pushq  %r13
    0x10019ffda <+10>:  pushq  %r12
    0x10019ffdc <+12>:  pushq  %rbx
    0x10019ffdd <+13>:  subq   $0x68, %rsp
    0x10019ffe1 <+17>:  movq   %r9, -0x88(%rbp)
    0x10019ffe8 <+24>:  movq   %r8, %r12
    0x10019ffeb <+27>:  movq   %rcx, %r13
    0x10019ffee <+30>:  movq   %rdx, %r15
    0x10019fff1 <+33>:  movq   %rsi, %r14
    0x10019fff4 <+36>:  movq   %rdi, %rbx
    0x10019fff7 <+39>:  xorps  %xmm0, %xmm0
    0x10019fffa <+42>:  movaps %xmm0, -0x40(%rbp)
    0x10019fffe <+46>:  movaps %xmm0, -0x50(%rbp)
    0x1001a0002 <+50>:  movaps %xmm0, -0x60(%rbp)
    0x1001a0006 <+54>:  movaps %xmm0, -0x70(%rbp)
    0x1001a000a <+58>:  movaps %xmm0, -0x80(%rbp)
    0x1001a000e <+62>:  movq   $0x0, -0x30(%rbp)
    0x1001a0016 <+70>:  leaq   -0x80(%rbp), %rdi
    0x1001a001a <+74>:  movl   $0x7, %esi
    0x1001a001f <+79>:  movl   $0xb, %edx
    0x1001a0024 <+84>:  callq  0x1000c6b10               ; EnterFrame
    0x1001a0029 <+89>:  movq   %rbx, -0x78(%rbp)
    0x1001a002d <+93>:  movq   %r14, -0x70(%rbp)
    0x1001a0031 <+97>:  movq   %r15, -0x68(%rbp)
    0x1001a0035 <+101>: movq   %r13, -0x60(%rbp)
    0x1001a0039 <+105>: movq   %r12, -0x58(%rbp)
    0x1001a003d <+109>: movq   -0x88(%rbp), %rax
    0x1001a0044 <+116>: movq   %rax, -0x50(%rbp)
    0x1001a0048 <+120>: movq   0x10(%rbp), %rax
    0x1001a004c <+124>: movq   %rax, -0x48(%rbp)
    0x1001a0050 <+128>: movq   %rbx, %rdi
    0x1001a0053 <+131>: callq  0x1000c6ec0               ; MutationCheck
->  0x1001a0058 <+136>: movq   %rbx, %rdi
    0x1001a005b <+139>: callq  0x1000c79b0               ; Kotlin_Interop_refToObjC
    0x1001a0060 <+144>: movslq 0x153361(%rip), %rcx      ; kobjcbodyoffs:io.ktor.client.engine.ios.IosClientEngine.object-1#internal
    0x1001a0067 <+151>: leaq   0x18(%rax,%rcx), %rdi
    0x1001a006c <+156>: movq   %r14, %rsi
    0x1001a006f <+159>: callq  0x1000b7340               ; UpdateRef
    0x1001a0074 <+164>: movq   -0x78(%rbp), %rbx
    0x1001a0078 <+168>: movq   -0x60(%rbp), %r14
    0x1001a007c <+172>: movq   %rbx, %rdi
    0x1001a007f <+175>: callq  0x1000c6ec0               ; MutationCheck
luca992 commented 5 years ago

The coroutines 1.1.0 release features:

runBlocking is integrated with Dispatchers.Unconfined by sharing an internal event loop. This 
change does not affect the semantics of the previously correct code but allows to mix multiple 
runBlocking and unconfined tasks

Does this help at all with running the http client on macos?

cy6erGn0m commented 5 years ago

@luca992 , unfortunately it doesn't

cy6erGn0m commented 5 years ago

There is a workaround, see how it is done in ktor's tests: https://github.com/ktorio/ktor/blob/master/ktor-client/ktor-client-ios/darwin/test/utils.kt so you can use help { } function instead of runBlocking { }

asm0dey commented 4 years ago

May I ask you to post heap dump please? It may be gathered with any of these methods: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/

e5l commented 4 years ago

Hi @luca992, could you reproduce this issue with 1.3.2?

luca992 commented 4 years ago

@e5l I tried with 1.3.2 with the ios client on mac and I'm still getting memory leaks

oleg-larshin commented 4 years ago

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.