sveltejs / kit

web development, streamlined
https://kit.svelte.dev
MIT License
17.81k stars 1.78k forks source link

Usage of streams causing file descriptor leak #12173

Open Hirevo opened 2 weeks ago

Hirevo commented 2 weeks ago

Describe the bug

I am attempting to make a SvelteKit API endpoint that makes a request to another backend API and streams the response body of its own response body, using fetch and response.body, like so:

// `filterHeaders` is simply a function to filter only some desired headers.

/** @type {import('./$types').RequestHandler} */
export async function GET(event) {
    const requestHeaders = filterHeaders(event.request.headers, ['range']);
    const response = await fetch('http://example.com/some/example/endpoint', {
        signal: event.request.signal,
        headers: requestHeaders,
    });
    const responseHeaders = filterHeaders(
        response.headers,
        ['content-length', 'content-type', 'content-range', 'accept-ranges'],
    );
    return new Response(response.body, {
        status: response.status,
        statusText: response.statusText,
        headers: responseHeaders,
    });
}

While this code works, it seems to leak a file descriptor each time it is requested, especially in the context of HTTP range requests while playing video content.

I have tried many variations of this code, like using pipeThrough or pipeTo, using the pipeline function from the node:stream/promises module, or using axios to make the request and converting the body back to a ReadableStream, but all of them demonstrated the issue, which prompts to me that maybe its the @sveltejs/adapter-node that doesn't close the response stream properly.

After some time, this issue leads to my app being unable to fulfil new requests, and the node process must be restarted.

Reproduction

I have made a minimal demo app to demonstrate the issue here:

https://github.com/Hirevo/sveltekit-stream-issue

The index page has steps of how to observe the issue, using lsof.

Logs

No response

System Info

System:
  OS: macOS 14.4.1
  CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
  Memory: 411.48 MB / 16.00 GB
  Shell: 0.1.0 - /Users/nicolaspolomack/.cargo/bin/psh
Binaries:
  Node: 22.0.0 - ~/.local/state/fnm_multishells/18305_1714256923846/bin/node
  Yarn: 1.22.22 - /usr/local/bin/yarn
  npm: 10.5.1 - ~/.local/state/fnm_multishells/18305_1714256923846/bin/npm
Browsers:
  Chrome: 124.0.6367.91
  Safari: 17.4.1
  Safari Technology Preview: 17.4
npmPackages:
  @sveltejs/adapter-node: ^5.0.1 => 5.0.1
  @sveltejs/kit: ^2.0.0 => 2.5.7
  @sveltejs/vite-plugin-svelte: ^3.0.0 => 3.1.0
  svelte: ^4.2.7 => 4.2.15
  vite: ^5.0.3 => 5.2.10

Severity

annoyance

Additional Information

From some of my research on this, I am aware that undici, which implements fetch in Node, does not release connection resources upon garbage collection.

I made multiple attempts to close it myself, but I was unsuccessful.

hanszoons commented 2 weeks ago

Perhaps related to #11617?