denoland / deno

A modern runtime for JavaScript and TypeScript.
https://deno.com
MIT License
93.37k stars 5.18k forks source link

Uploading file to GCP with bucket.upload in @google-cloud/storage hangs #24232

Open andyjp opened 1 week ago

andyjp commented 1 week ago

deno --version output: deno 1.44.2 (release, aarch64-apple-darwin) v8 12.6.228.9 typescript 5.4.5

When uploading a file > 128 kib with the bucket.upload function in @google-coud/storage, the function hangs.

Tested with the script found here: https://github.com/denoland/deno/issues/18730#issuecomment-2110467290 and @google-cloud/storage@7.11.2

Steps to reproduce

  1. Update the script to log progress, await storage.bucket(bucketName).upload(filePath, { destination: fileName, onUploadProgress: console.log })
  2. Create file greater than 128 kib, e.g. head -c 131073 </dev/random > myfile
  3. Run script deno run -A script.ts upload-file [bucketName] myfile ./myfile
  4. This outputs the below and hangs
    { bytesWritten: 65536, contentLength: "*" }
    { bytesWritten: 131072, contentLength: "*" }
    { bytesWritten: 131073, contentLength: "*" }

With a 128 kib file, it finishes:

{ bytesWritten: 65536, contentLength: "*" }
{ bytesWritten: 131072, contentLength: "*" }
File myfile uploaded to bucket

It works fine with tsx:

npx tsx script.ts upload-file [bucketName] myfile ./myfile
{ bytesWritten: 65536, contentLength: '*' }
{ bytesWritten: 131072, contentLength: '*' }
{ bytesWritten: 131073, contentLength: '*' }
File myfile uploaded to bucket

Related: https://github.com/denoland/deno/issues/24215

satyarohith commented 5 days ago

I narrowed the issue down to node-fetch. The example below works with Node but not with Deno.

import fetch from "node-fetch";
import { createReadStream, statSync } from "node:fs";

const url = "https://httpbin.org/put";

// head -c 131073 </dev/random > myfile # a byte more than 128KiB
const readableStream = createReadStream("./myfile");

const headers = {
  "Content-Type": "application/octet-stream",
  "Content-Length": statSync("./myfile").size,
};

const options = {
  method: "PUT",
  headers: headers,
  body: readableStream,
};

const res = await fetch(url, options);
console.log("Response:", await res.json());