file download example

[higoka]

is it possible to download multiple files in parallel using httpie? an example would be good

[fraxken]

Yes that's possible, since 4.0 and no stream you can use raw mode to get a Buffer. Basicaly you just need to manage Promise yourself (nothing crazy here).

const { data } = await httpie.get<Buffer>("url_to_file", { mode: "raw" });
console.log(data) // Buffer

Also possible with stream :)

function downloadOne(url, writable) {
  const cursor = httpie.stream("GET", url);

  return cursor(() => writable);
}

[higoka]

But how do i save the content to a file? And how do i download multiple files in parallel?

I have like 10k files i need to download

[fraxken]

@higoka Use a writable stream as writable (fs.createWriteStream for example). Use Promises to manage multiple files (with Promise.all or Promise.allSettled).

You probably need to study a bit more about Node and JavaScript.

[fraxken]

Using ESM

import fs from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import * as httpie from "@myunisoft/httpie";

// CONSTANTS
const __dirname = path.dirname(fileURLToPath(import.meta.url));

const kStoreDirectory = path.join(__dirname, "files");
const kRootApi = new URL("https://github.com");
const kGithubAgent = new httpie.Agent({
  connections: 10
});

fs.mkdirSync(kStoreDirectory, { recursive: true });

function downloadOne(archive) {
  const writableCallback = httpie.stream("GET", new URL(archive, kRootApi), {
    agent: kGithubAgent,
    maxRedirections: 1
  });

  const filename = archive.split("/").pop();
  const writable = fs.createWriteStream(path.join(kStoreDirectory, filename));

  return writableCallback(() => writable);
}

const filesURL = [
  "NodeSecure/i18n/archive/main.tar.gz",
  "NodeSecure/scanner/archive/master.tar.gz"
];

await Promise.allSettled(
  filesURL.map((url) => downloadOne(url))
);

However you still need to:

• Think how to properly manage errors
• Find a way to detect the filename (sometimes the API return it in headers).

return writableCallback(({ headers }) => writable);