Ngrok listener returning url but then isn't online

[alexandre-embie]

I have a script that gets a ngrok url and puts it in the .env file, but it doesn't work, do you guys have any idea why ? 🤔

Here is my code:

import fs from "fs";

import getPort from "get-port";
import ngrok from "@ngrok/ngrok";
import { config } from "dotenv";

config();

function insertEnv({ name, value }: { name: string; value: string }) {
  // eslint-disable-next-line no-console
  console.log(`Inserting ${name} into .env with value ${value}`);
  const envPath = ".env";
  const env = fs.readFileSync(envPath, "utf8");
  // eslint-disable-next-line lodash/prefer-lodash-method
  const envs = env.split("\n");
  // eslint-disable-next-line lodash/prefer-lodash-method
  const existingEnv = envs.find((env) => env.startsWith(name));
  if (existingEnv) {
    // eslint-disable-next-line lodash/prefer-lodash-method
    const newEnv = existingEnv.replace(new RegExp(`${name}=.*`), `${name}=${value}`);
    envs[envs.indexOf(existingEnv)] = newEnv;
  } else {
    envs.push(`${name}=${value}`);
  }
  fs.writeFileSync(envPath, envs.join("\n"));
}

async function extractUrl(port: number) {
  // eslint-disable-next-line no-console
  console.log(`Forwarding port ${port} with ngrok`);
  const listener = await ngrok.forward({ authtoken_from_env: true, addr: port });
  const url = listener.url();
  if (!url) {
    throw new Error("Could not extract ngrok URL");
  }
  return url;
}

export async function initializeNgrok() {
  const port = await getPort();
  const url = await extractUrl(port);
  if (!url) {
    throw new Error("No URL returned from ngrok");
  }
  insertEnv({ name: "SERVER_URL", value: url });
  insertEnv({ name: "PORT", value: port.toString() });
  return url;
}

void initializeNgrok();

and when I do tsx ngrok-initalizer.ts

I get:

Forwarding port 55984 with ngrok
Inserting SERVER_URL into .env with value https://f513-2a02-a03f-6b2d-101-487a-2ac2-23cc-2744.ngrok-free.app
Inserting PORT into .env with value 55984

Process finished with exit code 0

Then when I visit https://f513-2a02-a03f-6b2d-101-487a-2ac2-23cc-2744.ngrok-free.app I get:

And in my dashboard on ngrok.com, nothing appears as when doing the classic command ngrok http 3000.

Do you guys have any ideas why ? Help is much appreciated!

[bobzilladev]

Hello, thanks for writing in! It looks like the process is exiting, so the ngrok session is being torn down, and then there is no longer a way to connect through to the local application. If you add a sleep or something else to block the process from exiting, the remote connection should be able to be established. Let us know if that doesn't solve the issue!

The line that appears to be the process shutting down: Process finished with exit code 0.

[alexandre-embie]

But I don't get the point of this package then, i would assume running this code would open a listener in the background or something. But there, I create the listener, then it simply exits ?

I fixed it by doing:

import fs from "fs";
import { execSync } from "child_process";

import getPort from "get-port";
import ngrok from "@ngrok/ngrok";
import { config } from "dotenv";
import dotenv from "dotenv";
import replace from "lodash/replace";
import find from "lodash/find";
import startsWith from "lodash/startsWith";
import split from "lodash/split";

config();

/**
 * Inserts a new environment variable into the .env file.
 * If the environment variable already exists, its value is updated.
 *
 * @param {Object} param0 - An object containing the name and value of the environment variable.
 * @param {string} param0.name - The name of the environment variable.
 * @param {string} param0.value - The value of the environment variable.
 */
function insertEnv({ name, value }: { name: string; value: string }) {
  // eslint-disable-next-line no-console
  console.log(`Inserting ${name} into .env with value ${value}`);
  const envPath = ".env";
  const env = fs.readFileSync(envPath, "utf8");
  const envs = split(env, "\n");
  const existingEnv = find(envs, (env) => startsWith(env, name));
  if (existingEnv) {
    const newEnv = replace(existingEnv, new RegExp(`${name}=.*`), `${name}=${value}`);
    envs[envs.indexOf(existingEnv)] = newEnv;
  } else {
    envs.push(`${name}=${value}`);
  }
  fs.writeFileSync(envPath, envs.join("\n"));
}

/**
 * Forwards a local port using ngrok and returns the public URL.
 *
 * @param {number} port - The local port to forward.
 * @returns {Promise<string>} - The public URL provided by ngrok.
 */
async function extractUrl(port: number) {
  // eslint-disable-next-line no-console
  console.log(`Forwarding port ${port} with ngrok`);
  const listener = await ngrok.forward({
    authtoken_from_env: true,
    addr: port
  });
  const url = listener.url();
  if (!url) {
    throw new Error("Could not extract ngrok URL");
  }
  return url;
}

/**
 * Initializes ngrok, forwards a local port, and inserts the public URL and port into the .env file.
 *
 * @returns {Promise<string>} - The public URL provided by ngrok.
 */
export const initializeNgrok = async (): Promise<string> => {
  const port = await getPort();
  const url = await extractUrl(port);
  if (!url) {
    throw new Error("No URL returned from ngrok");
  }
  insertEnv({
    name: "SERVER_URL",
    value: url
  });
  insertEnv({
    name: "PORT",
    value: port.toString()
  });
  return url;
};

/**
 * Main function that initializes ngrok and starts the development server with environment variables loaded from the .env file.
 */
const main = async () => {
  await initializeNgrok();

  // Load environment variables from .env file
  const envConfig = dotenv.parse(fs.readFileSync(".env"));

  // Pass the environment variables to execSync
  execSync("next dev", { stdio: "inherit", env: { ...process.env, ...envConfig } });
};

void main();

Basically it does the next dev in the same process through a execsync now, so it doesn't close the ngrok process. I feel like it's a hack, and it prevented me a clean yarn run dev with

script: "tsx ngrok-initalizer.ts && next dev"

What do you think ?

[bobzilladev]

Hi there, the library does create the listener in the background, but the node process has to keep running in order for that connection to remain established. Often the SDK's are used within the same process as what is serving the traffic, so instead of pushing environment variables to a child process, it would all run in the same process. Depending on what framework is being used to serve traffic, there may be an example of this in the examples directory to show how this could be done. For instance, this is the NextJS example.

That said, as long as this parent process is blocking until the child process exits, that will allow the listener to remain open.

[bobzilladev]

Hello, hopefully that answers the question, it looks like you have a way forward either in a parent process or baking into the main process. Going to close the issue, feel free to reopen if there are more questions or something else comes up. Thanks for writing in!

[flazouh]

Hi @bobzilladev, I'm trying to follow the nextJS example

But for some reason, I get

npm run dev

> next dev

node:internal/modules/cjs/loader:1077
  const err = new Error(message);
              ^

Error: Cannot find module './scripts/ngrok-init.script'
Require stack:
- /Users/embie/WebstormProjects/work/jeunes-et-nature/next.config.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1077:15)
    at /Users/embie/WebstormProjects/work/jeunes-et-nature/node_modules/next/dist/server/require-hook.js:55:36
    at Module._load (node:internal/modules/cjs/loader:922:27)
    at Module.require (node:internal/modules/cjs/loader:1143:19)
    at mod.require (/Users/embie/WebstormProjects/work/jeunes-et-nature/node_modules/next/dist/server/require-hook.js:65:28)
    at require (node:internal/modules/cjs/helpers:121:18)
    at Object.<anonymous> (/Users/embie/WebstormProjects/work/jeunes-et-nature/next.config.js:2:1)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/embie/WebstormProjects/work/jeunes-et-nature/next.config.js'
  ]
}

Node.js v18.17.1

Process finished with exit code 1

I see that in the example, the next version is 13, could that be the reason ?

EDIT: I was able to make it work by converting the ts script to js. But now the thing is executed twice for some reason.

/Users/embie/.nvm/versions/node/v18.17.1/bin/npm run dev

> app-jeune-et-nature@0.0.0 dev
> next dev

Forwarding port 3000 with ngrok
Inserting SERVER_URL into .env with value https://b22b-2a02-a03f-6b2d-101-e091-487f-a19a-5056.ngrok-free.app
Forwarding port 3000 with ngrok
   ▲ Next.js 14.0.4
   - Local:        http://localhost:3000
   - Environments: .env

Inserting SERVER_URL into .env with value https://3203-2a02-a03f-6b2d-101-e091-487f-a19a-5056.ngrok-free.app
   Reload env: .env
 ✓ Ready in 1679ms
 ✓ Compiled /children in 433ms (1087 modules)
 ✓ Compiled /api/auth/[...nextauth] in 198ms (312 modules)
[next-auth][warn][DEBUG_ENABLED] 
https://next-auth.js.org/warnings#debug_enabled

[bobzilladev]

Hello! It looks like Next has change how they do their process management, I've made an update which deals with forked processes more generally which will work in 14, see this change: https://github.com/ngrok/ngrok-javascript/pull/136

The important part is changing: makeListener = true; to makeListener = process.send === undefined; to default it off when in a forked process. Thanks for sending this in!

[alexandre-embie]

@bobzilladev Hi Bob, now it seems to be only launched once. But There is this weird thing that went I push to my git remote, the script gets launched:

~/WebstormProjects/work/jeunes-et-nature git:[feat-automatic-ngrok-initializer]
gp

> app-jeune-et-nature@0.0.0 lint
> next lint

Forwarding port 3000 with ngrok
Killing ngrok
Forwarding port 3000 with ngrok
Killing ngrok
Forwarding port 3000 with ngrok
✔ No ESLint warnings or errors
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 697 bytes | 697.00 KiB/s, done.
Total 4 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To github.com:JeunesEtNature/jeunes-et-nature.git
   a42e2d7..f50a9b3  feat-automatic-ngrok-initializer -> feat-automatic-ngrok-initializer

So that's a bit weird in my opinion?

Here is my script:

   const fs = require("fs");

const ngrok = require("@ngrok/ngrok");
const { config } = require("dotenv");
const replace = require("lodash/replace");
const find = require("lodash/find");
const startsWith = require("lodash/startsWith");
const split = require("lodash/split");
const forEach = require("lodash/forEach");
const includes = require("lodash/includes");

config();

const ENV_FILENAME = ".env";

// setup ngrok ingress in the parent process,
// in forked processes "send" will exist.
const makeListener = process.send === undefined;
let host = "localhost";
let port = process.env.PORT || "3000";

forEach(process.argv, (item, index) => {
  if (includes(["--hostname", "-H"], item)) host = process.argv[index + 1];
  if (includes(["--port", "-p"], item)) port = process.argv[index + 1];
});

/**
 * Inserts a new environment variable into the .env file.
 * If the environment variable already exists, its value is updated.
 *
 * @param {Object} param0 - An object containing the name and value of the environment variable.
 * @param {string} param0.name - The name of the environment variable.
 * @param {string} param0.value - The value of the environment variable.
 */
function insertOrReplaceEnv({ name, value }) {
  // eslint-disable-next-line no-console
  console.log(`Inserting ${name} into .env with value ${value}`);

  const env = fs.readFileSync(ENV_FILENAME, "utf8");
  const envs = split(env, "\n");
  const existingEnv = find(envs, (env) => startsWith(env, name));
  if (existingEnv) {
    const newEnv = replace(existingEnv, new RegExp(`${name}=.*`), `${name}=${value}`);
    envs[envs.indexOf(existingEnv)] = newEnv;
  } else {
    envs.push(`${name}=${value}`);
  }
  fs.writeFileSync(ENV_FILENAME, envs.join("\n"));
}

const killNgrok = async () => {
  // eslint-disable-next-line no-console
  console.log("Killing ngrok");
  await ngrok.kill();
};

/**
 * Forwards a local port using ngrok and returns the public URL.
 *
 * @param {number} port - The local port to forward.
 * @returns {Promise<string>} - The public URL provided by ngrok.
 */
async function extractUrl(port) {
  // eslint-disable-next-line no-console
  console.log(`Forwarding port ${port} with ngrok`);
  const authtoken = process.env.NGROK_AUTH_TOKEN;
  if (!authtoken) {
    throw new Error("NGROK_AUTH_TOKEN environment variable is not set");
  }
  try {
    const listener = await ngrok.forward({
      authtoken,
      addr: port
    });
    const url = listener.url();
    if (!url) {
      throw new Error("Could not extract ngrok URL");
    }
    return url;
  } catch (e) {
    //Your account is limited to 1 simultaneous ngrok agent sessions.
    // eslint-disable-next-line lodash/prefer-lodash-method
    if (e.message.includes("Your account is limited to 1 simultaneous ngrok agent sessions")) {
      await killNgrok();
      return extractUrl(port);
    }
  }
}

/**
 * Initializes ngrok, forwards a local port, and inserts the public URL and port into the .env file.
 *
 * @returns {Promise<string>} - The public URL provided by ngrok.
 */
const initializeNgrok = async () => {
  const port = process.env.PORT;
  if (!port) {
    throw new Error("PORT environment variable is not set");
  }
  const url = await extractUrl(parseInt(port));
  if (!url) {
    throw new Error("Ngrok couldn't extract the URL");
  }
  insertOrReplaceEnv({
    name: "SERVER_URL",
    value: url
  });
  return url;
};

if (makeListener) void initializeNgrok();

The reason I insert the url in the env is that I have a service that needs to call a webhook

[bobzilladev]

Interesting, looks like next lint is using Next itself to do the lint checks, which must run through the same initialization code, so might have to put a check on what is on the command line to check for lint and not turn on the ngrok session.