upstash / qstash-js

Message queue for serverless
https://docs.upstash.com/qstash
MIT License
155 stars 15 forks source link

`Error: Cannot find module '/app/node_modules/@upstash/qstash/dist/base/index.js` #168

Closed mittalyashu closed 6 days ago

mittalyashu commented 1 week ago

I am getting this error after bump this package version to v2.7.0. It was working find with previous stable v2.6.5.

mittalyashu commented 1 week ago

This version is failing our production build, so we had to rollback to previous stable release.

mozeryansky commented 1 week ago

I just filed this too: https://github.com/upstash/qstash-js/issues/167

ogzhanolguncu commented 1 week ago

@mittalyashu See, #167 for the fix. Sorry for the inconvenience, but we are trying to move away from cjs to make bundle lighter.

mozeryansky commented 1 week ago

I mentioned here in the other now closed ticket, that this is still an issue.

  1. mkdir test && cd test
  2. npm i https://github.com/upstash/qstash-js.git\#f86ad8a (get latest on main)
  3. ls node_modules/@upstash/qstash/ Only files are: LICENSE README.md package.json

Same empty folder for npm i https://github.com/upstash/qstash-js.git\#v2.7.0.

However, npm i https://github.com/upstash/qstash-js.git\#v2.6.5 results in:

$ ls node_modules/@upstash/qstash/                      
LICENSE              bun.lockb            examples             platforms            src                  tsup.config.js
README.md            commitlint.config.js package.json         prettier.config.mjs  tsconfig.json
ogzhanolguncu commented 1 week ago

I was able to make it work with base path changes in node example. Can you try to clone it instead?

mozeryansky commented 1 week ago

When I build manually then reference the build it does work, but that's not the same process and what npm i will do, I'm not familiar with building packages though. I installed qstash from the latest commit abe03f9, and have the same issue of an empty folder for node_module/@upstash/qstash.

I'm able to use @upstash/redis just fine as I'd expect, would copying the build logic from that repo work for qstash?

ogzhanolguncu commented 1 week ago

When you install @upstash/qstash@2.7.0 it should work as expected if you are already using Typescript or ESM. If you are using node directly all you should do is add module to your package.json then import the qstash with full path.

mozeryansky commented 1 week ago

I'm still having trouble, I'm not sure we're on the same page here.

Can you tell me specifically what I am doing wrong with this: 1) new empty folder 2) npm i @upstash/qstash@2.7.0 3) create empty file test.ts and put only:

import { Client } from '@upstash/qstash'
const client = new Client({})

4) tsx test.ts Error:

node:internal/modules/cjs/loader:1182
  const err = new Error(`Cannot find module '${request}'`);
              ^
Error: Cannot find module '/Users/michael/repos/tmp/test/node_modules/@upstash/qstash/dist/base/index.js'

5) Change test.ts to use full path:

import { Client } from '@upstash/qstash/dist/base/index.mjs'
const client = new Client({})

Error:

node:internal/modules/esm/resolve:304
  return new ERR_PACKAGE_PATH_NOT_EXPORTED(
         ^
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './dist/base/index.mjs' is not defined by "exports" in /Users/michael/repos/tmp/test/node_modules/@upstash/qstash/package.json

To test js/node only 1) create empty folder 2) npm i @upstash/qstash@2.7.0 3) create empty file test.js and put only:

import { Client } from '@upstash/qstash/dist/base/index.mjs'
const client = new Client({})

5) package.json is minimal:

{
  "type": "module",
  "dependencies": {
    "@upstash/qstash": "^2.7.0"
  }
}

4) node test.js Error:

node:internal/modules/esm/resolve:304
  return new ERR_PACKAGE_PATH_NOT_EXPORTED(
         ^
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './dist/base/index.mjs' is not defined by "exports" in /Users/michael/repos/tmp/test/node_modules/@upstash/qstash/package.json imported from /Users/michael/repos/tmp/test/test.js

I've attached both folders here as zips: test_node.zip test_typescript.zip


As comparison, I can import @upstash/redis and use it without a custom path. Why does there need to be a difference in how qstash is packaged vs redis?

I am now forced to lock in a version to 2.6.5 as the latest published release 2.7.1 is broken. I've made all the changes you've suggested. Can you please tell me how you are verifying your fixes?

ogzhanolguncu commented 1 week ago

Sorry for mislead. I guess all you need is this:

//index.js

import { Client, Receiver } from "@upstash/qstash";
import "isomorphic-fetch";

async function main() {
  const q = new Client({
    token: "",
  });

  const res = await q.publish({
    url: "https://qstash-prod-andreas.requestcatcher.com/test",
    body: "Hello World",
  });
  console.log(res);

  // Validating a signature
  const receiver = new Receiver({
    currentSigningKey: "sig_3nj4aiyJ2JojDnQ1RRodpYubZAZxAJxNfQcRSKPwVUNbueYk2o",
    nextSigningKey: "sig_31zVqmL3s7Eo1vpu1jRSMpaetJXvAT3RvNcfoGUp1Toii8fsQE",
  });

  const isValid = await receiver
    .verify({
      signature:
        "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiIiLCJib2R5IjoicFpHbTFBdjBJRUJLQVJjeno3ZXhrTllzWmI4THphTXJWN0ozMmEyZkZHND0iLCJleHAiOjE2NTc1MzA3NTYsImlhdCI6MTY1NzUzMDQ1NiwiaXNzIjoiVXBzdGFzaCIsImp0aSI6Imp3dF83QXFHbkRLV3dLTmY2dEdUNExnRjhqdENEQjhqIiwibmJmIjoxNjU3NTMwNDU2LCJzdWIiOiJodHRwczovL3FzdGFzaC1wcm9kLWFuZHJlYXMucmVxdWVzdGNhdGNoZXIuY29tL3Rlc3QifQ.GzDXaBRUAqx0KPE-WxfZVVceJll3T1RgxdTRbWPZw8s",
      body: "Hello World",
      url: "https://qstash-prod-andreas.requestcatcher.com/test",
    })
    .catch((err) => {
      console.log(err);
      return false;
    });

  console.log({ isValid });
}

main();
//package.json

{
  "name": "nodejs",
  "version": "1.0.0",
  "description": "",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "license": "ISC",
  "dependencies": {
    "@upstash/qstash": "^2.7.0",
    "isomorphic-fetch": "^3.0.0"
  }
}

I just double checked this. You don't even need index.mjs and full path. Sorry. I'm using

node -v
v22.8.0
mozeryansky commented 1 week ago

I may just be misunderstanding something fundamental here, but I can't see why qstash is required to have a different build process from redis, or why qstash requires additional package modification while redis does not.

This is my redis script:

// redis.ts
import 'dotenv/config'

import { Redis } from '@upstash/redis'

const redis = new Redis({
  url: process.env.UPSTASH_REDIS_REST_URL || '',
  token: process.env.UPSTASH_REDIS_REST_TOKEN || '',
})

async function main() {
  const value = await redis.get('key')
  console.log({ value })
}
main()

I run it using tsx scripts/redis.ts and I have no issue, and I don't need to modify my package.json. I am sharing the package.json from my nextjs app. I always test dependencies using a script, and no other module has required these custom modifications you're suggesting.

Also, your docs say isomorphic-fetch is not needed for Node >17: https://upstash.com/docs/redis/sdks/ts/troubleshooting#referenceerror-fetch-is-not-defined. But I experimented with and without it, resulting in the same outcome.


Again here for comparison, this is my qstash.ts test script:

import 'dotenv/config'

import { Client } from '@upstash/qstash'

const client = new Client({
  token: process.env.QSTASH_TOKEN || '',
})

async function main() {
  const response = await client.publishJSON({
    url: 'http://example.com/',
    body: {
      hello: 'world',
    },
  })
  console.log(response)
}
main()

For version 2.7.0: tsx scripts/qstash.ts results in:

node:internal/modules/cjs/loader:1182
  const err = new Error(`Cannot find module '${request}'`);
              ^

Error: Cannot find module '/Users/michael/repos/test/node_modules/@upstash/qstash/dist/base/index.js'
    at createEsmNotFoundErr (node:internal/modules/cjs/loader:1182:15)
    ...
    at Module._load (node:internal/modules/cjs/loader:986:27) {
  code: 'MODULE_NOT_FOUND',
  path: '/Users/michael/repos/test/node_modules/@upstash/qstash/package.json'
}

Node.js v20.14.0

I'm using Node v20 since it's still the current LTS, but I don't see how v22 would fix this issue.

mozeryansky commented 1 week ago

omg you'll never believe this: it can't find dist/base/index.js because the file is named dist/base/index.mjs.

PR: https://github.com/upstash/qstash-js/pull/174

ogzhanolguncu commented 1 week ago

If you're using a bundler or similar tool, this will be handled automatically. Renaming the "mjs" module to "cjs" will mislead bundlers and other tooling.

As for running tsx scripts/qstash.ts, I’m not sure why you're doing that. It's possible that tsx is trying to access files as "cjs" modules, which is why it’s not finding them. If you're using ts, I'd recommend using Bun to handle the compilation process, or simply run it with node index.js.

mozeryansky commented 1 week ago

The PR changes "js" to "mjs" since there is no file with the "js" extension. I write proof of concept scripts while I evaluate 3rd party libraries and run them in isolation using tsx. Does the PR not address this ticket? My setup may be unique, but renaming solved the issue for me.

ogzhanolguncu commented 1 week ago

This package.json does the mapping for you if you setup everything right.

 "exports": {
    ".": {
      "types": "./dist/base/index.d.mts",
      "import": "./dist/base/index.mjs",
      "require": "./dist/base/index.js"
    },
    "./nextjs": {
      "import": "./dist/nextjs/index.mjs"
    },
    "./dist/nextjs": {
      "import": "./dist/nextjs/index.mjs"
    },
    "./h3": {
      "types": "./dist/h3/index.d.mts",
      "import": "./dist/h3/index.mjs"
    },
    "./nuxt": {
      "types": "./dist/h3/index.d.mts",
      "import": "./dist/h3/index.mjs"
    },
    "./svelte": {
      "types": "./dist/svelte/index.d.mts",
      "import": "./dist/svelte/index.mjs"
    },
    "./solidjs": {
      "types": "./dist/solidjs/index.d.mts",
      "import": "./dist/solidjs/index.mjs"
    },
    "./workflow": {
      "types": "./dist/workflow/index.d.mts",
      "import": "./dist/workflow/index.mjs"
    },
    "./hono": {
      "types": "./dist/hono/index.d.mts",
      "import": "./dist/hono/index.mjs"
    },
    "./cloudflare": {
      "types": "./dist/cloudflare/index.d.mts",
      "import": "./dist/cloudflare/index.mjs"
    }
  },

But, you made a good point we have a typo here: "require": "./dist/base/index.js". No need for that. I'll remove this tomorrow.

mozeryansky commented 1 week ago

Remove what, the "require" line?

If I install the latest, "npm i @upstash/qstash@latest", then rename the "require" line in the node_modules/@upstash/qstash/package.json (even without rebuilding or anything) then it works. Why remove this? I'd appreciate if the PR can be merged so I can move on from this bug.

ogzhanolguncu commented 1 week ago

Removing it suppose to force bundlers and compilers to fallback to mjs automatically. I'll double check tomorrow.

mozeryansky commented 1 week ago

I just tested. If you remove the line, then I will get this error:

node:internal/modules/esm/resolve:304
  return new ERR_PACKAGE_PATH_NOT_EXPORTED(
         ^

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in /Users/michael/repos/test/node_modules/@upstash/qstash/package.json
    at exportsNotFound (node:internal/modules/esm/resolve:304:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:594:13)
    at resolveExports (node:internal/modules/cjs/loader:592:36)
    at Module._findPath (node:internal/modules/cjs/loader:669:31)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1131:27)
    at resolve (/usr/local/lib/node_modules/tsx/dist/register-DfubRCxM.cjs:1:3084)
    at resolveRequest (/usr/local/lib/node_modules/tsx/dist/register-DfubRCxM.cjs:1:2618)
    at /usr/local/lib/node_modules/tsx/dist/register-DfubRCxM.cjs:1:3400
    at m._resolveFilename (file:///usr/local/lib/node_modules/tsx/dist/register-CFO5XQXL.mjs:1:832)
    at Module._load (node:internal/modules/cjs/loader:986:27) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

What happens if you don't remove it, and adopt the rename?

ytkimirti commented 1 week ago

Can you try adding type: "module" to your package.json? tsx does not use the esm import in the exports definition when there is no type field in the package.json.

It's fixed when I use "default" instead of "import" in the exports field. Maybe that could be a better solution. I will look further into it today

mozeryansky commented 1 week ago

@ytkimirti Can you help me understand what else is wrong with the proposed solution in the PR? Is the file misspelled or is it supposed to be “index.js”.

ytkimirti commented 1 week ago

I think we should not point commonjs imports to an es module. This might have fixed the issue for tsx but actual issue is the package.json not having "type": "module". But I think it was left there as a mistake. I will look for the best solution for this today

mozeryansky commented 1 week ago

I have my package locked at the last version that works (2.6.5), but I think I'll switch to a different provider. If you could, please accept or reject the PR so I can remove the fork from my account.

CahidArda commented 6 days ago

We decided to revert the recent changes which caused this issue and released version 2.7.6. This issue should be resolved there.

Can you let us know if the issue is resolved in the new release? @mozeryansky @mittalyashu

mozeryansky commented 6 days ago

@CahidArda Thanks for reverting, I've tested it and it works as expected. However, this issue has been going on for over 1 week and in that time I've already evaluated another alternative.

I'm now using pg_cron, which I find to be much more extensive in ability, far more precise, and has no limits. Where qstash would have a delay of up to 5s, pg_cron had an average delay of <0.01s.

select
  cron.schedule(
    'invoke-function-every-minute',
    '* * * * *', -- every minute
    $$
    select
      net.http_post(
          url:='https://test.requestcatcher.com/api/cron',
          headers:='{"Content-Type": "application/json"}'::jsonb,
          body:=concat('{"time": "', clock_timestamp(), '"}')::jsonb
      ) as request_id;
    $$
  );

These kind of issues have become a meme in the js community: https://x.com/levelsio/status/1833109334357635445

If you go back to where I pointed out the issue then you can see the build in the PR didn't even pass the tests, and again in a followup PR the builds didn't pass. So, I can't have confidence in this package.

CahidArda commented 6 days ago

I am sorry to hear that you decided to change the provider. Merging that PR was an error on our part. We will try to make sure that it doesn't happen in the future.