Too many requests / 500

lucis commented 1 year ago

Hey @ije, first of all: thanks for your work, esm.sh is an amazing tool!

Today, I've started getting these errors in Deno Deploy, where 500 is returned for assets (Fresh app):

An error occurred during route handling or page rendering. Error: Build failed with 2 errors:
deno:https://esm.sh/v100/@rjsf/core@5.0.0-beta.15/X-YS9AdHlwZXMvcmVhY3Q6QHR5cGVzL3ByZWFjdC9jb21wYXQscmVhY3Q6cHJlYWN0L2NvbXBhdAplLyo/deno/core.js:2:1041: ERROR: [plugin: deno] specifier was a bare specifier, but was not remapped to anything by importMap.
deno:https://esm.sh/v99/@preact/signals@1.1.2/X-ZS8q/deno/signals.js:2:175: ERROR: [plugin: deno] Encountered status code 500 while fetching https://esm.sh/v99/*@preact/signals-core@1.1.2.
    at failureErrorWithLog (https://deno.land/x/esbuild@v0.14.51/wasm.js:1601:15)
    at https://deno.land/x/esbuild@v0.14.51/wasm.js:1243:28
    at runOnEndCallbacks (https://deno.land/x/esbuild@v0.14.51/wasm.js:1023:63)
    at buildResponseToResult (https://deno.land/x/esbuild@v0.14.51/wasm.js:1241:7)
    at https://deno.land/x/esbuild@v0.14.51/wasm.js:1354:14
    at https://deno.land/x/esbuild@v0.14.51/wasm.js:655:9
    at handleIncomingPacket (https://deno.land/x/esbuild@v0.14.51/wasm.js:752:9)
    at readFromStdout (https://deno.land/x/esbuild@v0.14.51/wasm.js:621:7)
    at Object.worker.onmessage (https://deno.land/x/esbuild@v0.14.51/wasm.js:1767:36)
    at https://deno.land/x/esbuild@v0.14.51/wasm.js:1758:662

If I try to run locally (after clearing cache), I get different errors:

error: Uncaught (in promise) TypeError: Import 'https://esm.sh/v99/*@preact/signals-core@1.1.2' failed: 429 Too Many Requests
    at https://esm.sh/v99/@preact/signals@1.1.2/X-ZS8q/deno/signals.js:2:271
  await import(entrypointHref);

Are we doing something wrong here? Do you recommend changing the strategy? Here's our import_map (after some crazy fixes for errors that were happening yesterday)

{
  "imports": {
    "$deco/": "./",
    "octokit": "https://cdn.skypack.dev/@octokit/rest@19.0.4",
    "exponential-backoff": "https://cdn.skypack.dev/exponential-backoff@3.1.0",
    "std/": "https://deno.land/std@0.162.0/",
    "djwt": "https://deno.land/x/djwt@v1.9/mod.ts",
    "$fresh/": "https://deno.land/x/fresh@1.1.2/",
    "$live/": "https://deno.land/x/live@0.3.29/",
    "partytown/": "https://deno.land/x/partytown@0.1.3/",
    "streamResponse/": "https://deno.land/x/stream_response@v0.1.0-pre.4/",
    "tabler-icons/": "https://deno.land/x/tabler_icons_tsx@0.0.2/tsx/",
    "@builder.io/partytown": "https://esm.sh/v96/@builder.io/partytown@0.7.1/integration",
    "https://esm.sh/@builder.io/partytown@0.7.1/integration": "https://esm.sh/v96/@builder.io/partytown@0.7.1/integration",
    "@rjsf/utils": "https://esm.sh/v96/@rjsf/utils@5.0.0-beta.13?target=deno&alias=react:preact/compat&pin=v96",
    "@rjsf/validator-ajv8": "https://esm.sh/v96/@rjsf/validator-ajv8@5.0.0-beta.13?target=deno",
    "@supabase/supabase-js": "https://esm.sh/v96/@supabase/supabase-js@1.35.4",
    "supabase": "https://esm.sh/v96/@supabase/supabase-js@1.35.7",
    "json-schema": "https://esm.sh/v96/@types/json-schema@7.0.11/X-YS9yZWFjdDpwcmVhY3QvY29tcGF0CmQvcHJlYWN0QDEwLjEwLjY/index.d.ts",
    "@dnd-kit/accessibility": "https://esm.sh/v98/*@dnd-kit/accessibility@3.0.1?alias=react:preact/compat",
    "@dnd-kit/core": "https://esm.sh/v99/*@dnd-kit/core@6.0.5?alias=react:preact/compat,react-dom:preact/compat",
    "@dnd-kit/sortable": "https://esm.sh/v99/*@dnd-kit/sortable@7.0.1?alias=react:preact/compat",
    "@dnd-kit/utilities": "https://esm.sh/v99/*@dnd-kit/utilities@3.2.0?alias=react:preact/compat",
    "@preact/signals-core": "https://esm.sh/v99/*@preact/signals-core@1.1.2",
    "@preact/signals": "https://esm.sh/v99/*@preact/signals@1.1.2",
    "rjsf": "https://esm.sh/v99/*@rjsf/core@5.0.0-beta.15?alias=react:preact/compat,@types/react:@types/preact/compat",
    "https://esm.sh/v100/nanoid@3.3.4": "https://esm.sh/v100/nanoid@4.0.0",
    "https://esm.sh/v100/nanoid@3.3.4/": "https://esm.sh/v100/nanoid@4.0.0/",
    "preact-render-to-string": "https://esm.sh/v99/*preact-render-to-string@5.2.4",
    "lodash-es": "https://esm.sh/v99/lodash-es@4?no-check",
    "lodash-es/get": "https://esm.sh/v99/lodash-es@4/get?no-check",
    "lodash-es/has": "https://esm.sh/v99/lodash-es@4/has?no-check",
    "lodash-es/isEmpty": "https://esm.sh/v99/lodash-es@4/isEmpty?no-check",
    "lodash-es/isObject": "https://esm.sh/v99/lodash-es@4/isObject?no-check",
    "lodash-es/omit": "https://esm.sh/v99/lodash-es@4/omit?no-check",
    "lodash-es/pick": "https://esm.sh/v99/lodash-es@4/pick?no-check",
    "lodash-es/set": "https://esm.sh/v99/lodash-es@4/set?no-check",
    "lodash-es/unset": "https://esm.sh/v99/lodash-es@4/unset?no-check",
    "preact": "https://esm.sh/v99/preact@10.11.3",
    "preact/": "https://esm.sh/v99/preact@10.11.3/",
    "swr": "https://esm.sh/v96/swr@2.0.0?alias=react:preact/compat&external=preact/compat",
    "tabbable": "https://esm.sh/v96/tabbable@5.3.3/",
    "twind": "https://esm.sh/v96/twind@0.16.17",
    "twind/": "https://esm.sh/v96/twind@0.16.17/"
  }
}

lucis commented 1 year ago

Update, after fixing the problems locally, my Deploy is failing with this:

drawveloper commented 1 year ago

@ije if there's anyway we could help, we'd love to know how. We've been using your service for a few months now with generally no hiccups — until Sunday. Any big recent changes in architecture or infrastructure?

ije commented 1 year ago

Thanks, i am looking into it.

ije commented 1 year ago

@ije if there's anyway we could help, we'd love to know how. We've been using your service for a few months now with generally no hiccups — until Sunday. Any big recent changes in architecture or infrastructure?

i updated the CDN last weekend by adding edge worker to speed up the service, that may break something i don't know.

ije commented 1 year ago

@lucis not sure what happened, seems everything works fine for me now... can you please try again?

lucis commented 1 year ago

@ije sorry, I already deployed manual changes that fixed the problem. I supposed I could go back but, as you said, these seem like intermittent problems happening with esm. These were the changes I made ~1 hour ago:

Changing esm's version for the @rjsf/core dependency from 100 to 99.
Changing @preact/signals's version from 1.1.2 to 1.0.3 (This was never an issue, no idea why did it stop working)
Changing @preact/signals-corr's version from 1.1.2 to 1.0.1 (This was never an issue, no idea why did it stop working)
Readding nanoid entry to the import_map (We had this setup before, but realized that @rjsf/core output JS was suddenly importing from "/v100/nanoid@3.3.4" instead of from "nanoid". Because of a problems with this specific version, we had to manually add an import_map entry like this: "https://esm.sh/v100/nanoid@3.3.4": "https://esm.sh/v100/nanoid@4.0.0". I think that the behavior was changed again with esm and we needed to undo our fixes from yesterday)

@ije would there be a possibility of us sponsoring this project and getting a meeting session with you so we can understand better what's happening here?

ije commented 1 year ago

@lucis thanks! or could you please give me a minimum code base i can debug?

lucis commented 1 year ago

@ije the 429 is intermittent, now my deploy is failing because of Too Many Requests for the Preact module

I've invited you to the repo in question. Let me know other ways we can help.

KyleJune commented 1 year ago

Maybe it could be related to how you pin versions. I noticed you are adding /BUILD_VERSION/ to the front of your pathname but the documentation says to do ?pin=BUILD_VERSON. I'm not sure if it makes a difference though.

drawveloper commented 1 year ago

Worth trying as an alternative

Sent via Superhuman iOS @.***>

On Tue, Dec 27 2022 at 12:00 PM, Kyle June @.***> wrote:

Maybe it could be related to how you pin versions. I noticed you are adding /BUILD_VERSION/ to the front of your pathname but the documentation says to do ?pin=BUILD_VERSON. I'm not sure if it makes a difference though.

— Reply to this email directly, view it on GitHub https://github.com/ije/esm.sh/issues/477#issuecomment-1365964361, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOZ3WAWEITKWWYSOS7RGTWPMAA5ANCNFSM6AAAAAATKLSRXI . You are receiving this because you commented.Message ID: <ije/esm. @.***>

KyleJune commented 1 year ago

I think this issue should be fixed, but I'd also recommend vendoring your dependencies so that even if the site where they are hosted goes down, your app will continue to work fine.

drawveloper commented 1 year ago

We are looking into precisely that right now :) struggling a bit with how import maps are generated by deno vendor

Sent via Superhuman iOS @.***>

On Tue, Dec 27 2022 at 12:19 PM, Kyle June @.***> wrote:

I think this issue should be fixed, but I'd also recommend vendoring your dependencies so that even if the site where they are hosted goes down, your app will continue to work fine.

— Reply to this email directly, view it on GitHub https://github.com/ije/esm.sh/issues/477#issuecomment-1365977391, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOZ3UMVM5PPWFUHGLEZNTWPMCHJANCNFSM6AAAAAATKLSRXI . You are receiving this because you commented.Message ID: <ije/esm. @.***>

KyleJune commented 1 year ago

Use your current import map as an argument for the vendor command and update your deno.jsonc file to use the import map in your vendor directory. The generated vendor import map should contain all your import mappings in it.

drawveloper commented 1 year ago

Yes, I'm just struggling with this:

➜  deco git:(lucis/use-pin-for-esm) ✗ deno vendor dev.ts main.ts
Vendored 1716 modules into vendor/ directory.

Updated your local Deno configuration file with a reference to the new vendored import map at vendor/import_map.json. Invoking Deno subcommands will now automatically resolve using the vendored modules. You may override this by providing the `--import-map <other-import-map>` flag or by manually editing your Deno configuration file.
➜  deco git:(lucis/use-pin-for-esm) ✗ s
rm: deno.lock: No such file or directory
Task start deno run -A --watch=static/ dev.ts
Watcher Process started.
The manifest has been generated for 29 routes, 19 islands and 11 sections.
Githooks setup successfully: pre-commit
error: Uncaught (in promise) TypeError: Relative import path "preact/jsx-runtime" not prefixed with / or ./ or ../ and not in import map from "file:///Users/guilherme/Projects/deco/deco/routes/live/team/[team].tsx"
    at file:///Users/guilherme/Projects/deco/deco/routes/live/team/[team].tsx:1:1
  await import(entrypointHref);
  ^
    at async dev (file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/live@0.3.29/dev.ts:75:3)
    at async file:///Users/guilherme/Projects/deco/deco/dev.ts:4:1
Watcher Process finished. Restarting on file change...

Apparently the compiler options weren't considered.

drawveloper commented 1 year ago

Yeah, I've had to manually add "preact/jsx-runtime": "./esm.sh/preact@10.11.3/jsx-runtime.js", for it to work (and then there was a tabbable import with a trailing slash also, had to trim it... weird)

Now I'm having framework specific problems, seems like fresh is not so happy to be vendored, perhaps because it does funny stuff with your local files for automatic building...

Task start deno run -A --watch=static/ dev.ts
Watcher Process started.
The manifest has been generated for 29 routes, 19 islands and 11 sections.
Githooks setup successfully: pre-commit
Starting live middleware: siteId=1 site=deco
Listening on http://localhost:8000/
Page with path "/live" not found
[302]: 3391ms /live
[200]: 30ms /login
✘ [ERROR] [plugin deno] Module not found "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/src/runtime/main_dev.ts".

✘ [ERROR] [plugin deno] Module not found "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/plugins/twind/main.ts".

    deno:data:application/javascript,import hydrate from "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/plugins/twind/main.ts";import options from "file:///Users/guilherme/Projects/deco/deco/twind.config.ts";export default function(state) { hydrate(options, state); }:1:20:
      1 │ import hydrate from "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/plugins/twind/main.ts";import options from "file:///Users/guilherme/Projects/...
        ╵                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An error occurred during route handling or page rendering. Error: Build failed with 2 errors:
error: Module not found "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/src/runtime/main_dev.ts".
deno:data:application/javascript,import hydrate from "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/plugins/twind/main.ts";import options from "file:///Users/guilherme/Projects/deco/deco/twind.config.ts";export default function(state) { hydrate(options, state); }:1:20: ERROR: [plugin: deno] Module not found "file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/fresh@1.1.2/plugins/twind/main.ts".
    at failureErrorWithLog (file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:1601:15)
    at file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:1243:28
    at runOnEndCallbacks (file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:1023:63)
    at buildResponseToResult (file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:1241:7)
    at file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:1354:14
    at file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:655:9
    at handleIncomingPacket (file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:752:9)
    at readFromStdout (file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:621:7)
    at file:///Users/guilherme/Projects/deco/deco/vendor/deno.land/x/esbuild@v0.14.51/mod.js:1880:11

drawveloper commented 1 year ago

Just to report, last ~12h were smooth, deployments generally working again.

ije commented 1 year ago

Just to report, last ~12h were smooth, deployments generally working again.

Since I added a front layer with CF worker at edge for v100 update, that may break the cache for preview builds. Now the CDN has faster response speed and cache hit ratio, the new builds should work fine!

ije commented 1 year ago

i am going to close this, free to open if needs thanks

esm-dev / esm.sh

Too many requests / 500 #477