Closed lnewson closed 6 months ago
petebacondarwin
commented
7 months ago I think this is a duplicate of #4562, which may be fixed by #4867. Please can you try out the pre-release version on that PR to see if you problem is resolved? Let's keep the discussion of this error on that other issue, unless we can tell this is a clear different problem. Closing this for now. We can reopen if it is not a duplicate.
lnewson
commented
7 months ago @petebacondarwin the pre-release from #4562 made no difference, the error still occurs 100% of the time whenever trying to view any page. I don't believe it is the same, as that is a different error message, talks about it occurring with DO's and happens when reloading. In this case we cannot even get a page to ever render it always just shows the internal error.
To add some more details, we don't have anything complex, it's just a pretty simple worker that fetches HTML pages from an Amazon S3 bucket.
Console output for reference:
$ npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7766885793/npm-package-wrangler-4867 dev --remote
Need to install the following packages:
https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7766885793/npm-package-wrangler-4867
Ok to proceed? (y) y
npm WARN deprecated rollup-plugin-inject@3.0.2: This package has been deprecated and is no longer maintained. Please use @rollup/plugin-inject.
npm WARN deprecated sourcemap-codec@1.4.8: Please use @jridgewell/sourcemap-codec instead
⛅️ wrangler 0.0.0-fd900883
---------------------------
Running custom build: yarn build
yarn run v1.22.21
$ rollup -c rollup.config.js
src/main.ts → ./build/index.js...
created ./build/index.js in 497ms
✨ Done in 0.91s.
[wrangler:inf] Ready on http://localhost:8787
Total Upload: 89.31 KiB / gzip: 18.90 KiB
✘ [ERROR] Error in ProxyController: Error inside ProxyWorker
{
name: 'Error',
message: 'internal error',
stack: 'Error: internal error'
}Edit: I realized a new pre-release build is available to tried with npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/7792075553/npm-package-wrangler-4867 dev --remote as well. However, again it made no difference.
petebacondarwin
commented
7 months ago OK let's reopen this issue. It would be great if you were able to put together a small reproduction that we could debug. It will be quite hard to help without being able to reproduce it locally.
lnewson
commented
7 months ago @petebacondarwin I think I worked out how to replicate this, as I was having a lot of trouble replicating with a fresh setup. Even copying our worker code across to a new worker setup worked fine. However, one interesting thing about our setup is the worker name is dev_docs-prod. As soon as I use any name in wrangler.toml (even if the worker doesn't exist) that includes an underscore then I can reproduce this issue. It's also impossible, as far as I can tell, to create a worker nowadays with an underscore in the name.
For example, here was my worker and my wrangler.toml where I could reproduce using wrangler dev --remote and then opening the browser:
Worker (index.js):
export default {
async fetch(request, env, ctx) {
return new Response('Hello World!');
},
};wrangler.toml
name = "test_worker"
main = "./index.js"
account_id = "<ACCOUNT_ID>"
workers_dev = false
compatibility_date = "2022-09-01"
compatibility_flags = ["url_standard"]Hmm. This is strange. I just ran wrangler dev --remote and wrangler deploy with a worker that has
name = "worker_app"And it appears to be fine. I even had the same compat date and flags and workers_dev set to false.
petebacondarwin
commented
6 months ago What do you get when you try to deploy the worker that you have above? Can you share the debug logs?
petebacondarwin
commented
6 months ago Also are you able to debug and deploy workers that do not have underscores?
petebacondarwin
commented
6 months ago I wonder if there is something broken with your zone?
lnewson
commented
6 months ago So I created a brand new account on Cloudflare to make sure it's not our zone/account (as I can replicate on both our prod and test accounts). I was able to replicate the issue there as well. Here's the full steps I took:
npm create cloudflare@latest and answer as per this screenshot:
cd test-workerwrangler.toml and rename the worker to test_worker.npx wrangler deploy.npx wrangler dev --remote.b to open a browser and see the error occur in the terminal.[/tmp/test-worker] is 📦 v0.0.0 ➜ npx wrangler deploy
⛅️ wrangler 3.30.1
-------------------
Total Upload: 0.19 KiB / gzip: 0.16 KiB
Uploaded test_worker (1.33 sec)
Published test_worker (4.05 sec)
https://test_worker.izack3333.workers.dev
Current Deployment ID: ba47e59b-42dc-4f3e-960a-37315b0e12f4
[/tmp/test-worker] is 📦 v0.0.0 ➜ npx wrangler dev --remote
⛅️ wrangler 3.30.1
-------------------
[wrangler:inf] Ready on http://localhost:8787
Total Upload: 5.67 KiB / gzip: 1.73 KiB
✘ [ERROR] Error in ProxyController: Error inside ProxyWorker
{
name: 'Error',
message: 'internal error',
stack: 'Error: internal error'
}Re creating a worker with with an underscore in the name, it seems using deploy will create one (I did not expect that). Trying to create it via the UI or terminal from the start does not appear to work though:
petebacondarwin
commented
6 months ago @lnewson - Thank you so much for putting the above together. I really appreciate the help here. 😢 - I just followed the steps above (except for creating a new account) and cannot recreate the error!
➜ temp npm create cloudflare@latest
Need to install the following packages:
create-cloudflare@2.13.0
Ok to proceed? (y)
using create-cloudflare version 2.13.0
╭ Create an application with Cloudflare Step 1 of 3
│
├ In which directory do you want to create your application?
│ dir ./test-worker
│
├ What type of application do you want to create?
│ type "Hello World" Worker
│
├ Do you want to use TypeScript?
│ no typescript
│
├ Copying template files
│ files copied to project directory
│
├ Updating name in `package.json`
│ updated `package.json`
│
├ Installing dependencies
│ installed via `npm install`
│
╰ Application created
╭ Configuring your application for Cloudflare Step 2 of 3
│
├ Retrieving current workerd compatibility date
│ compatibility date 2024-02-23
│
├ Do you want to use git for version control?
│ no git
│
╰ Application configured
╭ Deploy with Cloudflare Step 3 of 3
│
├ Do you want to deploy your application?
│ no deploy via `npm run deploy`
│
├ APPLICATION CREATED Deploy your application with npm run deploy
│
│ Navigate to the new directory cd test-worker
│ Run the development server npm run start
│ Deploy your application npm run deploy
│ Read the documentation https://developers.cloudflare.com/workers
│ Stuck? Join us at https://discord.gg/cloudflaredev
│
╰ See you again soon!
➜ temp cd test-worker
➜ test-worker code .
➜ test-worker npx wrangler deploy
⛅️ wrangler 3.30.1
-------------------
Total Upload: 0.19 KiB / gzip: 0.16 KiB
Uploaded test_worker (3.66 sec)
Published test_worker (5.63 sec)
https://test_worker.petebd.workers.dev
Current Deployment ID: a63eef52-e766-49bc-91b3-b59fbddcf5ea
➜ test-worker npx wrangler dev --remote
⛅️ wrangler 3.30.1
-------------------
[wrangler:inf] Ready on http://localhost:8787
Total Upload: 5.68 KiB / gzip: 1.75 KiB
[wrangler:inf] GET / 200 OK (108ms)
[wrangler:inf] GET /favicon.ico 200 OK (50ms)
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [b] open a browser, [d] open Devtools, [l] turn on local mode, [c] clear console, [x] to exit
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯Hmm, OK so I created a brand new account and now I see:
➜ test-worker npx wrangler dev --remote
⛅️ wrangler 3.30.1
-------------------
[wrangler:inf] Ready on http://localhost:8787
✘ [ERROR] Could not create remote preview session on your account.But equally I see that the workers.dev domain is still initialising...
petebacondarwin
commented
6 months ago And once that completed (the initialization), it is working fine once again for me...
petebacondarwin
commented
6 months ago Looking back through your logs above I see:
workerd/jsg/util.c++:281: error: e = kj/compat/tls.c++:221: failed: TLS peer's certificate is not trusted; reason = Hostname mismatchI wonder if this is related?
petebacondarwin
commented
6 months ago I see on https://x509errors.org/ that the specific error you are getting there is documented as:
Explanation Information about the certificate’s subject (an entity associated with the certificate’s public key) is held in the subjectAltName extension or the subject field. However, the hostname of the server you are connecting to does not match the subject information in the certificate.
Security perspective You cannot verify the identity of the server to which you are connecting – you should not proceed. The server is either providing a wrong certificate (by being misconfigured) or is deliberately pretending to be a different entity to fool you. Sending or receiving data from unknown servers may put your systems at risk.
Next steps Compare the server hostname with the subjectAltName extension and the subject field of the certificate. Common misconfigurations include not including server aliases in the certificate (e.g., www.example.com for the server example.com).
petebacondarwin
commented
6 months ago Is there any chance you have a custom SSL certificate chain setup somehow?
lnewson
commented
6 months ago @petebacondarwin thanks for looking into this, it's a strange one. I've asked someone else to attempt to reproduce given these steps but they get the same error as me. I also tested with my personal macbook (running Sonoma 14.3.1) which doesn't have the same env as my work machine and still get the same error. I even tested using nix-shell in a pure env to try to reduce environmental factors (nix-shell --pure -p nodejs yarn).
As a test to double check, if I swap to redoing the steps without the rename to use the underscore then it all works as expected. If it was an SSL cert issue I'd expect it to fail regardless of the worker name 🤷
Note: In case it matters, my work macbook is ARM based and my personal one is Intel based.
lnewson
commented
6 months ago @petebacondarwin if it helps to track down the issue, I found https://github.com/cloudflare/workers-sdk/issues/3264 which seemed to be similar to the error mentioned (granted that was for Windows not macOS). If I replicate that test in a Node REPL then I get the same error:
[/tmp/test-worker] is 📦 v0.0.0 ➜ node
Welcome to Node.js v20.11.1.
Type ".help" for more information.
> const { Miniflare } = require('miniflare')
undefined
> mf = new Miniflare({
... modules: true,
... script: `export default {
... fetch() {
... return fetch("https://test_worker.izack3333.workers.dev/");
... }
... }`,
... });
Miniflare { dispatchFetch: [AsyncFunction: dispatchFetch] }
> res = await mf.dispatchFetch("http://localhost");
workerd/util/symbolizer.c++:96: warning: Not symbolizing stack traces because $LLVM_SYMBOLIZER is not set. To symbolize stack traces, set $LLVM_SYMBOLIZER to the location of the llvm-symbolizer binary. When running tests under bazel, use `--test_env=LLVM_SYMBOLIZER=<path>`.
workerd/jsg/util.c++:281: error: e = kj/compat/tls.c++:221: failed: TLS peer's certificate is not trusted; reason = Hostname mismatch
stack: 1035f818b 1035f87bf 104ab40ff 1035fffd4 104adecec 104adf134 10371ac7f 10371b6df 10371d0b7 1036fa414 103700fc0 1037170f0 10288f4d4 102e5bf58; sentryErrorContext = jsgInternalError
Response {
[Symbol(realm)]: { settingsObject: {} },
[Symbol(state)]: {
aborted: false,
rangeRequested: false,
timingAllowPassed: false,
requestIncludesCredentials: false,
type: 'default',
status: 500,
timingInfo: null,
cacheState: '',
statusText: 'Internal Server Error',
headersList: HeadersList {
cookies: null,
[Symbol(headers map)]: [Map],
[Symbol(headers map sorted)]: null
},
urlList: [],
body: { stream: undefined, source: null, length: null }
},
[Symbol(headers)]: HeadersList {
cookies: null,
[Symbol(headers map)]: Map(2) { 'content-length' => [Object], 'content-type' => [Object] },
[Symbol(headers map sorted)]: null
},
[Symbol(kWebSocket)]: null
}
> res.ok
falsewhereas if I just change the fetch URL to the hyphen version instead of the underscore version it works as expected:
> const { Miniflare } = require('miniflare')
undefined
> mf = new Miniflare({
... modules: true,
... script: `export default {
... fetch() {
... return fetch("https://test-worker.izack3333.workers.dev/");
... }
... }`,
... });
Miniflare { dispatchFetch: [AsyncFunction: dispatchFetch] }
> res = await mf.dispatchFetch("http://localhost");
Response {
[Symbol(realm)]: { settingsObject: {} },
[Symbol(state)]: {
aborted: false,
rangeRequested: false,
timingAllowPassed: false,
requestIncludesCredentials: false,
type: 'default',
status: 200,
timingInfo: null,
cacheState: '',
statusText: 'OK',
headersList: HeadersList {
cookies: null,
[Symbol(headers map)]: [Map],
[Symbol(headers map sorted)]: null
},
urlList: [],
body: { stream: undefined, source: null, length: null }
},
[Symbol(headers)]: HeadersList {
cookies: null,
[Symbol(headers map)]: Map(8) {
'alt-svc' => [Object],
'cf-ray' => [Object],
'content-length' => [Object],
'content-type' => [Object],
'date' => [Object],
'nel' => [Object],
'report-to' => [Object],
'server' => [Object]
},
[Symbol(headers map sorted)]: null
},
[Symbol(kWebSocket)]: null
}
> res.ok
trueI should also add if I do the fetch directly from the Node REPL it works:
[/tmp/test-worker] is 📦 v0.0.0 ➜ node
Welcome to Node.js v20.11.1.
Type ".help" for more information.
> res = await fetch("https://test_worker.izack3333.workers.dev/")
Response {
[Symbol(realm)]: null,
[Symbol(state)]: Proxy [
{
aborted: false,
rangeRequested: false,
timingAllowPassed: true,
requestIncludesCredentials: true,
type: 'default',
status: 200,
timingInfo: [Object],
cacheState: '',
statusText: 'OK',
headersList: [HeadersList],
urlList: [Array],
body: [Object]
},
{ get: [Function: get], set: [Function: set] }
],
[Symbol(headers)]: HeadersList {
cookies: null,
[Symbol(headers map)]: Map(10) {
'date' => [Object],
'content-type' => [Object],
'content-length' => [Object],
'connection' => [Object],
'report-to' => [Object],
'nel' => [Object],
'vary' => [Object],
'server' => [Object],
'cf-ray' => [Object],
'alt-svc' => [Object]
},
[Symbol(headers map sorted)]: null
}
}
> res.ok
trueYeah, I just can't reproduce that error. I am running a Mac M1, with the same OS version as you, and same node etc...
petebacondarwin
commented
6 months ago I wonder if this is related?
https://www.digicert.com/kb/ssl-support/underscores-not-allowed-in-fqdns.htm
petebacondarwin
commented
6 months ago And I am not able to reproduce because "I" am on an unusual network setup (i.e. Cloudflare's)?
lnewson
commented
6 months ago Maybe? I was also wondering if it was location based maybe (I'm in Australia). As an extra test, I also tried to replicate using a docker container (node:21) with an API token instead but wrangler dev --remote appears to work as expected there. What's interesting though, is if I try with the miniflare example above I do get the TLS error still.
Unrelated to the above, but as it does seem to be env specific, as a long shot I tried using node via NVM instead of homebrew, but the error still occurs. If it helps with env setup, here's more details on what I have installed via homebrew:
[/tmp/test-worker] is 📦 v0.0.0 ➜ node -e 'console.log(process.versions)'
{
node: '20.11.1',
acorn: '8.11.2',
ada: '2.7.4',
ares: '1.26.0',
base64: '0.5.1',
brotli: '1.1.0',
cjs_module_lexer: '1.2.2',
cldr: '44.1',
icu: '74.2',
llhttp: '8.1.1',
modules: '115',
napi: '9',
nghttp2: '1.59.0',
openssl: '3.2.1',
simdutf: '4.0.4',
tz: '2023c',
undici: '5.28.3',
unicode: '15.1',
uv: '1.48.0',
uvwasi: '0.0.19',
v8: '11.3.244.8-node.17',
zlib: '1.2.13.1-motley-5daffc7'
}
[/tmp/test-worker] is 📦 v0.0.0 ➜ openssl version
OpenSSL 3.2.1 30 Jan 2024 (Library: OpenSSL 3.2.1 30 Jan 2024)I note these difference in my output for that command:
nghttp2: '1.58.0',
nghttp3: '0.7.0',
ngtcp2: '0.8.1',
openssl: '3.0.12+quic',I wonder if this is related?
https://www.digicert.com/kb/ssl-support/underscores-not-allowed-in-fqdns.htm
So I think you might be right here. I just don't know why it's not failing all the time though (is there some branch whereby the following doesn't run?).
Looking at what the workerd code does quickly, as best as I can atm, it uses capnproto which uses OpenSSL to verify the connection: https://github.com/capnproto/capnproto/blob/64802802df1d7780625eeb07b71d249fe49fb68d/c%2B%2B/src/kj/compat/tls.c%2B%2B#L218
Now, if I use the openssl CLI to verify the hostname I get a similar error:
openssl s_client -connect test_worker.izack3333.workers.dev:443 -verify_hostname test_worker.izack3333.workers.dev
...
---
SSL handshake has read 4476 bytes and written 421 bytes
Verification error: hostname mismatchwhereas when I use the hyphen version:
openssl s_client -connect test-worker.izack3333.workers.dev:443 -verify_hostname test-worker.izack3333.workers.dev
...
---
SSL handshake has read 4476 bytes and written 421 bytes
Verification: OK
Verified peername: *.izack3333.workers.devThis fits on my side too... We use Cloudflare WARP to secure our Internet connection. When this is on, running the command from above I see:
Certificate chain
0 s:CN=test_worker.izack3333.workers.dev
i:C=US, ST=California, L=San Francisco, O=Cloudflare, Inc, CN=Cloudflare Corporate Zero Trust
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA256
v:NotBefore: Nov 2 13:42:16 2023 GMT; NotAfter: Aug 23 21:10:40 2024 GMT
1 s:C=US, ST=California, L=San Francisco, O=Cloudflare, Inc, CN=Cloudflare Corporate Zero Trust
i:C=US, ST=California, L=San Francisco, O=Cloudflare, Inc, CN=Cloudflare Corporate Zero Trust
a:PKEY: id-ecPublicKey, 521 (bit); sigalg: ecdsa-with-SHA512
v:NotBefore: Nov 2 13:42:16 2023 GMT; NotAfter: Oct 30 13:42:16 2033 GMT
...
SSL handshake has read 1694 bytes and written 421 bytes
Verification: OK
Verified peername: test_worker.izack3333.workers.devAnd when I turn it off:
Certificate chain
0 s:CN=izack3333.workers.dev
i:C=US, O=Let's Encrypt, CN=E1
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: Mar 3 22:55:43 2024 GMT; NotAfter: Jun 1 22:55:42 2024 GMT
1 s:C=US, O=Let's Encrypt, CN=E1
i:C=US, O=Internet Security Research Group, CN=ISRG Root X2
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: ecdsa-with-SHA384
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
2 s:C=US, O=Internet Security Research Group, CN=ISRG Root X2
i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
a:PKEY: id-ecPublicKey, 384 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
3 s:C=US, O=Internet Security Research Group, CN=ISRG Root X1
i:O=Digital Signature Trust Co., CN=DST Root CA X3
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
...
SSL handshake has read 4476 bytes and written 421 bytes
Verification error: hostname mismatchSo I think that Cloudflare's SSL providers are more lenient?
petebacondarwin
commented
6 months ago This seems to have come up before: https://community.cloudflare.com/t/cant-put-underscore-in-workers-custom-domain/483048
petebacondarwin
commented
6 months ago And I even mention it here! https://github.com/cloudflare/workers-sdk/issues/833#issuecomment-1127496133
petebacondarwin
commented
6 months ago So given our investigations, I think that this is basically working as expected(?) I guess the follow ups could be:
petebacondarwin
commented
6 months ago https://github.com/cloudflare/workers-sdk/issues/5223 and https://github.com/cloudflare/cloudflare-docs/issues/13355.
Closing this and tracking there.
lnewson
commented
6 months ago Sorry have been away. Yeah that sounds about right, thanks! The 3.19.0 release is still somewhat of a breaking change for some edge case users such as myself, but we'll have to live with using 3.18.0 until we can safely replace the worker name to remove the underscore 👍
zehawki
commented
6 months ago Phew, I also hit this issue. Suddenly started facing an error with wrangler dev with --remote (ie turn off local mode). Any POST to the worker results in:
503 Service Unavailable. Your worker restarted mid-request. Please try sending the request again. Only GET or HEAD requests are retried automatically.
Any browser open results in
✘ [ERROR] Error in ProxyController: Error inside ProxyWorker
{
name: 'Error',
message: 'internal error',
stack: 'Error: internal error'
}Whereas everything is just fine and working as expected in local.
This was a worker than has been extensively tested with --remote earlier. And sometime in between the last testing to now, Ive upgraded wrangler to 3.29, and prolly I was < 3.18 earlier.
Now I understand it may be due to my worker name which is indeed using (_).
Question - What is the impact of changing the underscore to dash - thats essentially a renaming of the worker - and deploying? Will it create a NEW worker on a new path, or will it seamlessly replace the earlier one and everything continues working as before?
petebacondarwin
commented
6 months ago If you rename the worker by simply renaming it in wrangler.toml it will create a new Worker, leaving the old one behind.
Instead you can rename the Worker in the CF dashboard.
I had a worker called test_worker, which I managed to create using Wrangler (the CF dashboard does not allow this).
I logged in the dashboard, clicked on the Worker -> Manage -> Rename Worker.
This allowed me to rename the Worker (the CF dashboard would not allow me to put an underscore in the name).
Note that the workers.dev subdomain will change accordingly, in case you rely upon this.
Note the warning that is given:
Now I would need to go and update the wrangler.toml with the new name so that next time I deploy from Wrangler it does not recreate the old Worker with the underscore in its name.
zehawki
commented
6 months ago Thank you for the detailed reply.
If you rename the worker by simply renaming it in wrangler.toml it will create a new Worker, leaving the old one behind.
For future reference, there is no data loss in this method right? Since the bindings remain the same. Its just that the new worker is on a new route - ALL else remains the same. I just have to switch my custom route to the new worker, and I'm done.
Which Cloudflare product(s) does this pertain to?
Wrangler core
What version(s) of the tool(s) are you using?
3.26.0
What version of Node are you using?
20.11.0
What operating system and version are you using?
macOS Sonoma 14.3
Describe the Bug
Observed behavior
Since version 3.19.0 of wrangler was released we're no longer able to run
wrangler dev --remote. It runs, but as soon as a page is viewed the following error is thrown:Expected behavior
Work as it did in version 3.18.0 and lower.
Steps to reproduce
Due to the code being internal, I cannot provide an exact example. Happy to provide any debug logs that might be useful. The error happens 100% of the time by just running
wrangler dev --remoteand then openinghttp://localhost:8787in a browser (or pressingb).In case it's applicable, we have these compatibility settings in
wrangler.toml. In saying that though I can replicate by removing both of these.Potentially worth noting is that I've added a
console.login the exportedfetchfunction to see if our worker code ever runs but it doesn't appear to (or at least the log message is never printed).Please provide a link to a minimal reproduction
No response
Please provide any relevant error logs
I've redacted some parts that I didn't want public and replaced it with placeholder strings (e.g.
<ACCOUNT>).Here's an extract of some extra logs by adding
--log-level debugof when the error occurs: