Promotion to R2's dist-prod bucket never happens

[targos]

/cc @flakey5

[targos]

It's possible that I'm using the wrong search prefix, but I get results with nodejs/release/v20.0.0/ for example.

[richardlau]

Suspect it'll be the same as we're seeing with the nightlies: https://github.com/nodejs/build/pull/3826#issuecomment-2227580402

[flakey5]

Suspect it'll be the same as we're seeing with the nightlies

+1

[targos]

closing as duplicate

[richardlau]

Moving conversation from https://github.com/nodejs/build/pull/3826 for visibility as that PR was merged.

Summary: https://github.com/nodejs/build/blob/533d7e1fbce29f834f4946d83541719152851139/ansible/www-standalone/tools/promote/upload_to_cloudflare.sh#L45 is failing to recursively copy from the staging bucket to prod. It appears that Cloudflare have not fully implemented S3 and aws s3 cp is setting x-amz-tagging-directive which is one of the unimplemented features.

Refs: https://community.cloudflare.com/t/cloudflare-r2-not-supporting-copy-object/672982

[flakey5]

We might be able to make aws s3api copy-object work (docs), but I don't think it supports recursion or pattern matching from my tests

Simplest option might just be having two copy calls, one for the smaller files and one for the larger ones that require multipart. Nowhere near ideal however, gonna try to see if there's anything else we could do

[flakey5]

Could we try aws s3 sync s3://dist-staging/<path> s3://dist-prod/<path>? Or just aws s3 sync s3://dist-staging s3://dist-prod

Also still not sure on why the uploads aren't happening, but can take a look later today

[richardlau]

I've changed the recursive aws s3 cp line in upload_to_cloudflare.sh with:

aws s3 sync $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --no-follow-symlinks

and ran for today's nightly:

root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# bash
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# site=nodejs
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# . settings
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# dstdir=$nightly_dstdir
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# . upload_to_cloudflare.sh nodejs v23.0.0-nightly20240719a523c345b1
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/SHASUMS256.txt to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/SHASUMS256.txt
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/addons.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/addons.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/addons.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/addons.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assert.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assert.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assert.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assert.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/hljs.css to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/hljs.css
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assert.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assert.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/all.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/all.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/all.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/all.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/style.css to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/style.css
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/api.js to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/api.js
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_context.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_context.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/addons.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/addons.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/js-flavor-esm.svg to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/js-flavor-esm.svg
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_context.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_context.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/js-flavor-cjs.svg to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/assets/js-flavor-cjs.svg
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_context.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_context.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/buffer.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/buffer.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_hooks.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_hooks.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/buffer.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/buffer.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/child_process.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/child_process.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/child_process.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/child_process.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_hooks.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_hooks.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/child_process.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/child_process.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/buffer.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/buffer.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_hooks.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/async_hooks.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cli.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cli.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cluster.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cluster.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cli.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cli.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/console.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/console.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cluster.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cluster.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cluster.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cluster.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/console.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/console.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/console.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/console.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cli.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/cli.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/corepack.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/corepack.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/corepack.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/corepack.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/debugger.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/debugger.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/corepack.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/corepack.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/debugger.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/debugger.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/crypto.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/crypto.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/crypto.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/crypto.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/deprecations.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/deprecations.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/crypto.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/crypto.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/debugger.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/debugger.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/deprecations.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/deprecations.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dgram.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dgram.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/deprecations.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/deprecations.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/diagnostics_channel.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/diagnostics_channel.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dgram.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dgram.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dgram.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dgram.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/diagnostics_channel.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/diagnostics_channel.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dns.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dns.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/documentation.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/documentation.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dns.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dns.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/domain.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/domain.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/documentation.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/documentation.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dns.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/dns.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/documentation.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/documentation.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/domain.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/domain.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/diagnostics_channel.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/diagnostics_channel.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/domain.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/domain.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/embedding.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/embedding.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/embedding.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/embedding.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/embedding.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/embedding.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/errors.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/errors.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/errors.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/errors.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/esm.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/esm.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/errors.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/errors.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/esm.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/esm.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/esm.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/esm.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/events.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/events.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/events.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/events.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/events.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/events.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/fs.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/fs.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/fs.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/fs.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/fs.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/fs.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/globals.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/globals.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/globals.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/globals.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/globals.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/globals.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http2.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http2.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/https.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/https.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http2.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http2.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/https.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/https.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/https.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/https.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/index.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/index.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http2.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/http2.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/index.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/index.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/inspector.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/inspector.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/inspector.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/inspector.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/intl.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/intl.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/intl.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/intl.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/intl.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/intl.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/module.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/module.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/inspector.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/inspector.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/module.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/module.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/module.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/module.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/modules.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/modules.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/modules.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/modules.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/n-api.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/n-api.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/n-api.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/n-api.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/n-api.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/n-api.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/modules.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/modules.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/net.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/net.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/net.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/net.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/index.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/index.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/net.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/net.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/os.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/os.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/packages.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/packages.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/packages.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/packages.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/os.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/os.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/os.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/os.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/path.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/path.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/packages.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/packages.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/path.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/path.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/path.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/path.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/permissions.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/permissions.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/perf_hooks.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/perf_hooks.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/perf_hooks.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/perf_hooks.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/permissions.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/permissions.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/permissions.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/permissions.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/process.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/process.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/perf_hooks.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/perf_hooks.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/process.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/process.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/punycode.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/punycode.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/process.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/process.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/punycode.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/punycode.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/querystring.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/querystring.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/punycode.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/punycode.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/readline.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/readline.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/querystring.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/querystring.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/readline.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/readline.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/querystring.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/querystring.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/readline.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/readline.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/repl.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/repl.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/repl.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/repl.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/report.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/report.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/report.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/report.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/repl.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/repl.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/single-executable-applications.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/single-executable-applications.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/sqlite.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/sqlite.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/sqlite.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/sqlite.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/single-executable-applications.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/single-executable-applications.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/single-executable-applications.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/single-executable-applications.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/sqlite.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/sqlite.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/report.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/report.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/stream.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/stream.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/stream.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/stream.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/stream.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/stream.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/string_decoder.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/string_decoder.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/string_decoder.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/string_decoder.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/string_decoder.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/string_decoder.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/synopsis.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/synopsis.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/synopsis.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/synopsis.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/synopsis.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/synopsis.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/test.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/test.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/test.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/test.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/timers.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/timers.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tls.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tls.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/timers.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/timers.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tracing.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tracing.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tls.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tls.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tracing.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tracing.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/timers.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/timers.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tls.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tls.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tty.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tty.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tracing.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tracing.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/test.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/test.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tty.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tty.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/url.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/url.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/url.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/url.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/url.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/url.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tty.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/tty.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/util.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/util.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/v8.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/v8.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/v8.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/v8.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/util.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/util.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/util.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/util.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/v8.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/v8.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/vm.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/vm.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/wasi.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/wasi.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/vm.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/vm.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/wasi.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/wasi.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/wasi.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/wasi.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webcrypto.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webcrypto.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webcrypto.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webcrypto.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/vm.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/vm.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webstreams.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webstreams.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webstreams.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webstreams.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webstreams.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webstreams.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webcrypto.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/webcrypto.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/worker_threads.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/worker_threads.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/worker_threads.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/worker_threads.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/zlib.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/zlib.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/worker_threads.html to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/worker_threads.html
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/zlib.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/zlib.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/zlib.md to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/api/zlib.md
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/apilinks.json to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/docs/apilinks.json
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/win-arm64/node.lib to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/win-arm64/node.lib
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/node-v23.0.0-nightly20240719a523c345b1-headers.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/node-v23.0.0-nightly20240719a523c345b1-headers.tar.xz
copy: s3://dist-staging/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/win-x64/node.lib to s3://dist-prod/nodejs/nightly/v23.0.0-nightly20240719a523c345b1/win-x64/node.lib
copy: s3://dist-staging/nodejs/nightly/index.json to s3://dist-prod/nodejs/nightly/index.json
copy: s3://dist-staging/nodejs/nightly/index.tab to s3://dist-prod/nodejs/nightly/index.tab
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote#

this succeeded without errors (but I guess only copied the files that weren't already there for that nightly). So 🤞 for the next nightly.

[richardlau]

Suspect it'll be the same as we're seeing with the nightlies: #3826 (comment)

I think it's less likely this is exactly the same as the nightlies as the nightlies were being partially promoted (prior to https://github.com/nodejs/build/issues/3838#issuecomment-2239481744). But releases don't appear at all (I released Node.js 22.5.1 earlier today).

Unfortunately I've lost my scrollback, but I did answer Y to the Upload files from %s to R2 staging bucket? [y/n] prompt and didn't get any errors. But I've not been able to find the uploaded .asc or .sig file in R2.

[richardlau]

oh! I've found where the signing artifacts are being copied to -- under / there's another nodejs dir!

(note this is not the same as the nodejs dir at root level)

So that's the signing artifacts... but still no sign of promoted assets for releases.

[flakey5]

oh! I've found where the signing artifacts are being copied to -- under / there's another nodejs dir!

Ahhhhh okay, is there also the / dir on the staging bucket?

[richardlau]

No, only on prod.

[richardlau]

I think it's the way we generate r2dir by cutting: https://github.com/nodejs/node/blob/03f7f8c1b82d0fc0ef2a4e1df8157034661d2c3b/tools/release.sh#L161

e.g.

$ echo /home/dist/nodejs/release/v22.5.1/SHASUMS256.txt | cut -c 11-
/nodejs/release/v22.5.1/SHASUMS256.txt
$

starts with a /. I'll open a PR to cut from position 12.

[richardlau]

I'll open a PR to cut from position 12.

https://github.com/nodejs/node/pull/53951

That will hopefully fix the uploads of the signing artifacts -- but we'll still need to work out why the things in staging aren't being promoted to prod during releases.

[richardlau]

In terms of release promotion, what is supposed to happen:

1. tools/release.sh calls dist-promote on the server
2. dist-promote calls promote_release.sh: https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/dist/dist-promote#L17
3. If the releaser proceeds (answers 'Y') then promote_release.sh calls _promote.sh passing in the site and version: https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/promote/promote_release.sh#L38
4. Because version is set (i.e. not empty) then this conditional is skipped (meaning upload_to_cloudflare.sh is not called here): https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/promote/_promote.sh#L52-L55
5. tools/release.sh then calls dist-sign on the server
6. dist-sign calls resha_release.sh: https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/dist/dist-sign#L17
7. Which calls _resha.sh: https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/promote/resha_release.sh#L26
8. which is supposed to update R2: https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/promote/_resha.sh#L57-L59
9. before resha_release.sh is supposed to call upload_to_cloudflare.sh: https://github.com/nodejs/build/blob/d9c4ea9b1a0e609ed9fdde41976e282dc07af9a8/ansible/www-standalone/tools/promote/resha_release.sh#L28
10. Then tools/release.sh downloads the generated SHASUMS256.txt file, signs it with the releaser's key, and then uploads the signing artifacts to R2 (which I can confirm happens, albeit at an incorrect path).

[richardlau]

Could we try aws s3 sync s3://dist-staging/<path> s3://dist-prod/<path>? Or just aws s3 sync s3://dist-staging s3://dist-prod

https://r2.nodejs.org/download/nightly/v23.0.0-nightly2024072179759fa0ac/

root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# bash
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# site=nodejs
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# . settings
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# dstdir=$nightly_dstdir
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote# . upload_to_cloudflare.sh nodejs v23.0.0-nightly2024072179759fa0ac
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-aix-ppc64.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-aix-ppc64.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-darwin-x64.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-darwin-x64.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-darwin-x64.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-darwin-x64.tar.xz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-arm64.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-arm64.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-arm64.msi to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-arm64.msi An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-headers.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-headers.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-arm64.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-arm64.tar.xz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-armv7l.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-armv7l.tar.xz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-armv7l.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-armv7l.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-s390x.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-s390x.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-s390x.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-s390x.tar.xz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-x64.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-x64.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-x64.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-linux-x64.tar.xz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-arm64.7z to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-arm64.7z An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-arm64.zip to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-arm64.zip An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-x64.zip to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-x64.zip An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-x64.7z to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-win-x64.7z An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-x64.msi to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac-x64.msi An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac.pkg to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac.pkg An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac.tar.gz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac.tar.gz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac.tar.xz to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/node-v23.0.0-nightly2024072179759fa0ac.tar.xz An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-arm64/node.exe to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-arm64/node.exe An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-arm64/node_pdb.zip to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-arm64/node_pdb.zip An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-x64/node.exe to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-x64/node.exe An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-arm64/node_pdb.7z to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-arm64/node_pdb.7z An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-x64/node_pdb.7z to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-x64/node_pdb.7z An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
copy failed: s3://dist-staging/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-x64/node_pdb.zip to s3://dist-prod/nodejs/nightly/v23.0.0-nightly2024072179759fa0ac/win-x64/node_pdb.zip An error occurred (NotImplemented) when calling the GetObjectTagging operation: GetObjectTagging not implemented
root@infra-digitalocean-ubuntu1604-x64-1:/home/staging/tools/promote#

[richardlau]

So it appears that

aws s3 cp $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks --copy-props none

will copy one subset of files across while

aws s3 sync $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --no-follow-symlinks

will copy the other set of files across.

I guess if we ignored the errors (which is less than ideal) we could call both? e.g.

aws s3 cp $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks --copy-props none || true
aws s3 sync $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --no-follow-symlinks || true

[richardlau]

I guess if we ignored the errors (which is less than ideal) we could call both? e.g.

aws s3 cp $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks --copy-props none || true
aws s3 sync $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --no-follow-symlinks || true

This is deployed and appears to be working for the nightlies.

We'll still need to work out what isn't happening when releases are promoted.

[flakey5]

Apologizes got side tracked - I'll ask with the R2 team again to see if they have any suggestions.

There is Super Slurper, which is a tool that migrates a full bucket over to another and it works with R2. We can call it through Cloudflare's v4 api, but I don't know if any of the endpoints are documented publicly so I don't know how comfortable I'd be with us relying on it.

[flakey5]

Can we try just this command?

aws s3 cp $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks --metadata-directive COPY || true

Also been a bit too busy to be able to look into the promotion issue but hoping to be able to tmow

[flakey5]

It looks like promotion worked for v20.16.0? nodejs/release-cloudflare-worker#130

[flakey5]

Aghh it was just the sig files https://r2.nodejs.org/dist/v20.16.0/

[richardlau]

Aghh it was just the sig files https://r2.nodejs.org/dist/v20.16.0/

Yes, I kind of expected that. We still don't know what isn't happening when releases are promoted.

Can we try just this command?

aws s3 cp $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks --metadata-directive COPY || true

FYI I might not get round to this this week -- and I'm away next week (but if any other build-infra member is up for testing feel free).

[flakey5]

I'm really not sure what the issue is for promoting. Would it be possible to add this to this line for the next release just to see?

ssh ${customsshkey} "$webuser@$webhost" "site=nodejs && . /home/staging/tools/promote/settings && dstdir=$release_dstdir && . /home/staging/tools/promote/upload_to_cloudflare.sh nodejs $version"

[flakey5]

Hmm I'm curious of the worker aws profile exists for the dist and staging users https://github.com/nodejs/node/blob/43f699d4d2799cfc17cbcad5770e1889075d5dbe/tools/release.sh#L22 , I'd imagine it would though

[targos]

Verified it exists for dist and staging.

[flakey5]

Grabbed the release.sh and the promotion scripts and did a dry run of a release (aka just adding echo in front of each command that actually does something), and the line that promotes the assets from dist-staging to dist-prod is being called. I don't really know what the issue is here