search

nodejs / build

Better build and test infra for Node.
507 stars 166 forks source link

Discuss state of r2 migration on Build WG meeting #3508

Open MoLow opened 1 year ago

MoLow commented 1 year ago

Opening this issue just to track this in the meeting. we are pretty much done with getting https://r2.nodejs.org/dist/ working. we want to make sure we did not miss anything and discuss the next steps - mainly when we should pull the plug on the migration the Cloudflare team asked for an estimated date so they can prepare (they are depending on https://nodejs.org/dist for installing node on their infra :), they want to avoid circular reference issue )

CC @flakey5 @ovflowd

richardlau commented 1 year ago

Opening this issue just to track this in the meeting.

FYI unless https://github.com/nodejs/build/issues/3503 changes the meeting times, the next scheduled Build WG meeting is in the inconvenient-for-everybody slot on Wednesday 18th October. The next meeting after that, in the earlier slot, is on Tuesday 7th November.

I guess one question is whether we want to migrate before or after Node.js 21 -- usually a semver major release results in a spike in traffic.

ovflowd commented 1 year ago

Opening this issue just to track this in the meeting.

FYI unless #3503 changes the meeting times, the next scheduled Build WG meeting is in the inconvenient-for-everybody slot on Wednesday 18th October. The next meeting after that, in the earlier slot, is on Tuesday 7th November.

I guess one question is whether we want to migrate before or after Node.js 21 -- usually a semver major release results in a spike in traffic.

Oh yeah waiting till November is definitely a no go IMO. It would make sense to discuss about R2 way before that.

mhdawson commented 1 year ago

I just got an error when trying to hit the link mentioned

image

mhdawson commented 1 year ago

Does it make any sense to switch over a portion of our downloads first to validate on a smaller subset versus everything. Is that possible/worth any additional work.

mhdawson commented 1 year ago

When I hit the page again, it now seems to work instead of the error

ovflowd commented 1 year ago

I just got an error when trying to hit the link mentioned

image

@flakey5 ☝️

flakey5 commented 1 year ago

I just got an error when trying to hit the link mentioned

Looking into it now to see what happened

Does it make any sense to switch over a portion of our downloads first to validate on a smaller subset versus everything. Is that possible/worth any additional work.

As in, for example, requests going to /dist/... hit the worker and all requests going to /download/... hit the origin? In that case it's definitely possible and would just be up to the routing config. For deeper paths like /dist/latest hitting the worker while /dist/vX.X.X hits the origin it should be possible, but I'm not 100% sure on that.

flakey5 commented 1 year ago

Believe I know what the error is, should be fixed in nodejs/release-cloudflare-worker#44

mhdawson commented 1 year ago

As in, for example, requests going to /dist/... hit the worker and all requests going to /download/... hit the origin?

Something like that. Roll out in smaller steps than a big bang. Do you have a suggestion for a first good step?

richardlau commented 1 year ago

I'm running into errors trying to promote Node.js 18.18.1 😞:

# Creating SHASUMS256.txt ...
upload failed: nodejs/release/v18.18.1/docs/api/all.html to s3://dist-prod/nodejs/release/v18.18.1/docs/api/all.html SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/docs/api/all.html EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz?uploadId=AK5en5fhbO8ZsJyjv_fBucFKF_nxOZWmnYVx8Q3garrFPWRyq2q8drTOSKCzgKUhPajKU0Amnxyb2bP2ibtu23YtoUzUWurgo-eyOrTCGb7-r84FCEeUXsm9nK2fPct0NOWqy39ojE_p7kl2lGHDPl76k0bmcmHag7Rz-srSGElHAqv-EdcbuQZaXdZQaYDidov2W1Oi3W1FvPMrrfF1Jw6aqMaNspg1MH6ESNXsyVdovcRmlDS89m2oSX2-zr_CiirdKWSA8o0X0bWPNWS2zoXujri0NFD7yT39wKmrGCbGG20JkHj5a1C7WnTmtFoJNuV398LIyXsQQbeCdxxNKgI&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz?uploadId=AH2Q3Vic8_76OWmwnrMacUFjhcSDZH4Q5BH-k9Ox65vWGdaedlgo0IMLJra90QGpY-VFwr6MzXIRaganfleHJK_-vfcD-KxuBW3Pz3M3edpEqJaE-2qc3Y_inkMdP8i_gvN4atAFEybBvGyYEaw0z999v5miTeym7udzACkhiJGGJcv97e4LrFL9K6vZ3KjCyidIZDNrTgcUfGJAWYY9JmgMveCwjJ6OuZkZYYxCPHVpgNqLeAuFr7gTBFU8AvrQB5DGoVaKve-ElV1zqPIUDggDeyNv5prkhIPvEEv-gOT4dduZSxO3VbqTjvXrGYrSHtn7bjW4QZkmLgCaDQbnJHo&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.xz An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 2): We encountered an internal connectivity issue. Please try again.
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz?uploadId=AHpycjrfMp3Ou8gBMPUA88nOYF1d23N-rQtud7r-4Q2nvFp9Y2HDYhBzCl76vriPZRJx4bIr--HetvrX_pyEBKaq2Md1QsDUvmH69wAYRnajN9I-v5I6sQrZbX5vv_FUxwA8Oj2qWTAL6nKZz-KFH5SCpSvtGe_i9YALGX2m7aata16Up0NtaDynsL_vaT_FQZxh6M9aOpO1mv9ucbxsOsFwXZhZOC6ZJhscmyGPUH47jwvJDwXZVGJTQAG8QvKjB4uZQPZyn4aJ5_EtupYx8K8X6yKK-3cTgyraNlFX9SWtn022WQiq8vfoJQWmqkwad2_qaee2m3NCIFLK2YXtBh4&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz?uploadId=AD0two2TSNT8hnMWVnQOf4M5G-xs2gouFzsq33f5bMnQKRgg0pKP0jF54WSjAvZ3a0vbyTGzDmVPFpuzhWzqc84hzrL6BlGn3yJa8qkxUNlPQA2TZ9b23KjSlcIixVEGpyWKtkADiZAWEpyYjOzTySL181clydXJAMMa_ewH155cVd_m6ysMSIYN5ltYOhsSfxYh080uL_AhymJCVlDfm7SBRLIRFOOXeOpmYGsHkCmLgfWZLNCPqAtIFqNd9YKIeo1GgQnPiDyDw4Fs4XjALtQu1hyk-qqkLvI7hW_XmpXmhm136441N6XhlSfRW0tmvV6yhtk81uLo8GtIEzseOiU&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-headers.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-headers.tar.gz An error occurred (502) when calling the UploadPart operation (reached max retries: 2): Bad Gateway
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz?uploadId=ABMnDebWVxhBc1k04Wcg99_ZuWBywvH307mvd1RGbzBQ-Nl9Cd8WcOFPgqrceIebh0o8OfNfLQxbWSEgf0FcSJYFZwl9RBG4VaAOaz_kUCvpqyWh6rjc9ovLQZ7XwBgHxW8qk-aKrdjAK5FuAu7h6Eo8Kh5Q8MdtRcd9xaTTBdAMWiDNrUnGFdnCjhw53jW9tUEt1lMevfOKlvjkAI9_kT833WFcCwdacQBF1xAsKC9ugXPRhWQbyeXrOCxZvTQzxFbYVRaCFgrWvR6-wCaaae4amq7cYqL0EiP7DIUIRLvWvYZhKsvx5xDXdgLA_z20tr0pjedEN1TK6xeQQqu5UIw&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz?uploadId=APNla1tU948J1WlUYujXN074miBtOynCvXW_pEHKPz33APXaSBXYPv68C5aYgawgnL4EV_T_ziGWXPLnMPD8fLS8vPnxStqmIUI8HR9_d62NbT1oXH0-sRSm3s7HVYnf8QlKK2xmQPr4IFqspoWUgG-vbEegPG1f3qrYajeSx6oOmGu9H5mTi893neyUdcqIkI--VWUtxTCfazc9lSBeIlT3qFKGBlLd_lnj2iPCrbACchTTE46RI5OxDQzfL9Hy7Wb9ZGe-lg8TnTEg7w9NKMkywUTGa_d5PulEEq0wdSNKOtXd_GfXdpXyTwbziuR6BI3vhPk5gAe67X27b9Hl-Jo&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz?uploadId=AIwdPt9ASHPR_MWigdIMmSVw177O9XCiEOxWvAFjsr_4EipKvxjx46ylZXRo7WL4m-ubBC1vBAo3V33m8WVu7C4_Yd7WCkMEltL23-FMyiVJg_jWmY2wOwBpFoeXn2VlTio_4O2JS2s72miIlWFdRZE6D_DrzBzQFJ65i29ubGX1Xl31Nf8ZDex0_E31pajt2IuO6ZR-8fygJS5Q7qGCjNzM7U2uBTt3Y4gcxZPFH_uE-LTHBVCzsA4-8bWqBWDpLzkDBCG-05GjlUfsvRQyku5vXNTvYTVQ2vypFuSnfQYACXHHb2w7zed8Gq1tzXqX6nAp0hqI89_Wl_XnRd8xvM8&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz?uploadId=AMlL9loRZNEFKsqDwdnyjRu_0Jnvw29aDfoZI2nAxWQtM8C95V5Q5LZ76xP13PIRGoCOB0bIRZFJG6p66tF3cO8fknqJ-bLjbzE-XEUGmnCplxMI0RERotWU9BgUDLqF2DiFa1qt8xIuZ4MvO1YjtXGlhK4ESAZYqHY5vdY9GaTFTHAIEX7OiqFO07oaLmQ85dGhIfj0PHI_johOm-RL705XuKo3ErIL45vc_9PdNExbae175azbJCJqBbiczwypHCz5lf0310Nhpo_HTGnN7Cj2qfVx22_o7htu-eMtrzDd5hYE5UbuTLE5EGi8FJIDMQ7_kY6i7WXG5zll_d4tEKI&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz?uploadId=AD-QybSCbx5AtyxVFP_zOUXmdCl05J95Oyp1QFB_W6lf1vmlXOZjpdKAYd-4L_4FTLyjo34oX4rDdBnfl-BoPUzzcjQ4wCxmRARrDNErNZXh8WbAJg6-PQzrVYUKf3EqAP8N_8yWckdxrS0DiQ494XqNruNOVwvKVxPboc2onfJbWW_ZgZT9O9PcvdXuSaKsBI13Ai_G2wJYsDb_wX7Cs0QtfNudADtFaZ5Fgzk48SK83SrO_f9GbHWxHab767TjAqM7hXsVzTH0T3Uh8N81Qkcq7AgbeteYEWeGZ2D-iRcWE0SnL3jRAtMFITG_qxCy01tMShYHDFb2oAsMOc_Olng&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz?uploadId=AAwOyFGuwNTBenVpMm0focOuZcnDW2g2d8M1FsVPr4FmPOjDRl4-Tv0ecly20EwmTfgYWGMqWwHxnR8472eQmadv-sG_KUXU-HRsndQKaZtihGUc6kbXE_xrR2tVYRQvRZvaKtsOFS3zfovVkB4-GHudNOqVGhNYyAHdkV1CM3Smnl8O2JCR5RO2Qfr0LE02iTaMk3vXDnJf65EoP41QBiI5KJXgiWkgJyMhor9WL31mDwVGZm8kOOdgBDOB_LZQF2qCk9LYWSCAy_8se9K2WtKqwWjkAn7XR2Xy9-W0eiSCg5DwJtHAswn3moQal2MPBVsd0OdZhmXPis7W2Nd0nWQ&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz?uploadId=ANEZzsnpRXUgADvxhKSOUYXW7t7hxbJ8qxUTqTT7C2YHpgAse6Zn_e3BGjDxZUsMPLqfSYdHXXyy7IuWw5OhqFIGpnExWcKufbbx4SYcrYEDiZApoFHWwDUcwnmkJ99IMCh74dnL7W53-EkZuP-yGpDur3L_X5I9UUVjPkdKLgNI9Eno-uhcuv4UKcuHeLmE9QeivLTMO8RRiuvSqP0DCygRWxgnpyePwD7nQrh2JkE2IAt8D9yacsRxVY0oC6KTDDzgj4ttDIkz15qTXiVbhWPF0e6vzoLMgtWgFYZrPjIexCqkbm5RA6qNAMj-qxIUw6fxwulYTFtaN_zqDtOrvvo&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz?uploadId=AB0nJ81_nbe8AA148NLE4Upjz0ASRbpbhIQVzEe3E24H6xKq5QiNASuuto5fKmsoeWW6VkQ0HTYll-EkEM9jtNMe-N3uOXjSfMrn2k0Za5hsncZv67kxEDbK1_F6coNcGI_cbQ_m6Un1VmLflRDqSPEM4kxZrQ2pRArEsfLo66t3FPEQdQtngUm1IbxIz5LDivhSEIKnL2FrQw1av0_1qRR9X4MMCueOo5WJF803JmE5df94NPWOle5Sv4NxI3o1dPPj4IFbmYgE6hEcmtC7kANhOSnIlKm5XwToPOwOC60O0YNWw0fh0EczU5GIYufceUKmo6S1KqBqtVgqURkw93g&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z?uploadId=AA-QUfH57CHLUnp4zMoteTRVdHCm62UUaGR1ThGNmJ6J5YDGsjxbKE1xGTbT95_u-T2ADPSqOyH69wRDwI4XEDWxm_Sw411XiuzpXfZeDREctWd_aJnLIcSJXrQtUkyn0FSWkbtMra6koyKOrM_d-rf2GDDMukpYfo6XYQheM04frt1NjefGOnP2sfanSP_Uys6ghF2iq3427OqfjJueTrIqRBhpuJY50Vi53ZUKAjgWiTYwtHtza3xFTryYv8jQ45oqOnmfS-UwLVTxgebJuDblOEupILAMfLTEN0eMLBFV0D-sDocHhgPguO2yxv4dUmuujYruXMN6ckAYawkPTuA&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z Connection was closed before we received a valid response from endpoint URL: "https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z?uploadId=AOJKmxdqJZpGbcdtMJZR0A4rJwZHQsOsBB7XWGnOh6-CQ0Xh8kLWXfr-gtf8aKszYSdssvfXaCWOE27QY0Y5KGMejyQpQHSvrnZ6_hEl3mO8-MdYAy0PFtfqIBo9HfKlohlXhVv3-QeWDXc_PbicVyydWbDnCiUJERwxFmkLPRqzWWfBMTlid151dqgD_WoXgpD-quF9OD2h5YcgJ_-xRd5orGJ-bAAuD-5t_DD-a_oLJ3UPB1oZ5mLXsYcBTmibIxa4fMj4Uhs0meYAiC3Eaajh7Vx4sDP3lWH-0YlFB6SVr0TkylKWpmbY2GR0VkZgb-H3GseeTAJGuKj4YtJ0FZA&partNumber=3".
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip?uploadId=AAqI5JU0k7ReQ25MvcfNJaUi394xfgSlaL5ly2aKUaUiKVAhJPLiBULxLhUQ4rYvSw9I4Bf_8klqsz66njzUA2YfioWH9-s6tUzs6fPGnGhqlF2cs_v2usDXBHYDhqisiAkoNEg5lQCbo6ljVC-B0Saw2zY9jWE37ajXtO_fcI6zdDhAmRSnG9C6tTIx9UY3mFN4i_TaqXcu-fn1LEfNFr6ZKnuk7KUn_g5iX72bomuHzPe9ad3jimx5JJwQ9fnScOKdyL8_P2hxPXRpWn00Iej0Gssu-ljMqXBdlG4bKsS_On4ajXOfB-fR3EN0C8AacJt1CQzLoR5KJybGEU_yEwo&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip?uploadId=AHG6FbmDzD4q4xehQgIRvTjTrsOsEQyFX4YVfVM0m3QPrzNtBts07XqdqRRNeWDA8gMp8lSYD2MzHC5kvpGRONr1JsPuqn8oIFq3ZNRlMU2I0dI57SfzWCPmPRhW7_zPn9_iutokUyHIaDPuj8kHhbM-oEXdtIi-gPo_aG6Xs3cja9iyfhH1VtA-t0TX89Po5XDjLfoc5U5no9TJsB4XXGGnWn1A76PoQuoc37ytb49DqSp2Hekte6rgN2ZLhglQXweap6qPqsMGAaFXUKv-ZMbw8wAe6Lwo22Gwi4gDzq-EtZUFyo6-rbaRf1xtB4I1-kGe5_63hSWdsKOM5uVg97U&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-x64.msi to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x64.msi SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x64.msi?uploadId=AP4c7XUSrPT5bxe2-O_NMN2I45wgE_UFZM94P_lN3Y17FjXmFG_llHaCObiHOmFyTxwyMvCSKOziSIPEa_4puQaaJ2vSyhTdIRo3AlrME9zcOJ0glO4CuQ3KeWz_WfAAEbECldzqIdw3cxuwhpnS3Jj7VPUwtoMFpBgDqi6_yaVE_vNM2Nd6d42jYaQb8f7tQruH6KgqUcXP9w8A3FFaeLkP1-qimKjpifH4yDbjHWgjcnwoCh24e5pEgusRPq7UM65p76pVQcuXqog13GXMErRbMILeEGEe9MLdZ9Cb9eLdiuZlhABrBLXRkD73ebEm-w-1YO2YhKEHyfqEZDlV8no&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-x86.msi to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x86.msi SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x86.msi?uploadId=AOQzX2Yx7ZNW_rN9FPslfmTV7YYQoOclj-ONBSslE5Poy0GVTxMObVa1kAl9RqqabxX6r_uptKjwWGaAGzN_DLAXV9A-JjvFW8i9d2xuBPWC7yO8-l-p6rtwTTjT-yu4YrVxjb54-gz1mBQDtXBj6VOFb0YzglG4MP3BnUSS6wGprnqMxUo5kZgVW7S4qWhqsG8f1fYlxtBKFp8tLaDp33QRoofnOJgxQui7jiAyF8J068U8cq2SmdJqulmR_B_biR6V7wdYo0LetMU0ZW5HV9-zXQYd-b5pIR6wT1pK_6NeoHk1-K0LDzCE264VmLtCFJiUnll6gJtOrPDS7ysZC1g&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1.pkg to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1.pkg SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1.pkg?uploadId=AO6MuwqBn4EJDVxIKoUum-rAvZaXF8ETUfFLcu4gtm271pga0BuoEnEUC-Sd8ZRL2MFUE-i9XlCB2KMEN2B2Iji5D_hs78Nx1NSmDKKdYZSNfWACxUbl2K8SBbJmBUfRiGSLJv5fk0ui8ne9qOvllysIBy4MODDs4bbf2zgc3WQPNV_U1LUAxNbkW73NmQmAie1mnW5spoP7FQQgh4q0nFdYW27niJihu7OEGMxDS9eLwQptxrcFbSxajpTn4qja_Y8QJUbiDAu_spk7WmCkDAj78l8Ap1-d61_xpwsPMKwUDPQCTr1jXix-HmRFfAbGt5UW_Hph5MlJa2QTxRf9O1c&partNumber=9 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1.tar.gz?uploadId=ADBk6NL9Q2ehez8_McAh1Mf7CR4zyh1QK6kPXYmpj9MDU7Jz0HalsCPX-6mfqTSGcct3FBdnDSbadj3EuB1uIEUo8vdi4bzQvEuGSUsjeTvEjVdVBRZtQ40b_CcIVXZZnQJ7tBj66uejgkMt_zkpkqkgvyQNI6FNeWa0K9c1t8t6BzxBDRIPrq5Y-BkyzDpaUDi0KeELWe0mT9vXeYGTtLReoN3oHR2aSsmr0weiLE7agGf2bhW09EWaPJ8PY8DxrjEVxD5WpnGxipRTaNs4S8XX6O3vK7d9YdlcXxo49CZ0eR32ze8n3LC6o1W7fy6XSyysY27YjWcwkLTrO6dzJmY&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node.exe to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node.exe SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node.exe?uploadId=ALfJRer4UbzGgzZAAC8E447xGQBu2Jva9QUTLsObZH6BoPzSxx9PaSK9efBK-gkyZZUsj05NjUwuqznjS6ilQufuVQ4AG-jbAeVyfPNNuN1i5SabdlV0433dg38nJIIY7-OL9gUp-f18nkPULFmLeKvoMObMzV3BioYLKMJ9JZpMPavvahzupJBI__i1k5XWofooRqb3wVOsMvcXV49uT5ZZxsitP7hW5cIjZSz9mhDrfh4RmVUgAI8tHHyhGDPBIACqby_HB2ChdlMaCpUfgMOS8S22TiJ4DSavjiXqAT4qo7Iw4FITgcs-r6n_x3AvGJg4B8FgNR62NhuUHYvXL60&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node.lib to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node.lib SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node.lib?uploadId=AON6QbdBHLTVcjEC5SOke9-pPSp9FYvUZED-5SWbFU3Y-npTbO1Arj5_k0FP0Lapgfkg5l5pYx6OAZnTtVla4RMHfk-wdKMPIVtt6fUDkYLcfqIeBoCQVf5Pm8fJra-YY6qJsng1YrpjKe9xi2lSepRbChKOvOB75cuHXoyFLXDFEa0xQCAkR-aMhans_W_Cad8YqH3gwVLTJhyCS1kStiIWvluKOuRx9zt7qpIkQHcP5v5fTJWEPyBLMM3aYWeoUahni1JGaVBgkvY03fWkBMQWmc5Jckl4sc0PtBAU9el5RfFpPxdqhgzIQQyUTDHrNU8DGqI0ruROAf6IQzuDbbs&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node_pdb.7z to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.7z?uploadId=AA5kO1d2ZgdVAlWKzAF3NGddqb8WIGngKT8D4VrJxExlYz2B2A7NaHYOqFUzzMi9aWgA3ofzf7cf2jjT5JhrXytavKVsR0QBv3l72ti2E04xFnjhJQNWYSfFzgzH6dYwQtlxC05KAFjOWrIkU5NPge1CBuTpYtfJQgMaYGYh8yjH9Y5YP4bCmnhlnKB6nE9C-rvQ8cfc79-hvxPMelBJWP8gsQx6D8V3tBGrPboIQKUSZMt2BxjxaRvQxbR44wjzp6x_ohxV4t7XFbJdGpKAJDVpClzHZ9L0dxG8SBVFB7GD9fRXg36rafyl3Wy07oDHNbNREtHDpVLDJcbcyDRTANk&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node_pdb.zip to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.zip?uploadId=AH8ieQkTJSvGmxBlv6PbrBh1NjuAXibvLkqgvCa_13-pCedouLVWSccfKhd2hDc-IeNQzNuB19Z2DY-7zh3K5i5ULhGW11_xPoQF4KbwuCaE3y2sBEk-Ylg5E5CzIY3RhJfyd2zA1KnjrrVE29o9qPtt-Lyss6IwXWDtdCcFu0bNjh-fcen5-iZXq4fjg1Ir75j8K2dZZGlLhsp9e5BvOVcXsPghcBkK5vJBSgho5DktNcaUi8YeoSAsGyxNSvAfjyBMw5SM0tXGTf_0QO-KNlWkExvAPsC-7EpQA4iyTTVpU2SV0wUfwiH5D2P6jOgQxvCkQojycFCAps-962m2-v8&partNumber=5 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node.exe to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node.exe SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node.exe?uploadId=AHnm_19Q4ivEGat7YNepjZOnNDPl6CSp5Npjev_9XgvK-Kbmp_7eGtTyp7X6qd26LA_Bvoh3eEERG4kjrN1U6ouSb6vMBXtjTqdsaSZzr49c98ca9p5eNWQ7NeBnbD99F3y_y555xMmlEZ3vVzUH0bSWAgwuegLNBNXWNDSlWLPyqyf0TV1kO6n0VsmnFa7YsHc6QNQsijh-9RlDmr_sqMLJwrsSjuP87muUS5spJSGrguWYPsc_Q2q0lQBv6__lWWi1UUTsakjeT-6CxjZvHXT69R1GBfSIFjGW3bdW_jyWbzPzxxNCU97zvy5B96ARhuuNN0hPMK3pshpd_Bv4nZc&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node.lib to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node.lib SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node.lib?uploadId=ANnBZCq-6ijn_8ZcDCdA32R3ZA_kVl4hw_gL98dmCXJcrvQMuEOBB0NU39VZZihB6XyvBb9ZN3b0wlSMCwhdlW3rOvEHWJzkZJKmmTLK9RviH-yl4whPjmvkITyBXBAC8wyLRjCuLpqs_y2SuhDLtxMacpe9D3uWqIaSXAuvPnS3WqOLABzJas2BE2VPnK4AVkvon5YlSoud2EYnR1quqe0uzmMdfXNYXlKAyAD2ukQDf_FF1n29SekhxbcLlBLioCI-SNZHarYlCmPKPLxjNyNQgCos_ntwvk8b0LF-Lowu7D-LLQViNPAElzMfF50ZEXrUsBTw1DmTICqkFj54HeI&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node_pdb.7z to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.7z?uploadId=AJ3AQL3aZCrLsy7myySv_L61kLKVTa_7t_U68o-pUOHUYY1lS29itps-etR8ZS8xFrjwqmpXOUiaOxIugul14pijacW7diecbA3fLm_kV8e6ROwI0bU2d0N-oF4Ao68CJNqbrfaQuUg8Xx_XmPcQVcgqTiSNbYXn33VRqJM9ngZTDl5OZvPtFysq6bff6EL7n6NojFy9oxeXbdmBAILyCIk2Y3N8DRSNXFhH_yima17OVJ-cqW12xt43rV_0NGP-Xl8F1H16UETzzSx_S2gn4zpN_pnsl6md4QOLj0z1GTND1UPiWED3P_95nDjeshrFzs7InJgN8AlZ1MdDf5A3VD8&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node_pdb.zip to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.zip?uploadId=ABmd5cHDQxWUZ4F_1XW_YTuBDpYxAU5QZWK7bFdb6xzkbWidqPfoWiWIjMybXNAtG_VH9rHPUT_mybY_mpGqCwOIWJzaydvvXFH44kIx5dRDPwv2HMMhwQbl4A3W8MVGkhVUnRSAqosRt7Mc2wLG-b7tPD5MrmkLxX2HnMrQK9Lbuz_zqoQbaF_WHhnD9EPMmN5_dPMflpmEKtvqaJTdako82Ws2wV7Wsxal0ZoGVrQaARlG71rbzGJTfp2lqMEat7CknovUyWnxketGWognkKyhj4EMX8zfQKN7zndS4Z4gNCiA7J5el4fssVPeYeBKsQ_dlFHOtBI6_EmowyUUkx0&partNumber=5 EOF occurred in violation of protocol (_ssl.c:2427)

The errors are very slow -- presumably the uploads are being attempted and erroring one by one.

This is just running tools/release.sh as per https://github.com/nodejs/node/blob/main/doc/contributing/releases.md#15-promote-and-sign-the-release-builds.

flakey5 commented 1 year ago

Something like that. Roll out in smaller steps than a big bang. Do you have a suggestion for a first good step?

First step could be /metrics, that way if anything does go wrong with how we route to the worker we can fix it without disrupting too much

I'm running into errors trying to promote Node.js 18.18.1

@MoLow

richardlau commented 1 year ago

FYI I've had to comment out the calls to upload_to_cloudflare.sh from the promotion scripts so that tools/release.sh can actually sign the releases (the errors from the upload were causing the release script to bail out).

_promote.sh-1267-# https://github.com/nodejs/build/issues/3508
_promote.sh:1313:#    . ${__dirname}/upload_to_cloudflare.sh $site $subdir
_promote.sh-1371-    fi
--
resha_release.sh-505-# https://github.com/nodejs/build/issues/3508
resha_release.sh:551:# . ${__dirname}/upload_to_cloudflare.sh $site $2
resha_release.sh-601-
--
ovflowd commented 1 year ago

Why would there be SSL validation failures? @flakey5 do you want to share this with the Cloudflare team? Afaik we have one Cloudflare Engineer and our Account Manager on the #nodejs-build channel

jasnell commented 1 year ago

Check the certs provided by the server. As a temporary workaround also try setting allow unauthorized to skip the cert validation. It's possibly with either a mismatch in the cname expected by the cert or a CA issue.

flakey5 commented 1 year ago

The cert is valid from September 6th to December 5th. I'm not really sure what the issue with it is, there's no status page or anyone talking about a similar issue in the Cloudflare Discord so I'm more inclined to think it's something with the origin server itself. But iirc this was working on the origin server already, so I don't know what happened.

Could we retry syncing it just to see if it's reproducible?

MoLow commented 1 year ago

btw it seems the artifacts did partially upload, https://r2.nodejs.org/dist/v18.18.1/, https://r2.nodejs.org/docs/latest-v18.x/api/ and https://r2.nodejs.org/docs/latest-hydrogen/api/ are updated but https://r2.nodejs.org/dist/latest-v18.x/ and https://r2.nodejs.org/dist/latest-hydrogen/ are not

richardlau commented 1 year ago

@MoLow Are those separate uploads or are those latest-* directories supposed to be handled by https://github.com/nodejs/release-cloudflare-worker? I see https://github.com/nodejs/release-cloudflare-worker/pull/47 has updated the JSON to point at 18.18.1.

ovflowd commented 1 year ago

@MoLow Are those separate uploads or are those latest-* directories supposed to be handled by nodejs/release-cloudflare-worker? I see nodejs/release-cloudflare-worker#47 has updated the JSON to point at 18.18.1.

I also wonder if the redirect updates should be done automatically? Because creating the manual need of updating those redirects feels weird. Also, @MoLow or @flakey5 are the caches for those latest-* folders updated/cleared if needed?

flakey5 commented 1 year ago

I also wonder if the redirect updates should be done automatically?

The redirect update is scheduled for every night at midnight (https://github.com/nodejs/release-cloudflare-worker/blob/3fcba3ef2149a639a77b3369109426b3f7a600af/.github/workflows/update-links.yml#L6) in addition to being able to be manually triggered. Someone on the web infra team does need to approve the prs still though.

are the caches for those latest-* folders updated/cleared if needed?

Right now yes since the entire zone's cache is still being purged. Will probably need to update the worker's cache purge endpoint though to work with the redirects we're doing.

MoLow commented 1 year ago

Are those separate uploads or are those latest-* directories supposed to be handled by nodejs/release-cloudflare-worker? I see https://github.com/nodejs/release-cloudflare-worker/pull/47 has updated the JSON to point at 18.18.1.

yes, as @flakey5 said this is entirely automated. this is very weird as the nightly builds seem to be uploaded correctly.

MoLow commented 1 year ago

@nodejs/build @flakey5 @ovflowd what is the current status than?

flakey5 commented 1 year ago

We're ready for the worker to be deployed on

Not nodejs.org/dist* since there may still be a circular dependency with R2's CI, I'm waiting to hear back from them on it. nodejs.org/download/release does link to the same folder as nodejs.org/download/release, but it should be fine.

targos commented 11 months ago

Before doing the next release (next week), I'll try to investigate why binaries were not synced to R2 when I released v21.2.0.

ovflowd commented 11 months ago

Right, let's maybe also reference this on > https://github.com/nodejs/release-cloudflare-worker/issues/74

targos commented 11 months ago

The last step of sign is to copy the shasum files to the server:

https://github.com/nodejs/node/blob/1858341377c268c5ffce9345517dc07b0e6c240a/tools/release.sh#L145-L151

The sync to R2 should only be done after that. Currently, it runs too early:

https://github.com/nodejs/node/blob/1858341377c268c5ffce9345517dc07b0e6c240a/tools/release.sh#L106

https://github.com/nodejs/build/blob/0734aaac357b1cff2965638b1b6e92afc86592e5/ansible/www-standalone/tools/dist/dist-sign

https://github.com/nodejs/build/blob/0734aaac357b1cff2965638b1b6e92afc86592e5/ansible/www-standalone/tools/promote/resha_release.sh#L28

targos commented 11 months ago

Add echo statements in promotion script

I added set -x to all relevant scripts.

targos commented 11 months ago

Status: the script is currently runnning aws s3 cp /home/dist/nodejs/release/v20.10.0/ s3://dist-prod/nodejs/release/v20.10.0/ ... and that's taking a long time (with no feedback to the console).

targos commented 11 months ago

And it just started to fail with the "EOF occurred in violation of protocol" error 😭

targos commented 11 months ago

@richardlau I think your hunch was right. The release puts a high load on the server and it takes to long to send the files to R2 (we probably hit a timeout).

targos commented 11 months ago

I disabled the upload script, signed the release, and re-synced the folder manually to R2.

ovflowd commented 11 months ago

What should we do then? Yo be able to upload these files during release? Or should we run this after the release or should we maybe maybe shard or thread this?

targos commented 11 months ago

I tried again today for the release of v18.19.0. It failed to sync with R2 again. This time not because of timeout/ssl errors, but because some file name is too long?

Full error output: https://gist.github.com/targos/039dc834d68fb1480bc8a96e6fb757d4

Last line is

Completed 161.7 KiB/161.7 KiB (630.1 KiB/s) with 1 file(s) remaining\rupload: nodejs/release/index.tab to s3://dist-prod/nodejs/release/index.tab: File name too long

I tried to run the signing step multiple times but it always failed with the same error. I disabled the sync in resha_release.sh for now.

ovflowd commented 11 months ago

I tried again today for the release of v18.19.0. It failed to sync with R2 again. This time not because of timeout/ssl errors, but because some file name is too long?

Full error output: gist.github.com/targos/039dc834d68fb1480bc8a96e6fb757d4

Last line is

Completed 161.7 KiB/161.7 KiB (630.1 KiB/s) with 1 file(s) remaining\rupload: nodejs/release/index.tab to s3://dist-prod/nodejs/release/index.tab: File name too long

I tried to run the signing step multiple times but it always failed with the same error. I disabled the sync in resha_release.sh for now.

How is that file name too long O.o

targos commented 11 months ago

The actual error is probably unrelated to index.tab. R2 contains the latest version of that file.

targos commented 11 months ago

We discussed this during the last Build WG meeting.

Suggested approach to implement:

richardlau commented 11 months ago

Suggested approach to implement:

  • In addition to copying assets to the DO server (www), release jobs should also push directly to R2

    • Like on www, they should be pushed to a staging place, and moved to production when promoted

On this particular point, currently the uploads to www are done in the Makefile and vcbuild.bat as a series of scp and ssh commands.

Generally, for each asset:

  1. Use scp to upload the asset to www.
  2. Use ssh to touch a corresponding *.done file. I believe these are to prevent the promotion scripts from promoting a partially uploaded file.

To upload to R2 we either need to install the S3 client and secret onto every release machine, or we might be able to use ssh to run the upload from www after we have uploaded the asset (e.g. something like ssh $(STAGINGSERVER) "upload-to-cloudflare.sh ...").

ovflowd commented 11 months ago

The 2nd option feels simpler for the time being, and sorta ensures we always have one copy of the assets somewhere besides Cloudflare

ovflowd commented 11 months ago

But the 1st option feels better for the longterm probably?

richardlau commented 11 months ago

The 2nd option feels simpler for the time being, and sorta ensures we always have one copy of the assets somewhere besides Cloudflare

And on that note, one reason for now to keep uploading to the DO droplet (besides being cautious) is that the backup server (actual backup server and NOT the fail over www server) currently rsyncs with the DO droplet. Today, each binary asset is:

IIRC the rsyncs to unencrypted and backup are run on timers on those machines (i.e. they are pulled and not pushed from www).

ovflowd commented 11 months ago

qq: failover would the joyent server?

richardlau commented 11 months ago

qq: failover would the joyent server?

Yes.

MoLow commented 11 months ago

We discussed this during the last Build WG meeting.

Suggested approach to implement:

  • In addition to copying assets to the DO server (www), release jobs should also push directly to R2

    • Like on www, they should be pushed to a staging place, and moved to production when promoted
  • During promotion, only the shasums and signatures would need to be pushed directly to R2 (by the script run by the releaser)

FWIW, I started implementing something similar in https://github.com/nodejs/build/pull/3501

richardlau commented 1 week ago

Was able to fix promotion to R2's dist-prod bucket for releases while releasing Node.js 22.11.0: https://github.com/nodejs/build/issues/3838#issuecomment-2444654845

targos commented 1 week ago

I'm trying to sync DO with R2, but it doesn't work with SHASUMS256.txt (only this file):

$ aws s3 sync nodejs/release/v22.10.0/ s3://dist-prod/nodejs/release/v22.10.0/ --delete --no-follow-symlinks --endpoint-url=https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com --profile=worker
upload: nodejs/release/v22.10.0/SHASUMS256.txt to s3://dist-prod/nodejs/release/v22.10.0/SHASUMS256.txt
delete: s3://dist-prod/nodejs/release/v22.10.0/SHASUMS256.txt

The file is uploaded and then immediately deleted.

From --debug output: 2024-10-30 15:40:10,035 - MainThread - awscli.customizations.s3.syncstrategy.delete - DEBUG - syncing: (None) -> dist-prod/nodejs/release/v22.10.0/SHASUMS256.txt (remove), file does not exist at source (/home/dist/nodejs/release/v22.10.0/SHASUMS256.txt) and delete mode enabled

targos commented 1 week ago

If I don't pass --delete, then it will always reupload this file (happens with all releases). Reason: 2024-10-30 15:55:35,729 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: /home/dist/nodejs/release/v22.10.0/SHASUMS256.txt -> dist-prod/nodejs/release/v22.10.0/SHASUMS256.txt, file does not exist at destination

targos commented 1 week ago

To be clear, the file exists at both source and destination (https://r2.nodejs.org/download/release/v22.10.0/SHASUMS256.txt)

richardlau commented 1 week ago

@jahands suggested in https://github.com/nodejs/build/pull/3840#discussion_r1821461643 trying rclone as an alternative to the aws s3 client. That should be available from Ubuntu's package repository (so would not be a random client download).

targos commented 1 week ago

rclone seems to work better. I did:

apt install rclone
su dist
rclone config

Followed the docs from https://rclone.org/s3/#cloudflare-r2

This created a config file in /home/dist/.config/rclone/rclone.conf

Commands like rclone sync nodejs/release/v22.9.0/ r2:dist-prod/nodejs/release/v22.9.0/ or rclone copy r2:dist-staging/nodejs/release/v22.8.0/ r2:dist-prod/nodejs/release/v22.8.0/ worked fine.