[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

Mark-Simulacrum commented 1 year ago

We have seen this error several times recently: on:

2023-10-10T01:05:38.498Z
2023-10-23T01:03:25.895Z
2023-10-26T01:08:47.479Z
2023-11-04T00:59:29.566Z
2023-12-26 (25?)
2023-12-30
2024-03-29T00:50:30.097Z
2024-03-29T00:58:30.285Z
2024-03-30T00:48:14.868Z to 2024-03-30T01:10:33.716Z many failures

See below for a capture from the most recent occurence (2023-10-23). This appears to be a spurious error, unfortunately we don't yet have request IDs or similar to investigate further, and depending on when the failure occurs they may not be available. This may be due to new client software (e.g., awscli) or due to upstream changes, it's not clear.

Both failures happened using awscli 1.22.34-1 from Ubuntu 22.04 archives, starting just after we landed https://github.com/rust-lang/promote-release/commit/19ed30b03123ec08860e0460a582febac24559e0 (which rebuilt the docker image). But we were using that awscli previously as well.

I'm not sure whether underlying openssl versions or similar changed across that time range.

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                             message                                                                                                                                                                                                                                             |
|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1698022319473 | upload failed: release/docs/core/core_arch/arm_shared/neon/generated/fn.vtstq_s16.html to s3://static-rust-lang-org/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vtstq_s16.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vtstq_s16.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                                     |
| 1698022321511 | upload failed: release/docs/core/core_arch/arm_shared/neon/generated/fn.vzipq_s32.html to s3://static-rust-lang-org/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vzipq_s32.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/core/core_arch/arm_shared/neon/generated/fn.vzipq_s32.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                                     |
| 1698022931768 | upload failed: release/docs/nightly-rustc/rustc_target/abi/call/loongarch/sidebar-items.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/abi/call/loongarch/sidebar-items.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/abi/call/loongarch/sidebar-items.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                         |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/sidebar-items.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/sidebar-items.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/sidebar-items.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)        |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_ohos/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_ohos/fn.target.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_ohos/fn.target.html".                                             |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_gnu/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_gnu/fn.target.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_linux_gnu/fn.target.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)              |
| 1698023001838 | upload failed: release/docs/nightly-rustc/rustc_target/spec/wasm32_unknown_emscripten/index.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/wasm32_unknown_emscripten/index.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/wasm32_unknown_emscripten/index.html".                                                         |
| 1698023003878 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_sun_solaris/index.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_sun_solaris/index.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_sun_solaris/index.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                                            |
| 1698023003878 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/fn.target.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_apple_watchos_sim/fn.target.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)              |
| 1698023005895 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_unknown_l4re_uclibc/sidebar-items.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_l4re_uclibc/sidebar-items.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_unknown_l4re_uclibc/sidebar-items.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)  |
| 1698023005895 | upload failed: release/docs/nightly-rustc/rustc_target/spec/x86_64_uwp_windows_gnu/fn.target.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_uwp_windows_gnu/fn.target.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_target/spec/x86_64_uwp_windows_gnu/fn.target.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)                    |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Mark-Simulacrum commented 10 months ago

We disabled debug logging because it was rather excessive, but this happened again today:

running "aws" "s3" "sync" "--storage-class" "INTELLIGENT_TIERING" "--delete" "--only-show-errors" "/codebuild/output/src2664974370/src/release/docs/" "s3://static-rust-lang-org/doc/nightly/"
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustVersionMinor.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustVersionMinor.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustVersionMinor.html".
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrepareThinLTOInternalize.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrepareThinLTOInternalize.html Connection was closed before we received a valid response from endpoint URL: "https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrepareThinLTOInternalize.html".
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMSetGlobalConstant.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMSetGlobalConstant.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMSetGlobalConstant.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrintStatistics.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrintStatistics.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/fn.LLVMRustPrintStatistics.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
upload failed: release/docs/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/foreigntype.ModuleBuffer.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/foreigntype.ModuleBuffer.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_codegen_llvm/llvm_/ffi/foreigntype.ModuleBuffer.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

We might want to consider an in-code retry of all aws s3 commands or something...

Mark-Simulacrum commented 10 months ago

Happened again today:

upload failed: release/docs/alloc/macro.vec!.html to s3://static-rust-lang-org/doc/nightly/alloc/macro.vec!.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/alloc/macro.vec%21.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
[...] dozens more [...]
upload failed: release/docs/nightly-rustc/rustc_passes/errors/struct.OnlyHasEffectOn.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/rustc_passes/errors/struct.OnlyHasEffectOn.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/rustc_passes/errors/struct.OnlyHasEffectOn.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)
Error: failed command:"aws" "s3" "sync" "--storage-class" "INTELLIGENT_TIERING" "--delete" "--only-show-errors" "/codebuild/output/src2335245485/src/release/docs/" "s3://static-rust-lang-org/doc/nightly/"

Mark-Simulacrum commented 10 months ago

I manually enabled S3 access logs for static-rust-lang-org. My hope is that gives us request IDs without changing the client, but given the error is an SSL error I'm not super optimistic it'll be tied to any particular request.

Mark-Simulacrum commented 10 months ago

Failed again today:

2024-01-04T01:08:33.019Z upload failed: release/docs/nightly-rustc/src/cargo/core/resolver/context.rs.html to s3://static-rust-lang-org/doc/nightly/nightly-rustc/src/cargo/core/resolver/context.rs.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/src/cargo/core/resolver/context.rs.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

Looking at S3 logs (now that we have those), I'm not seeing any references to this file, though there are uploads around the same time from the codebuild job. I don't see any references to this particular file path or obvious errors for PUT requests.

So unfortunately no real luck from having those logs...

Mark-Simulacrum commented 9 months ago

Occurred again today:

upload failed: release/docs/core/arch/x86/fn._mm512_cvtepu8_epi64.html to s3://static-rust-lang-org/doc/nightly/core/arch/x86/fn._mm512_cvtepu8_epi64.html SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/core/arch/x86/fn._mm512_cvtepu8_epi64.html [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

Mark-Simulacrum commented 9 months ago

upload failed: release/docs/nightly-rustc/trait.impl/rustc_hir/intravisit/nested_filter/trait.NestedFilter.js to s3://static-rust-lang-org/doc/nightly/nightly-rustc/trait.impl/rustc_hir/intravisit/nested_filter/trait.NestedFilter.js SSL validation failed for https://static-rust-lang-org.s3.us-west-1.amazonaws.com/doc/nightly/nightly-rustc/trait.impl/rustc_hir/intravisit/nested_filter/trait.NestedFilter.js [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)

rust-lang / promote-release

[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007) #77