search

IBM / core-dump-handler

Save core dumps from a Kubernetes Service or RedHat OpenShift to an S3 protocol compatible object store
https://ibm.github.io/core-dump-handler/
MIT License
131 stars 40 forks source link

Using an s3 compatible storage - fails to connect on error upload Failed reqwest: #152

Closed sanasz91mdev closed 4 months ago

sanasz91mdev commented 8 months ago

Why is my request failing in dump handler pods:

[2024-01-04T06:45:20Z INFO  core_dump_agent] INotify Starting...
[2024-01-04T06:45:20Z INFO  core_dump_agent] INotify Initialised...
[2024-01-04T06:45:20Z INFO  core_dump_agent] INotify watching : /var/mnt/core-dump-handler/cores
[2024-01-04T06:45:35Z INFO  core_dump_agent] Uploading: /var/mnt/core-dump-handler/cores/0f4c0a44-58d3-46db-9c83-5fb7f2aa45fe-dump-1704350735-segfaulter23-segfaulter-1-4.zip
[2024-01-04T06:45:35Z INFO  core_dump_agent] zip size is 29122
[2024-01-04T06:45:35Z ERROR core_dump_agent] Upload Failed reqwest: error sending request for url (https://68.218.153.165/dumps-bucket/0f4c0a44-58d3-46db-9c83-5fb7f2aa45fe-dump-1704350735-segfaulter23-segfaulter-1-4.zip): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (self signed certificate)

y does it fail to uplaoad to my valid s3 compatible storage? i can upload things via S3 browser on same URL

No9 commented 8 months ago

HI @sanasz91mdev Self signed certs are currently not supported as the underlying S3 library only supports them as a build definition rather than a runtime definition https://github.com/durch/rust-s3/blob/32a5a69363cb74c86ed8b68f713ca14d8f5a2798/s3/src/request.rs#L58C28-L58C41

If you must have support for self signed certs can I suggest building a version of the agent using this Dockerfile https://github.com/IBM/core-dump-handler/blob/main/Dockerfile

with the "no-verify-ssl" feature enabled here https://github.com/IBM/core-dump-handler/blob/main/core-dump-agent/Cargo.toml#L21-L29

Then modifying the chart to use your self signed cert enabled image https://github.com/IBM/core-dump-handler/blob/main/charts/core-dump-handler/values.yaml#L4-L6

Hope this helps

sanasz91mdev commented 8 months ago

@No9 i did the exact steps ... i am still getting error:

[2024-01-04T14:39:44Z ERROR core_dump_agent] Upload Failed reqwest: error sending request for url (https://flexifyapp.australiaeast.cloudapp.azure.com/dumps-bucket/a03b4dfe-0761-4105-bb76-89e80f7a4a68-dump-1704376803-segfaulter51-segfaulter-1-4.zip): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (self signed certificate)

No9 commented 8 months ago

That's strange - it looks like it is passed through the reqwest layer now but is getting thrown out by OpenSSL. This is surprising as we should be using rstls Can you confirm the agents Cargo.toml?

sanasz91mdev commented 8 months ago

That's strange - it looks like it is passed through the reqwest layer now but is getting thrown out by OpenSSL. This is surprising as we should be using rstls Can you confirm the agents Cargo.toml?

yup i edited agent's Cargo.toml

[target.x86_64-unknown-linux-musl.dependencies.rust-s3]
version = "0.31.0"
default-features = false
features = ["no-verify-ssl"]

[target.aarch64-unknown-linux-musl.dependencies.rust-s3]
version = "0.31.0"
default-features = false
features = ["no-verify-ssl"]
No9 commented 8 months ago

Ah that explains why you were getting the SSL errors. If you are using musl you need to keep the "tokio-rustls-tls" features too Not sure if you missed the standard build so including it for completeness.

[target.x86_64-unknown-linux-musl.dependencies.rust-s3]
version = "0.31.0"
default-features = false
features = ["tokio-rustls-tls", "no-verify-ssl"]

[target.aarch64-unknown-linux-musl.dependencies.rust-s3]
version = "0.31.0"
default-features = false
features = ["tokio-rustls-tls", "no-verify-ssl"]

[target.x86_64-unknown-linux-gnu.dependencies.rust-s3]
version = "0.31.0"
features = ["no-verify-ssl"]
sanasz91mdev commented 8 months ago

got fixed with this ... thanks. can you also tell why it does not work with self signed certificates ... i even used a non self signed certificate today but it gave error @No9 ...

(https://20.253.24.196/dumps-bucket/cd025bb8-d1af-4cbf-ade3-71e91d2c2bcc-dump-1704735065-segfaulter-segfaulter-1-4.zip): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get local issuer certificate) [2024-01-08T17:31:21Z INFO core_dump_agent] Uploading: /var/mnt/core-dump-handler/cores/694eaf1b-15e6-402c-bb7a-e8e20dad4c00-dump-1704735081-segfaulter2-segfaulter-1-4.zip [2024-01-08T17:31:21Z INFO core_dump_agent] zip size is 29716 [2024-01-08T17:31:21Z ERROR core_dump_agent] Upload Failed reqwest: error sending request for url (https://20.253.24.196/dumps-bucket/694eaf1b-15e6-402c-bb7a-e8e20dad4c00-dump-1704735081-segfaulter2-segfaulter-1-4.zip): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get local issuer certificate)

Why is this so? ...

Eventually i had to build a new image with ssl-no-verify

No9 commented 4 months ago

ssl-no-verify isn't currently supported because of the static build item mentioned in this comment. https://github.com/IBM/core-dump-handler/issues/152#issuecomment-1876734976