CAS S3 backend fails during file upload due to SSL certificate validation, despite the CA certificate added for both CAS and controlplane

[hanygirgis]

When adding a file to an attestation (with a default S3/MinIO CAS backend), it fails in the CLI with this error:

$ chainloop att add --value bom.json 
INF uploading bom.json - sha256:....
ERR adding material: crafting material: upload and craft error: uploading material: rpc error: code = Internal desc = server error

The CAS pod has this error in the logs : tls: failed to verify certificate: x509: certificate signed by unknown authority

Stacktrace:

{"level":"info","ts":1720176149.4226923,"component":"service","msg":"selected provider","provider":"AWS-S3"}
{"level":"info","ts":1720176149.42272,"component":"credentials/vault","msg":"reading credentials","path":"chainloop/11ee7ee9-7998-4834-9f79-576826d838ac/80ffff5f-f916-4f30-8957-ec6a983caa95"}
{"level":"error","ts":1720176149.751981,"component":"service","msg":"failed to read from bucket: RequestError: send request failed\ncaused by: Head \"https://hostname:port/chainloop/sha256%3A445136956fe117254a3da0ed8d1a3fc176585a1c8ba0bd9144fcae916f4d3ba2\": tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"github.com/go-kratos/kratos/contrib/log/zap/v2.(*Logger).Log\n\t/home/runner/go/pkg/mod/github.com/go-kratos/kratos/contrib/log/zap/v2@v2.0.0-20230823024326-a09f4d8ebba9/zap.go:41\ngithub.com/go-kratos/kratos/v2/log.(*logger).Log\n\t/home/runner/go/pkg/mod/github.com/go-kratos/kratos/v2@v2.7.0/log/log.go:30\ngithub.com/go-kratos/kratos/v2/log.(*Helper).Error\n\t/home/runner/go/pkg/mod/github.com/go-kratos/kratos/v2@v2.7.0/log/helper.go:121\ngithub.com/chainloop-dev/chainloop/pkg/servicelogger.LogAndMaskErr\n\t/home/runner/work/chainloop/chainloop/pkg/servicelogger/logger.go:57\ngithub.com/chainloop-dev/chainloop/app/artifact-cas/internal/service.(*ByteStreamService).Write\n\t/home/runner/work/chainloop/chainloop/app/artifact-cas/internal/service/bytestream.go:87\ngoogle.golang.org/genproto/googleapis/bytestream._ByteStream_Write_Handler\n\t/home/runner/go/pkg/mod/google.golang.org/genproto/googleapis/bytestream@v0.0.0-20240318140521-94a12d6c2237/bytestream.pb.go:846\ngithub.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).StreamServerInterceptor.func4\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.1-0.20210315223345-82c243799c99/server_metrics.go:122\ngoogle.golang.org/grpc.getChainStreamHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532\ngithub.com/chainloop-dev/chainloop/app/artifact-cas/internal/server.NewGRPCServer.StreamServerInterceptor.func12\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/auth/auth.go:66\ngithub.com/chainloop-dev/chainloop/app/artifact-cas/internal/server.NewGRPCServer.StreamServerInterceptor.func13\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/selector/selector.go:50\ngoogle.golang.org/grpc.getChainStreamHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532\ngithub.com/go-kratos/kratos/v2/transport/grpc.NewServer.(*Server).streamServerInterceptor.func2\n\t/home/runner/go/pkg/mod/github.com/go-kratos/kratos/v2@v2.7.0/transport/grpc/interceptor.go:81\ngoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1523\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1687\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027"}

[migmartri]

Thanks for reporting this, we'll take a look cc/ @jiparis

[migmartri]

I believe CAS deployment pod doesn't have support for mounting customCAs. We will need to do this https://github.com/chainloop-dev/chainloop/pull/964 but for the CAS pod

[jiparis]

Hi, a new version of the chart has been released with the ability to add customCAs to the CAS deployment.

@hanygirgis could you try it? Just add the PEM CA certificate to the cas section in your values.yaml. Something like this:

cas:
  ...
  customCAs:
    - |-
      -----BEGIN CERTIFICATE-----
      MIIFmDCCA4CgAwIBAgIQU9C87nMpOIFKYpfvOHFHFDANBgkqhkiG9w0BAQsFADBm
      BhMCVVMxMzAxBgNVBAoTKihTVEFHSU5HKSBJbnRlcm5ldCBTZWN1cml0eSBSZXNl
      REDACTED
      5CunuvCXmEQJHo7kGcViT7sETn6Jz9KOhvYcXkJ7po6d93A/jy4GKPIPnsKKNEmR
      7DiA+/9Qdp9RBWJpTS9i/mDnJg1xvo8Xz49mrrgfmcAXTCJqXi24NatI3Oc=
      -----END CERTIFICATE-----

[hanygirgis]

It works now, thanks a lot.