search

cert-manager / istio-csr

istio-csr is an agent that allows for Istio workload and control plane components to be secured using cert-manager.
https://cert-manager.io/docs/usage/istio-csr/
Apache License 2.0
161 stars 71 forks source link

Panic: runtime error on new installation #413

Open brettcurtis opened 2 months ago

brettcurtis commented 2 months ago

I've just started messing around with this the other day so I'm sure I've got something off. However, I followed the docs and my istio-csr pod goes into CrashLoopBackOff and I'm seeing the following:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x284a549]

goroutine 103 [running]:
istio.io/istio/security/pkg/server/ca/authenticate/kubeauth.(*KubeJWTAuthenticator).authenticate(0xc000b2eb00, {0xc000bde007, 0x4ce}, {0xc000013c70, 0xf})
    istio.io/istio@v0.0.0-20240821191124-89d73dc1f639/security/pkg/server/ca/authenticate/kubeauth/kube_jwt.go:113 +0x369
istio.io/istio/security/pkg/server/ca/authenticate/kubeauth.(*KubeJWTAuthenticator).authenticateGrpc(0xc000b2eb00, {0x36882e0, 0xc000bd2f60})
    istio.io/istio@v0.0.0-20240821191124-89d73dc1f639/security/pkg/server/ca/authenticate/kubeauth/kube_jwt.go:106 +0xa6
istio.io/istio/security/pkg/server/ca/authenticate/kubeauth.(*KubeJWTAuthenticator).Authenticate(0xc000171f80?, {{0x36882e0?, 0xc000bd2f60?}, 0x0?})
    istio.io/istio@v0.0.0-20240821191124-89d73dc1f639/security/pkg/server/ca/authenticate/kubeauth/kube_jwt.go:82 +0x49
github.com/cert-manager/istio-csr/pkg/server.(*Server).authRequest(0xc0001ae540, {0x36882e0, 0xc000bd2f60}, 0xc000a8f090)
    github.com/cert-manager/istio-csr/pkg/server/auth.go:43 +0xdd
github.com/cert-manager/istio-csr/pkg/server.(*Server).CreateCertificate(0xc0001ae540, {0x36882e0, 0xc000bd2f60}, 0xc000a8f090)
    github.com/cert-manager/istio-csr/pkg/server/server.go:201 +0x45
istio.io/api/security/v1alpha1._IstioCertificateService_CreateCertificate_Handler.func1({0x36882e0?, 0xc000bd2f60?}, {0x303a500?, 0xc000a8f090?})
    istio.io/api@v1.23.0/security/v1alpha1/ca_grpc.pb.go:109 +0xcb
github.com/cert-manager/istio-csr/pkg/server.(*Server).Start.(*ServerMetrics).UnaryServerInterceptor.func4({0x36882e0, 0xc000bd2f60}, {0x303a500, 0xc000a8f090}, 0x410dc5?, 0xc0007b5668)
    github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0x7d
istio.io/api/security/v1alpha1._IstioCertificateService_CreateCertificate_Handler({0x2ed5e60, 0xc0001ae540}, {0x36882e0, 0xc000bd2f60}, 0xc000a9cd00, 0xc0007582a0)
    istio.io/api@v1.23.0/security/v1alpha1/ca_grpc.pb.go:111 +0x143
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00076e200, {0x36882e0, 0xc000bd2ed0}, {0x369a5c0, 0xc00094c9c0}, 0xc000b39d40, 0xc000bd2570, 0x55951e0, 0x0)
    google.golang.org/grpc@v1.66.0/server.go:1393 +0xe11
google.golang.org/grpc.(*Server).handleStream(0xc00076e200, {0x369a5c0, 0xc00094c9c0}, 0xc000b39d40)
    google.golang.org/grpc@v1.66.0/server.go:1804 +0xe8b
google.golang.org/grpc.(*Server).serveStreams.func2.1()
    google.golang.org/grpc@v1.66.0/server.go:1029 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 115
    google.golang.org/grpc@v1.66.0/server.go:1040 +0x125

I get this whenever I try to start my Istio gateway or deploy another workload that is Istio enabled. I'd appreciate a kick in the right direction if anyone has any ideas.

EDIT istio-csr v0.12.0 and Istio v1.23.2

brettcurtis commented 2 months ago

Following up, I rolled back and tested v0.11.0 and my istio-csr POD didn't crash. That led me to the following logs:

failed to authenticate request" err="could not get cluster plt-us-east1-sb's kube client" logger="grpc-server" serving-addr="0.0.0.0:6443

This led me to the app.server.clusterID value. After I fixed that, I was able to successfully get a certificate from cert-manager with both versions.

robbo10 commented 1 month ago

Hi there @brettcurtis - I’m running into the same error on 1.23.2, what did you have to change to get past the issue? 😃

I have set the cluster ID to be the same value as the multi cluster clusterName in the istio helm chart.

brettcurtis commented 1 month ago

That's all I did. Once I set the istio-csr helm value app.server.clusterID to match my global.multiCluster.clusterName helm value for Istio, it worked.