search

apache / solr-operator

Official Kubernetes operator for Apache Solr
https://solr.apache.org/operator
Apache License 2.0
243 stars 112 forks source link

Operator crash after update to v0.7.0 #583

Closed angelprada closed 11 months ago

angelprada commented 1 year ago

Hello.

I've tried to upgrade solr-operator from v0.6.0 to v0.7.0 in AWS EKS 1.27 but it crashes constantly:

2023-07-03T08:49:18Z    INFO    setup   solr-operator Version: v0.7.0
2023-07-03T08:49:18Z    INFO    setup   solr-operator Git SHA: 0e16c6b
2023-07-03T08:49:18Z    INFO    setup   solr-operator Build Time: Mon Apr 24 19:23:35 UTC 2023
2023-07-03T08:49:18Z    INFO    setup   Go Version: go1.19.8
2023-07-03T08:49:18Z    INFO    setup   Go OS/Arch: linux / amd64
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12ab950]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00004f560, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc00004e3f0, 0x15}, {0xc000049060, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x1815ff3?)
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc000001e00?)
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1a8a320, 0xc0002b32f0})
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0004c50d0)
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
    /go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1a8a320?, 0xc0002b32f0?})
    /go/pkg/mod/k8s.io/client-go@v0.26.1/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:86 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:119
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc000358870?, {0x0, 0x0, 0x2fb3e8039a338f01?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/client/apiutil/dynamicrestmapper.go:99 +0x176
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc00011d6c0?)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc0001c3200, {0xc000143a58, 0x1, 0x0?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc00011d6c0, 0x0, 0x0, {{0x1a87000, 0xc00012f1c0}, 0x0}, 0x1, {0x0, 0x0}, ...})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/manager/manager.go:351 +0xf9
main.main()
    /workspace/main.go:171 +0x7b5
HoustonPutman commented 1 year ago

Unfortunately I can't replicate this on a KinD cluster running 1.27, but I did find this when googling:

https://github.com/elastic/cloud-on-k8s/issues/6848

So maybe we just need to update the version of the kubernetes client for the operator.

angelprada commented 1 year ago

Hello.

We had similar problems with external-secrets https://github.com/external-secrets/external-secrets/issues/2344

Hope it helps.

HoustonPutman commented 11 months ago

Fixed this. I'll start a v0.7.1 release soon.

HoustonPutman commented 11 months ago

@angelprada The release candidate is out to try, please let me know if the docker image apache/solr-operator:v0.7.1-rc1 works for you. It should be a drop-in replacement for v0.7.0.

angelprada commented 11 months ago

@HoustonPutman Tried and worked as expected!

2023-07-17T17:46:46Z    INFO    setup   solr-operator Version: v0.7.1
2023-07-17T17:46:46Z    INFO    setup   solr-operator Git SHA: 5205b93
2023-07-17T17:46:46Z    INFO    setup   solr-operator Build Time: Mon Jul 17 17:08:17 UTC 2023
2023-07-17T17:46:46Z    INFO    setup   Go Version: go1.19.11
2023-07-17T17:46:46Z    INFO    setup   Go OS/Arch: linux / amd64
2023-07-17T17:46:47Z    INFO    controller-runtime.metrics  Metrics server is starting to listen    {"addr": ":8080"}
2023-07-17T17:46:47Z    INFO    setup   starting manager
2023-07-17T17:46:47Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-07-17T17:46:47Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
I0717 17:46:47.169060       1 leaderelection.go:248] attempting to acquire leader lease kube-system/88488bdc.solr.apache.org...
I0717 17:47:17.922432       1 leaderelection.go:258] successfully acquired lease kube-system/88488bdc.solr.apache.org

Thank you!