search

apache / solr-operator

Official Kubernetes operator for Apache Solr
https://solr.apache.org/operator
Apache License 2.0
243 stars 112 forks source link

Solr startup Probe failure - Node IP used #623

Closed brickpattern closed 6 months ago

brickpattern commented 9 months ago

Issue: IP used in startup Probe happens to be EKS Node IP and not the K8S cluster internal IPs Setup: Environment: AWS EKS EKS : 1.27 Solr Operator : v0.7.1 ZK: inbuilt Solr Cloud: v9.3 All modules running in same namespace Solr 

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  19m   default-scheduler  Successfully assigned solr/solr-solrcloud-0 to ip-xx-yy-xx-yy.us-west-2.internal
  Normal   Pulled     19m   kubelet            Container image "library/busybox:1.28.0-glibc" already present on machine
  Normal   Created    19m   kubelet            Created container cp-solr-xml
  Normal   Started    19m   kubelet            Started container cp-solr-xml
  Normal   Pulling    19m   kubelet            Pulling image "my-custom-sol-registry:0.2"
  Normal   Pulled     19m   kubelet            Successfully pulled image "my-custom-sol-registry:0.2" in 17.833156703s (17.833176263s including waiting)
  Normal   Created    19m   kubelet            Created container solrcloud-node
  Normal   Started    19m   kubelet            Started container solrcloud-node
  Warning  Unhealthy  19m   kubelet            Startup probe failed: Get "http://xx.yy.xx.yy:8983/solr/admin/info/system": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Q - should the Startup Probe is hitting the actual NODE (underlying eks node) IP?

brickpattern commented 9 months ago

the xx.yy.xx.yy in the logs above is of EKS NODE IP (i.e. outside the cluster and not the standard K8S cluster internal IPs )

brickpattern commented 9 months ago

not seeing this error in pod/solr-solrcloud-1 and pod/solr-solrcloud-2

startup probe failed seen only in pod/solr-solrcloud-0 ( in describe pod)

HoustonPutman commented 7 months ago

Sorry for the late response. The solr operator didn't set an ip address for the pod startup probes, so it must have been a kubernetes thing.

If you are still seeing this issue, did the startup probes for the -1 and -2 pods also use the EKS Node IP?

brickpattern commented 7 months ago

will check and get back