search

splunk / splunk-operator

Splunk Operator for Kubernetes
Other
202 stars 112 forks source link

Splunk Operator: splunk-operator can't connect to nodes via https to ip:8089/services #1078

Open yaroslav-nakonechnikov opened 1 year ago

yaroslav-nakonechnikov commented 1 year ago

Please select the type of request

Bug

Tell us more

Describe the request noticed next error on splunk-operator:edge:

2023-02-17T15:36:22.299895889Z  ERROR   ApplySearchHeadCluster  Unable to retrieve search head cluster member info      {"controller": "searchheadcluster", "controllerGroup": "enterprise.splunk.com", "controllerKind": "SearchHeadCluster", "SearchHeadCluster": {"name":"c-dev","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "c-dev", "reconcileID": "b77a95a9-b165-414c-8e79-cab3e771b247", "memberName": "splunk-c-dev-search-head-1", "error": "Get \"
https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/services/shcluster/member/info?count=0&output_mode=json\": dial tcp 100.65.6.227:8089: connect: connection refused"}
github.com/splunk/splunk-operator/pkg/splunk/enterprise.(*searchHeadClusterPodManager).updateStatus
        /workspace/pkg/splunk/enterprise/searchheadcluster.go:578
github.com/splunk/splunk-operator/pkg/splunk/enterprise.(*searchHeadClusterPodManager).Update
        /workspace/pkg/splunk/enterprise/searchheadcluster.go:445
github.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplySearchHeadCluster
        /workspace/pkg/splunk/enterprise/searchheadcluster.go:183
github.com/splunk/splunk-operator/controllers.glob..func7
        /workspace/controllers/searchheadcluster_controller.go:115
github.com/splunk/splunk-operator/controllers.(*SearchHeadClusterReconciler).Reconcile
        /workspace/controllers/searchheadcluster_controller.go:105
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234

and i thought that service is not working, so i did this:

[yn@ip-10-216-35-53 bin]$ kubectl exec  pod/splunk-operator-controller-manager-5965c4dfc-hcbjx -c manager -n splunk-operator -it -- curl https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
command terminated with exit code 60

that means, that self-signed certificate is not working.

Expected behavior i believe there should be some flag which will allow such connections.

Splunk setup on K8S eks

Reproduction/Testing steps start cluster with built-in ssl config

vivekr-splunk commented 1 year ago

have you tried sending insecure to curl command for testing 

[yn@ip-10-216-35-53 bin]$ kubectl exec  pod/splunk-operator-controller-manager-5965c4dfc-hcbjx -c manager -n splunk-operator -it -- curl --insecure -v https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089
vivekr-splunk commented 1 year ago

@iaroslav-nakonechnikov we will try out the scenario you have mentioned and get back to you. meanwhile please try the above command and let us know if that is working.

yaroslav-nakonechnikov commented 1 year ago

@vivekr-splunk, yes, here are 2 requests:

[yn@ip-10-216-35-53 bin]$ kubectl exec  pod/splunk-operator-controller-manager-5965c4dfc-hcbjx -c manager -n splunk-operator -it -- curl --insecure -v https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089
* Rebuilt URL to: https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/
*   Trying 100.65.2.134...
* TCP_NODELAY set
* Connected to splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local (100.65.2.134) port 8089 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=SplunkServerDefaultCert; O=SplunkUser
*  start date: Feb 18 03:57:13 2023 GMT
*  expire date: Feb 17 03:57:13 2026 GMT
*  issuer: C=US; ST=CA; L=San Francisco; O=Splunk; CN=SplunkCommonCA; emailAddress=support@splunk.com
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> Host: splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 20 Feb 2023 10:23:31 GMT
< Content-Type: text/xml; charset=UTF-8
< X-Content-Type-Options: nosniff
< Content-Length: 2588
< Connection: Keep-Alive
< X-Frame-Options: SAMEORIGIN
< Server: Splunkd
<
<?xml version="1.0" encoding="UTF-8"?>
<!--This is to override browser formatting; see server.conf[httpServer] to disable. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .-->
<?xml-stylesheet type="text/xml" href="/static/atom.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest">  <title>splunkd</title>
  <id>https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/</id>
  <updated>2023-02-20T10:23:31+00:00</updated>
  <generator build="de405f4a7979" version="9.0.4"/>
  <author>
    <name>Splunk</name>
  </author>
  <entry>
    <title>rpc</title>
    <id>https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/rpc</id>
    <updated>1970-01-01T00:00:00+00:00</updated>
    <link href="/rpc" rel="alternate"/>
  </entry>
  <entry>
    <title>services</title>
    <id>https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/services</id>
    <updated>1970-01-01T00:00:00+00:00</updated>
    <link href="/services" rel="alternate"/>
  </entry>
  <entry>
    <title>servicesNS</title>
    <id>https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/servicesNS</id>
    <updated>1970-01-01T00:00:00+00:00</updated>
    <link href="/servicesNS" rel="alternate"/>
  </entry>
  <entry>
    <title>static</title>
    <id>https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/static</id>
    <updated>1970-01-01T00:00:00+00:00</updated>
    <link href="/static" rel="alternate"/>
  </entry>
</feed>
* Connection #0 to host splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local left intact

[yn@ip-10-216-35-53 bin]$ kubectl exec  pod/splunk-operator-controller-manager-5965c4dfc-hcbjx -c manager -n splunk-operator -it -- curl -v https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089
* Rebuilt URL to: https://splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local:8089/
*   Trying 100.65.2.134...
* TCP_NODELAY set
* Connected to splunk-c-dev-search-head-1.splunk-c-dev-search-head-headless.splunk-operator.svc.cluster.local (100.65.2.134) port 8089 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
command terminated with exit code 60
vinitmasaun commented 1 year ago

We are experiencing the same errors in the splunk operator pods and also, the REST API endpoint does not load on port 8089 via the browser. We are able to hit the rest endpoints within the cluster using curl but not able to interact with the REST endpoint using a client like postman or browser using ingress or port forwarding.

chipzzz commented 11 months ago

@vivekr-splunk , after upgrading to Splunk to 9.0.5 / splunk-operator to 2.3.0 - any idea?

We experience the same on the splunk-operator bundle push to master(manager) node, while unsecure works just gives me 401 as i'm not setting proper headers and context

[nonroot@splunk-operator-controller-manager-SOME_IDl /]$ curl -vvv https://splunk-cluster-cluster-master-service.splunk.svc.cluster.local:8089/services/cluster/manager/control/default/apply
*   Trying SOME_IP...
* TCP_NODELAY set
* Connected to splunk-cluster-cluster-master-service.splunk.svc.cluster.local (SOME_IP) port 8089 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
vivekr-splunk commented 11 months ago

tracking CSPL-2432