search

splunk / splunk-operator

Splunk Operator for Kubernetes
Other
209 stars 115 forks source link

Monitoring Console: Keep restarting #1080

Open Sinamcp opened 1 year ago

Sinamcp commented 1 year ago

Please select the type of request

Bug

Tell us more

Describe the request

Splunk setup on K8S

apiVersion: enterprise.splunk.com/v4
kind: MonitoringConsole
metadata:
  name: mc-test
  namespace: default
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterManagerRef:
    name: cm-test
  etcVolumeStorageConfig:
    storageCapacity: 2Gi
  varVolumeStorageConfig:
    storageCapacity: 5Gi

after applying the manifest, monitoring console pod start ansible, but it fails with the below error

TASK [splunk_monitor : dmc_asset_build_full - POST] ****************************
fatal: [localhost]: FAILED! => {
    "cache_control": "no-store, no-cache, must-revalidate, max-age=0",
    "changed": false,
    "connection": "Close",
    "content_length": "360",
    "content_type": "text/xml; charset=UTF-8",
    "date": "Sat, 18 Feb 2023 12:26:24 GMT",
    "elapsed": 0,
    "expires": "Thu, 26 Oct 1978 00:00:00 GMT",
    "redirected": false,
    "server": "Splunkd",
    "status": 503,
    "url": "https://127.0.0.1:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC%20Asset%20-%20Build%20Full/dispatch",
    "vary": "Cookie, Authorization",
    "x_content_type_options": "nosniff",
    "x_frame_options": "SAMEORIGIN"
}

MSG:

Status code was 503 and not [200, 201, 409]: HTTP Error 503: Service Unavailable
kashok-splunk commented 1 year ago

Hi @Sinamcp We don't need to provide cluster-manager info in the MC yaml, we can remove this information. Regarding the 503 error, are all the services up? as 503 error indicates service is not up. Additionally, can you try running this GET call from MC pod? curl -X GET -k -u admin:<PASSWORD> https://127.0.0.1:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC%20Asset%20-%20Build%20Full?output_mode=json, this query just builds the MC asset table. Let me know what response you get here

akondur commented 1 year ago

Hi @Sinamcp , please let us know if there are any updates here.

Sinamcp commented 1 year ago

Hi @akondur 

(okd4venv) (okd4venv) [centos@okd4-ts-2-utility ~]$ oc rsh splunk-mc-test-monitoring-console-0
sh-4.4$ curl -X GET -k -u admin:D7YSwvrMWXdxpzb7MOB7nIM4 https://127.0.0.1:8089/servicesNS/nobody/splunk_monitoring_console/saved/searches/DMC%20Asset%20-%20Build%20Full?output_mode=json
curl: (7) Failed to connect to 127.0.0.1 port 8089: Connection refused
Sinamcp commented 1 year ago

I don't know what should i do, please help me to fix