search

sassoftware / viya4-monitoring-kubernetes

Provides simple scripts and customization options to deploy monitoring, alerts, and log aggregation for Viya 4 running on Kubernetes
Apache License 2.0
50 stars 31 forks source link

No data exported. Fails to load data. No dashboards in the list. #649

Open ahsokm07 opened 3 weeks ago

ahsokm07 commented 3 weeks ago

Its a standalone deployed not included with viya deployment. I confirmed there is sufficient resource in the cluster,

Opensearch content is not exported. There is no data connection. No dashboards available in the dashboard list.

INFO Helm client version: 3.14.4 Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. INFO Kubernetes client version: v1.27.12 INFO Kubernetes server version: v1.27.9

Deploying logging components to the [logging] namespace [Fri Jun 7 16:48:51 CEST 2024]

secret/internal-user-kibanaserver labeled secret/v4m-root-ca-tls-secret created secret/v4m-root-ca-tls-secret annotate secret/v4m-root-ca-tls-secret labeled secret/kibana-tls-secret created secret/kibana-tls-secret annotate secret/kibana-tls-secret labeled

"opensearch" has been added to your repositories secret/v4m-osd-tls-enabled created INFO Deploying OpenSearch Dashboards Release "v4m-osd" does not exist. Installing it now. NAME: v4m-osd LAST DEPLOYED: Fri Jun 7 16:51:00 2024 NAMESPACE: logging STATUS: deployed REVISION: 1 TEST SUITE: None NOTES:

  1. Get the application URL by running these commands: https://myexample.com/dashboards INFO OpenSearch Dashboards has been deployed serviceaccount/v4m-osd-dashboards patched

secret/internal-user-admin labeled secret/internal-user-logcollector labeled secret/internal-user-metricgetter labeled secret/es-transport-tls-secret created secret/es-transport-tls-secret annotate secret/es-transport-tls-secret labeled secret/es-rest-tls-secret created secret/es-rest-tls-secret annotate secret/es-rest-tls-secret labeled secret/es-admin-tls-secret created secret/es-admin-tls-secret annotate secret/es-admin-tls-secret labeled secret/opensearch-cert-subjects created secret/opensearch-cert-subjects labeled configmap/run-securityadmin.sh created configmap/run-securityadmin.sh labeled "opensearch" has been added to your repositories

INFO Copying OpenSearch Security Configuration files from [/home/ashokm/viya4-monitoring-kubernetes/logging/opensearch/securityconfig] secret/opensearch-securityconfig created secret/opensearch-securityconfig labeled INFO Deploying OpenSearch

Release "opensearch" does not exist. Installing it now. NAME: opensearch LAST DEPLOYED: Fri Jun 7 16:56:13 2024 NAMESPACE: logging STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Watch all cluster members come up. $ kubectl get pods --namespace=logging -l app.kubernetes.io/component=v4m-search -w INFO Waiting on OpenSearch pods to be Ready pod/v4m-search-0 condition met Looping: 0 Fri Jun 7 14:59:01 UTC 2024 RC: 0 | run_securityadmin.sh script starting [Fri Jun 7 14:59:01 UTC 2024] | ** | This tool will be deprecated in the next major release of OpenSearch | https://github.com/opensearch-project/security/issues/1755 | ** | Security Admin v7 | Will connect to localhost:9200 ... done | Connected as "CN=es-admin,O=v4m" | OpenSearch Version: 2.12.0 | Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ... | Clustername: opensearch-cluster | Clusterstate: GREEN | Number of nodes: 3 | Number of data nodes: 3 | .opendistro_security index does not exists, attempt to create it ... done (0-all replicas) | Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/ | Will update '/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml | SUCC: Configuration for 'config' created or updated | Will update '/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml | SUCC: Configuration for 'roles' created or updated | Will update '/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml | SUCC: Configuration for 'rolesmapping' created or updated | Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml | SUCC: Configuration for 'internalusers' created or updated | Will update '/actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml | SUCC: Configuration for 'actiongroups' created or updated | Will update '/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml | SUCC: Configuration for 'tenants' created or updated | Will update '/nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml | SUCC: Configuration for 'nodesdn' created or updated | Will update '/whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml | SUCC: Configuration for 'whitelist' created or updated | Will update '/allowlist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/allowlist.yml | SUCC: Configuration for 'allowlist' created or updated | SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null | SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null | SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null | Done with success INFO OpenSearch has been deployed

INFO Deploying Elasticsearch metric exporter ...

"prometheus-community" has been added to your repositories

Release "es-exporter" does not exist. Installing it now. NAME: es-exporter LAST DEPLOYED: Fri Jun 7 17:00:34 2024 NAMESPACE: logging STATUS: deployed REVISION: 1 TEST SUITE: None NOTES:

  1. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace logging -l "app=v4m-es-exporter" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:9108/metrics to use your application" kubectl port-forward $POD_NAME 9108:9108 --namespace logging INFO Elasticsearch metric exporter has been deployed

INFO Loading Content into OpenSearch ERROR Unable to identify the temporary port used for port-forwarding [v4m-search]; exiting script. ERROR Exiting script [deploy_logging.sh] due to an error executing the command [logging/bin/deploy_opensearch_content.sh].

Resources deployed in logging namespace:

NAME READY STATUS RESTARTS AGE pod/v4m-es-exporter-6854fd79f7-gl72g 1/1 Running 0 31m pod/v4m-fb-459n2 0/1 Pending 0 11m pod/v4m-fb-4n57z 1/1 Running 0 11m pod/v4m-fb-9ztnh 1/1 Running 0 12m pod/v4m-fb-kvrnl 1/1 Running 0 12m pod/v4m-fb-lm79d 1/1 Running 0 12m pod/v4m-fb-ncb6h 1/1 Running 0 12m pod/v4m-fb-pkcl4 1/1 Running 0 12m pod/v4m-fb-wqw25 1/1 Running 0 12m pod/v4m-fb-zrg9p 1/1 Running 0 12m pod/v4m-osd-78f8ddfc57-j98k5 1/1 Running 0 40m pod/v4m-search-0 1/1 Running 0 35m pod/v4m-search-1 1/1 Running 0 35m pod/v4m-search-2 1/1 Running 0 35m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/v4m-es-exporter ClusterIP 10.0.3.144 9108/TCP 31m service/v4m-fb ClusterIP 10.0.180.50 2020/TCP 12m service/v4m-osd ClusterIP 10.0.253.147 5601/TCP 40m service/v4m-search ClusterIP 10.0.121.64 9200/TCP,9300/TCP 35m service/v4m-search-headless ClusterIP None 9200/TCP,9300/TCP,9600/TCP 35m

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/v4m-fb 9 9 8 2 8 12m

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/v4m-es-exporter 1/1 1 1 31m deployment.apps/v4m-osd 1/1 1 1 40m

NAME DESIRED CURRENT READY AGE replicaset.apps/v4m-es-exporter-6854fd79f7 1 1 1 31m replicaset.apps/v4m-osd-78f8ddfc57 1 1 1 40m

NAME READY AGE statefulset.apps/v4m-search 3/3 35m

I also tried executing deploy_osd_conten.sh that fails with same response:

INFO Helm client version: 3.14.4 INFO Kubernetes client version: v1.27.12 INFO Kubernetes server version: v1.27.9 INFO Loading user environment file: /home/ashokom/viya4-monitoring-kubernetes/logging/user.env

INFO Configuring OpenSearch Dashboards...this may take a few minutes INFO Waiting for OpenSearch Dashboards pods to be ready (Fri Jun 7 17:29:02 CEST 2024 - timeout 10m) pod/v4m-osd-78f8ddfc57-j98k5 condition met INFO Waiting (up to more 8 minutes) for OpenSearch Dashboards API endpoint to be ready ERROR Unable to identify the temporary port used for port-forwarding [v4m-osd]; exiting script. ERROR Unable to obtain the URL for the OpenSearch Dashboards API Endpoint

ahsokm07 commented 3 weeks ago
opsen
ahsokm07 commented 3 weeks ago

FYI this specifically happens only in WSL environment.

gsmith-sas commented 3 weeks ago

@ahsokm07 We are sorry you are running into this problem. Thank you for including the entire set of log messages generated by the script, that is very helpful.

As a next step, we will have you run the deploy_opensearch_content.sh script (NOT the deploy_osd_content.sh script) in the logging/bin directory of the project repo. But, please set some environment variables to enable some additional debugging. Please set the environment variable LOG_DEBUG_ENABLE to 'true'. This will generate some DEBUG-level messages that might be helpful. In addition, please set the environment variable KEEP_TMP_DIR to 'true'. This will keep some temporary files generated during the deployment process around so we can review them after the script completes execution. You can set these environment variables and run the script in a single command using the following syntax: LOG_DEBUG_ENABLE=true KEEP_TMP_DIR=true ./logging/bin/deploy_opensearch_content.sh Please share the output generated (including all messages) when the script executes.

In addition, one of the last messages displayed will be something like this: INFO TMP_DIR [/tmp/sas.mon.jq0ztxIp] was not removed. Please share the contents of the output.txt file found in the temporary directory identified in the message.

Finally, the error messages are reporting a problem when attempting to use Kubernetes port-forwarding capability. Please try the following command to see if any helpful error messages are generated when you attempt to use port-forwarding manually: kubectl -n logging port-forward --address localhost svc/v4m-search :9200 If the command executes successfully, you will see a couple of messages like the following:

$ kubectl -n logging port-forward --address localhost svc/v4m-search :9200
Forwarding from 127.0.0.1:37813 -> 9200
Forwarding from [::1]:37813 -> 9200

You can terminate the port-forwarding by pressing Ctrl-C which should return you to the command line. Please let us know if that command succeeded or failed and share any error messages generated.

I regularly use WSL with an Ubuntu image in my work, so I don't think WSL is the issue. Please let me know if you are using WSL with a different image.

Regards, Greg