[kubernetes] API does not start from helm chart

s3than commented 5 years ago

Expected Behavior

Application to start

Current Behavior

API gets to the following stage and then after readinessProbe timeout will restart.

Configure and run the helm chart pointing to mongodb and elasticsearch, enable debug Fix to debug, other issue

Last log item it as below, no additional errors or other information provided.

00:22:14.550 [graviteeio-node] DEBUG i.g.p.c.i.CachedPluginClassLoaderFactory - Created plugin ClassLoader for repository-elasticsearch with classpath [file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/lib/gravitee-common-elasticsearch-1.22.0.jar, file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/lib/rxjava-2.1.9.jar, file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/lib/reactive-streams-1.0.2.jar, file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/lib/vertx-web-client-3.5.4.jar, file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/lib/vertx-rx-java2-3.5.4.jar, file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/lib/vertx-web-common-3.5.4.jar, file:/opt/graviteeio-management-api/plugins/.work/gravitee-repository-elasticsearch-1.22.0/gravitee-repository-elasticsearch-1.22.0.jar]

Possible Solution

Steps to Reproduce (for bugs)

1.update the mongo and elasticsearch connection string 2.helm install 3.wait for ui, and gateway to start 4.wait for api to restart us never get's to be ready.

Context

Unable to POC the application

Your Environment

1.22.0 and 1.23.0
kubernetes 1.11.7
ElasticSearch 5 and 6
Mongodb 3 and 4

s3than commented 5 years ago

Removing the plugin for elasticsearch allows the api to start however at this point we don't have any details as to why the elasticsearch plugin is causing the issue.

brasseld commented 5 years ago

So, the gateway is starting well but not the mgmt-api, right ?

s3than commented 5 years ago

Yes it does have some errors but appears to be running.

However this is only determined via the running state of the pod

s3than commented 5 years ago

@brasseld Let me know what I can run to produce more information.

The only changes to the helm chart where the connection to

MongoDB connection string Elasticsearch connection string

However, I did find that the index doesn't get created when running the startup. Manually creating the index did not affect the outcome.

brasseld commented 5 years ago

Can you provide the complete log file here please for both the gateway and the management-api.

Thanks a lot

s3than commented 5 years ago

Api Logs api.log Gateway gateway.log

brasseld commented 5 years ago

It seems that the gateway and the management-api are not able to reach Elasticsearch...

s3than commented 5 years ago

I can curl from the pod to elastic search though? I'll do some more tests

On Thu, 21 Mar 2019, 18:13 Brassely David, notifications@github.com wrote:

It seems that the gateway and the management-api are not able to reach Elasticsearch...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/gravitee-io/issues/issues/2050#issuecomment-475141970, or mute the thread https://github.com/notifications/unsubscribe-auth/AGkBcP9a5jV-ObhcZppa96TC6HQB0ZG3ks5vYz8cgaJpZM4b4wWg .

s3than commented 5 years ago

Ok going through the logs, I can't see anything that says where it's trying to connect to? Just that it can't reach "elasticsearch" This makes it hard to debug if it's one of the following

Misconfigured in Gravitee config eg. connection string incorrect
Or Elasticsearch not responding to gravitee therefor connection timeout

Ok so I've exec'd into the Gateway

` /opt/graviteeio-gateway # wget -O- http://admin:tvjetvx7@gravitee-elasticsearch.poc.svc.cluster.local:9200/_cluster/health --2019-03-21 10:28:40-- http://admin:*password*@gravitee-elasticsearch.poc.svc.cluster.local:9200/_cluster/health Resolving gravitee-elasticsearch.poc.svc.cluster.local... 100.67.230.47 Connecting to gravitee-elasticsearch.poc.svc.cluster.local|100.67.230.47|:9200... connected. HTTP request sent, awaiting response... 401 Unauthorized Authentication selected: Basic realm="Search Guard" Reusing existing connection to gravitee-elasticsearch.poc.svc.cluster.local:9200. HTTP request sent, awaiting response... 200 OK Length: 399 [application/json] Saving to: 'STDOUT'

0%[ ] 0 --.-KB/s {"cluster_name":"gravitee-elasticsearch","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":6,"active_shards":13,"relocating_shards":0,"initializing_shards":0,"u- 100%[======================================================================================================================>] 399 --.-KB/s in 0s

2019-03-21 10:28:41 (67.0 MB/s) - written to stdout [399/399]

/opt/graviteeio-gateway # ` The elasticsearch is a poc standalone and is available from the gateway but more importantly I can see a 401... so I'll be looking there even though I get the response back

s3than commented 5 years ago

Re-running with Curl I get

`curl -v http://admin:tvjetvx7@gravitee-elasticsearch.poc.svc.cluster.local:9200/_cluster/health

Trying 100.67.230.47...
TCP_NODELAY set
Connected to gravitee-elasticsearch.poc.svc.cluster.local (100.67.230.47) port 9200 (#0)
Server auth using Basic with user 'admin'

GET /_cluster/health HTTP/1.1 Host: gravitee-elasticsearch.poc.svc.cluster.local:9200 Authorization: Basic YWRtaW46dHZqZXR2eDc= User-Agent: curl/7.61.1 Accept: /

< HTTP/1.1 200 OK < access-control-allow-credentials: true < content-type: application/json; charset=UTF-8 < content-length: 399 <
Connection #0 to host gravitee-elasticsearch.poc.svc.cluster.local left intact {"cluster_name":"gravitee-elasticsearch","status":"green","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":3,"active_primary_shards":6,"active_shards":13,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}`

s3than commented 5 years ago

At the moment I'm not sure where to proceed from here as the cluster is accessible from the gateway, the auth as provided to the config looks ok

brasseld commented 5 years ago

What I do not understand here is that others are running the same helm chart without any issues.

Did you set any CPU / memory restrictions?

s3than commented 5 years ago

Could this be blocking the request? Connecting to gravitee-elasticsearch.poc.svc.cluster.local|100.67.230.47|:9200... connected. HTTP request sent, awaiting response... 401 Unauthorized Authentication selected: Basic realm="Search Guard"

brasseld commented 5 years ago

No it couldn't.

How did you size pods for gw and api ?

s3than commented 5 years ago

1 pod for api 2 pod for gateway

No scaling

The gravitee.yml is attached the cluster and mongo are all poc with temp passwords so no issues providing it here. gravitee.txt

Going from the documentation it looks fine.

Could there be an issue with environment variables? On the gateway there is some set, but no reporter env's such as below

GRAVITEE_REPORTERS_ELASTICSEARCH_ENDPOINTS_0

brasseld commented 5 years ago

Hum.., I'm not sure about the way you're managing authentication: http://admin:tvjetvx7@gravitee-elasticsearch.poc.svc.cluster.local:9200

Please try with:

reporters:
  elasticsearch:
    endpoints:
      - http://gravitee-elasticsearch.poc.svc.cluster.local:9200
    security:
      username: admin
      password: tvjetvx7

And do the same for mgmt-api

s3than commented 5 years ago

`
elasticsearch: enabled: true # Is the reporter enabled or not (default to true) endpoints:

{{ .Values.es.protocol }}://{{ .Values.es.host }}:{{ .Values.es.port }} index: {{ .Values.es.index }} cluster: {{ .Values.es.cluster }} bulk: actions: 200 # Number of requests action before flush flush_interval: 5 # Flush interval in seconds concurrent_requests: 5 # Concurrent requests ` This is from the helm chart I'll try the changes and get a PR for the helm chart as well if it works

s3than commented 5 years ago

Damn sorry having code edit issues for some reason in github, But short story auth isn't configured for helm chart

aelamrani commented 5 years ago

@s3than here's an exemple with credentials https://github.com/gravitee-io/gravitee-gateway/blob/master/gravitee-gateway-standalone/gravitee-gateway-standalone-distribution/src/main/resources/config/gravitee.yml#L129

s3than commented 5 years ago

I've confirmed that resolves the issue, if you can leave this issue open till tomorrow so I can confirm at work tomorrow. I'll also have a PR for the helm chart and elastic search auth for you by COB tomorrow

s3than commented 5 years ago

@aelamrani Thanks @brasseld Thanks

Unfortunately, I reviewed the config in the helm chart and assumed a standard http connection for the elastic search.

Thanks again

s3than commented 5 years ago

Please find the changes for the Helm chart

Elasticsearch url as string and multiple endpoints
Elasticsearch security values now settable via config

https://github.com/gravitee-io/gravitee-kubernetes/pull/13

Thanks again for your help.

brasseld commented 5 years ago

Thanks a lot @s3than

Closed by https://github.com/gravitee-io/gravitee-kubernetes/pull/13

gravitee-io / issues