Gravitee 3.3 | Elasticsearch connection Problems

[IvoIlic95]

Hello guys

Currently I am trying to install the new Gravitee 3.3 on OpenShift BUT there are some issues.

The only problem factor is Elasticsearch which causes some issues while configuring and starting Gravitee.

I have added the following environment variables for the APIM-Gateway and APIM-Management-Api:

---

Gateway:

Management-Api:

---

The log output inside the OpenShift Console:

Gateway:

INFO i.g.n.r.vertx.ReporterManagerImpl - Starting reporter: io.gravitee.node.reporter.vertx.eventbus.EventBusReporterWrapper@76847bdc INFO i.g.n.m.h.v.e.ManagementEndpointManagerImpl - Register a new endpoint for Management API: GET / [io.gravitee.node.management.http.node.NodeEndpoint] INFO i.g.n.m.h.v.e.ManagementEndpointManagerImpl - Register a new endpoint for Management API: GET /configuration [io.gravitee.node.management.http.configuration.ConfigurationEndpoint] INFO i.g.n.m.h.v.v.ManagementVerticle - HTTP listener for Node Management bind to port TCP:18082 INFO i.g.n.r.vertx.ReporterManagerImpl - Starting reporter: io.gravitee.node.reporter.vertx.eventbus.EventBusReporterWrapper@3331c002 INFO i.g.common.service.AbstractService - Initializing service io.gravitee.reporter.elasticsearch.ElasticsearchReporter INFO i.g.r.e.ElasticsearchReporter - Starting Elastic reporter engine... WARN i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-worker-thread-0,5,main]=Thread[vert.x-worker-thread-0,5,main] has been blocked for 60403 ms, time limit is 60000 ms   | io.vertx.core.VertxException: Thread blocked

With this current config I am getting this log ouput where it freezes at "Starting Elastic reporter engine..." and then after some time I get the "Thread blocked" error. This behavior only occurs at the Gateway.

Management-Api:

INFO i.g.r.a.r.p.RepositoryPluginHandler - Register a new repository: repository-elasticsearch [io.gravitee.repository.elasticsearch.ElasticsearchRepository] INFO i.g.r.a.r.p.RepositoryPluginHandler - Repository [ANALYTICS] loaded by elasticsearch

The Management-Api freezes at "Repository [ANALYTICS] loaded by elasticsearch". Now I do not know if this means "Yeah it was all successful" or should be there more log output? This behavior occurs only at the Management-Api.

Moreover, my coworker provided me two Elasticsearch links: esprod01.xxx.xxx.com:9200 esprod01api.xxx.xxx.com:9200

Which one should be used for the Elasticsearch connection?

Many thanks for the help in advance!

Kind regards, Ivo Ilic, BSc

[aelamrani]

Hi @IvoIlic95

Which one should be used for the Elasticsearch connection?

Difficult to say, as we don't know what your coworker shares with you, maybe you can ask him?

Are you able to try to curl both from the gateway container?

[IvoIlic95]

Good morning

Asking my coworker will be quite difficult. I just got the message that he is on a break till Saturday ... sadly ...

I have tried CURL inside the terminal but it does not get recognized. Therefore, I used wget and this is what I got:

wget https://esprod01.vih.xxx.com:9200/ Connecting to esprod01.vih.xxx.com:9200 (172.xx.xxx.28:9200) ssl_client: esprod01.vih.xxx.com: handshake failed: Connection reset by peer wget: error getting response: Connection reset by peer

I am getting this error non stop when using this URL " https://esprod01.vih.xxx.com:9200/ ". Somehow ElasticSearch does not accept when Gravitee (OpenShift) tries to connect.

I have also made a build that installs the SSLCA3 and RootCA2 certificate for every pod (gateway, management-api, portal-ui, management-ui).

These issues must come from ElasticSearch itself, because I do not think that this is a problem from Gravitee or OpenShift.

Kind regards, Ivo

[aelamrani]

Can you try without the check of the certificate?

wget --no-check-certificate ...

[IvoIlic95]

Yeah sure! Here is the output but it will not surprise you:

wget --no-check-certificate https://esprod01.vih.xxx.com:9200/

Connecting to esprod01.vih.xxx.com:9200 (172.xx.xxx.28:9200) ssl_client: esprod01.vih.xxx.com: handshake failed: Connection reset by peer wget: error getting response: Connection reset by peer

[aelamrani]

Ok so seems this is not related to the SSL but to the connection itself to the ES endpoint: Connection reset by peer

[IvoIlic95]

What does "Connection reset by peer" mean. I have googled it but I did not get a logical answer to that.

So this means that the issue is from ElasticSearch itself or is it not possible to say that tight now?

[aelamrani]

Means that the TCP connection has been reset on the server side (ES or a proxy)

[IvoIlic95]

Hmm ... well I need to contact my coworker. Hopefully he will find something in Elastic that causes the problem. What do you think could be the problem Mr. Elamrani Azize? Do you have a guess what is the troublemaker?

[aelamrani]

I would say that it can be a problem on the link between the Gateway pod and your ES.

Because, regarding your previous message, the REST-API is starting well and the connection with ES is successful.

[IvoIlic95]

I have to figure it out why the Gateway is such a troublemaker. I could be that OpenShift or Elasticsearch is in Diva mode. Maybe there is a configuration problem on the OpenShift or Elasticsearch side.

Yeah, the Management-Api has a successful connection.

By the way: My coworker provided me two URLs, a Elasticsearch Cluster URL and a Elasticsearch API URL. Now I do not know which URL should be used for the gateway and management-api...

[aelamrani]

I would suggest to use the cluster URL as I suppose it is the URL of the infrastructure component in front of your ES instances.

[IvoIlic95]

Okay thank you.

I have now tried a different ElasticSearch DB with the following Cluster URL: http://xxxxx9999.xxx.com:9200/ This URL is from the Gravitee 1.3 ElasticSearch. I included this URL in the new Gravitee (3.3) config and it worked perfectly.

Unfortunately, the URL is only http which results into that my company has worries about the security. The other Elastic URL (which I want to use for the Gravitee 3.3) is https (this is the preferred one) and this wont work. So the issue must be on the ElasticSearch side.

[aelamrani]

We have a lot of users deploying Gravitee with an ES exposed in HTTPS. So this is probably an infrastructure problem.

[IvoIlic95]

Okay yeah, I will contact my coworker and then we will look where the problem actually is. For now thank you for your help. Please let this thread open if it is possible. I will write back hopefully on Monday!

Kind regards, Ivo

[IvoIlic95]

We have a lot of users deploying Gravitee with an ES exposed in HTTPS. So this is probably an infrastructure problem.

Good morning

Do you have a paper (config file ... something like this) which describes how Elasticsearch itself should be configured (when I want to use it for Gravitee)?

And is there a instruction on how to set up Gravitee on OpenShift? The instruction (Configuration Guide) on Gravitee itself does not help me that much, because there is no description which environment variables need to be set up, so that Gravitee runs on OpenShift.

Kind regards, Ivo

[aelamrani]

Hi @IvoIlic95

No we do not provide a configuration of Elasticsearch because there is no specific things to do for a proper use with Gravitee.io except provide enough resource to handle indexation / retention. But as this can vary a lot depending on your calls volume / data retention...

Here a link on our blog explaining how to install Gravitee.io on K8S: https://medium.com/graviteeio/gravitee-io-apim-on-kubernetes-c0c0107d5e3c

Hope it helps

[IvoIlic95]

Good morning

For now I am using a old deployment of Elasticsearch which works fine.

Now I am struggling to configure the Management-UI, Portal-UI and the Management-Api who are dependent on this.

In OpenShift you can expose a route (I exposed a route for the Management-Api). Do I need to paste this route to the Management-UI and Portal-UI?

Here is what I have done:

1. Expose a route for the Management-Api
2. Paste the exposed route with the right path to the Environment variables of the Management-UI and Portal-UI.

Now I do not know if I have to expose a new route for the Portal-UI and Management-UI.

Here are some pictures which shows what I have done or more to say configured so far:

Management-API:

Management-UI:

Portal-UI:

I have tried this example to reproduce: https://github.com/gravitee-io/gravitee-docker/blob/master/platform/3.x/docker-compose.yml#L139 but without any success.

The question is, how do I know which route I need to create and paste to the Portal-UI and Management-UI so that I can access the web interface for Gravitee 3.*.

Kind regards, Ivo

[aelamrani]

Hi @IvoIlic95

May I ask you to create a dedicated issue for this different subject? And close this one if it is resolved.

Thanks, Azize

[IvoIlic95]

Okay, no Problem! I will copy my problem to a new issue!

[aelamrani]

Thanks, closing this one then