search

gravitee-io / issues

Gravitee.io - API Platform - Issues
64 stars 26 forks source link

AM + Kubernetes (Minikube) gives me "io.vertx.core.VertxException: Thread blocked!" #6303

Closed andrearossi90 closed 3 years ago

andrearossi90 commented 3 years ago

Hi, I'm not new to Gravitee, but I'm pretty new to Kubernetes. My current setup is:

Here the steps to reproduce the problem:

  1. Install Minikube
  2. Install Helm via script
  3. Install and start Docker
  4. Add user to Docker group so it can use docker even if not root
  5. Start Minikube minikube start --addons ingress --kubernetes-version=v1.21.5
  6. Bring up Gravitee AM helm install graviteeio-am graviteeio/am --set ui.service.type=NodePort --set mongodb-replicaset.enabled=true --set "mongo.uri=mongodb+srv://graviteeio-am-mongodb-replicaset.default.svc.cluster.local/?tls=false&ssl=false" --set api.service.type=NodePort --set gateway.service.type=NodePort
  7. Gateway and Management API cannot properly boot due to a long stack trace with root io.vertx.core.VertxException: Thread blocked

What can I do?

Thanks in advance.

leleueri commented 3 years ago

Hi,

Could you provide to us the logs output to see the full stacktrace and previous logs messages ?

Thanks

andrearossi90 commented 3 years ago

@leleueri

Management API (attaching the complete log):

management.log

=========================================================================

  Gravitee.io - Access Management - Management API

  GRAVITEE_HOME: /opt/graviteeio-am-management-api

  GRAVITEE_OPTS: 

  JAVA: /opt/java/openjdk/bin/java

  JAVA_OPTS:  -Djava.net.preferIPv4Stack=true -Xms256m -Xmx256m -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8

[management.log](https://github.com/gravitee-io/issues/files/7277800/management.log)

  CLASSPATH: /opt/graviteeio-am-management-api/lib/gravitee-am-management-api-standalone-bootstrap-3.11.0.jar

=========================================================================

11:19:30.063 [graviteeio-node] [] INFO  i.g.n.c.s.e.PropertiesConfiguration - Loading Gravitee configuration.

11:19:30.237 [graviteeio-node] [] INFO  i.g.n.c.s.e.PropertiesConfiguration -   Gravitee configuration loaded from /opt/graviteeio-am-management-api/config/gravitee.yml

11:19:30.912 [graviteeio-node] [] INFO  i.g.n.c.s.e.PropertiesConfiguration - Loading Gravitee configuration. DONE

11:19:31.046 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor - Loading plugin handlers

11:19:31.238 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor - Find 15 plugin handler(s):

11:19:31.242 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.resource.plugin.ResourcePluginHandler

11:19:31.254 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.plugin.policy.internal.PolicyPluginHandler

11:19:31.375 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.node.plugins.service.handler.ServicePluginHandler

11:19:31.442 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.node.tracing.plugin.TracingPluginHandler

11:19:31.503 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.plugin.notifier.internal.NotifierPluginHandler

11:19:31.516 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.botdetection.plugin.BotDetectionPluginHandler

11:19:31.532 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.certificate.plugin.CertificatePluginHandler

11:19:31.551 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.plugin.repository.internal.RepositoryPluginHandler

11:19:31.554 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.node.reporter.plugin.ReporterPluginHandler

11:19:31.573 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.plugin.cockpit.CockpitPluginHandler

11:19:31.615 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.factor.plugin.FactorPluginHandler

11:19:31.620 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.idp.plugin.IdentityProviderPluginHandler

11:19:31.624 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.reporter.plugin.ReporterPluginHandler

11:19:31.681 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.am.plugins.extensiongrant.plugin.ExtensionGrantPluginHandler

11:19:31.763 [graviteeio-node] [] INFO  i.g.p.c.s.PluginHandlerBeanFactoryPostProcessor -   io.gravitee.plugin.alert.internal.AlertPluginHandler

11:19:40.724 [graviteeio-node] [] INFO  i.g.n.c.NodeDeployerFactoriesLoader - Loading instances for type io.gravitee.node.api.NodeDeployer

11:19:40.910 [graviteeio-node] [] INFO  i.g.node.tracing.TracingService - Tracing support is enabled with tracer: name[null]

11:19:51.216 [graviteeio-node] [] INFO  i.g.p.r.i.RepositoryPluginHandler - Loading repository for scope MANAGEMENT: mongodb

11:19:51.216 [graviteeio-node] [] INFO  i.g.p.r.i.RepositoryPluginHandler - Loading repository for scope OAUTH2: mongodb

11:19:53.121 [graviteeio-node] [] INFO  org.eclipse.jetty.util.log - Logging initialized @46888ms to org.eclipse.jetty.util.log.Slf4jLog

11:19:54.999 [graviteeio-node] [] INFO  i.g.node.container.AbstractContainer - Starting Gravitee.io - AM Management API...

11:19:55.080 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode - Gravitee.io - AM Management API is now starting...

11:19:55.103 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: PluginEventListener

11:19:55.122 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service io.gravitee.plugin.core.internal.PluginEventListener

11:19:55.130 [graviteeio-node] [] INFO  i.g.c.event.impl.EventManagerImpl - Register new listener PluginEventListener for event type class io.gravitee.plugin.core.api.PluginEvent

11:19:55.131 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: PluginRegistry

11:19:55.178 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service io.gravitee.plugin.core.internal.PluginRegistryImpl

11:19:55.178 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Initializing plugin registry.

11:19:55.178 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugins from /opt/graviteeio-am-management-api/plugins

11:19:55.183 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-botdetection-recaptcha-v3-3.11.0.zip

11:19:55.359 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-botdetection-recaptcha-v3-3.11.0/gravitee-am-botdetection-recaptcha-v3-3.11.0.jar

11:19:55.417 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-certificate-javakeystore-3.11.0.zip

11:19:55.505 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-certificate-javakeystore-3.11.0/gravitee-am-certificate-javakeystore-3.11.0.jar

11:19:55.524 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-certificate-pkcs12-3.11.0.zip

11:19:55.527 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-certificate-pkcs12-3.11.0/gravitee-am-certificate-pkcs12-3.11.0.jar

11:19:55.545 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-extensiongrant-jwtbearer-3.11.0.zip

11:19:55.574 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-extensiongrant-jwtbearer-3.11.0/gravitee-am-extensiongrant-jwtbearer-3.11.0.jar

11:19:55.577 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-factor-email-3.11.0.zip

11:19:55.631 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-factor-email-3.11.0/gravitee-am-factor-email-3.11.0.jar

11:19:55.635 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-factor-otp-3.11.0.zip

11:19:55.765 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-factor-otp-3.11.0/gravitee-am-factor-otp-3.11.0.jar

11:19:55.802 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-factor-sms-3.11.0.zip

11:19:55.849 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-factor-sms-3.11.0/gravitee-am-factor-sms-3.11.0.jar

11:19:55.853 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-azure-ad-3.11.0.zip

11:19:55.906 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-azure-ad-3.11.0/gravitee-am-identityprovider-azure-ad-3.11.0.jar

11:19:55.910 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-facebook-3.11.0.zip

11:19:55.917 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-facebook-3.11.0/gravitee-am-identityprovider-facebook-3.11.0.jar

11:19:55.920 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-franceconnect-3.11.0.zip

11:19:55.928 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-franceconnect-3.11.0/gravitee-am-identityprovider-franceconnect-3.11.0.jar

11:19:55.932 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-github-3.11.0.zip

11:19:56.007 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-github-3.11.0/gravitee-am-identityprovider-github-3.11.0.jar

11:19:56.010 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-google-3.11.0.zip

11:19:56.020 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-google-3.11.0/gravitee-am-identityprovider-google-3.11.0.jar

11:19:56.026 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-gravitee-3.11.0.zip

11:19:56.099 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-gravitee-3.11.0/gravitee-am-identityprovider-gravitee-3.11.0.jar

11:19:56.101 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-http-3.11.0.zip

11:19:56.106 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-http-3.11.0/gravitee-am-identityprovider-http-3.11.0.jar

11:19:56.108 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-inline-3.11.0.zip

11:19:56.113 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-inline-3.11.0/gravitee-am-identityprovider-inline-3.11.0.jar

11:19:56.116 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-jdbc-3.11.0.zip

11:19:56.499 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-jdbc-3.11.0/gravitee-am-identityprovider-jdbc-3.11.0.jar

11:19:56.501 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-ldap-3.11.0.zip

11:19:56.718 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-ldap-3.11.0/gravitee-am-identityprovider-ldap-3.11.0.jar

11:19:56.738 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-linkedin-3.11.0.zip

11:19:56.746 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-linkedin-3.11.0/gravitee-am-identityprovider-linkedin-3.11.0.jar

11:19:56.799 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-mongo-3.11.0.zip

11:19:56.915 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-mongo-3.11.0/gravitee-am-identityprovider-mongo-3.11.0.jar

11:19:56.917 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-oauth2-generic-3.11.0.zip

11:19:56.922 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-oauth2-generic-3.11.0/gravitee-am-identityprovider-oauth2-generic-3.11.0.jar

11:19:56.924 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-salesforce-3.11.0.zip

11:19:56.927 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-salesforce-3.11.0/gravitee-am-identityprovider-salesforce-3.11.0.jar

11:19:57.003 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-identityprovider-twitter-3.11.0.zip

11:19:57.024 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-identityprovider-twitter-3.11.0/gravitee-am-identityprovider-twitter-3.11.0.jar

11:19:57.036 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-management-api-services-purge-3.11.0.zip

11:19:57.041 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-management-api-services-purge-3.11.0/gravitee-am-management-api-services-purge-3.11.0.jar

11:19:57.114 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-management-api-services-sync-3.11.0.zip

11:19:57.127 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-management-api-services-sync-3.11.0/gravitee-am-management-api-services-sync-3.11.0.jar

11:19:57.134 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-policy-enrich-auth-flow-3.11.0.zip

11:19:57.212 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-policy-enrich-auth-flow-3.11.0/gravitee-am-policy-enrich-auth-flow-3.11.0.jar

11:19:57.214 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-policy-enrich-profile-3.11.0.zip

11:19:57.218 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-policy-enrich-profile-3.11.0/gravitee-am-policy-enrich-profile-3.11.0.jar

11:19:57.220 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-reporter-file-3.11.0.zip

11:19:57.228 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-reporter-file-3.11.0/gravitee-am-reporter-file-3.11.0.jar

11:19:57.230 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-reporter-jdbc-3.11.0.zip

11:19:57.653 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-reporter-jdbc-3.11.0/gravitee-am-reporter-jdbc-3.11.0.jar

11:19:57.660 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-reporter-kafka-3.11.0.zip

11:19:57.745 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-reporter-kafka-3.11.0/gravitee-am-reporter-kafka-3.11.0.jar

11:19:57.806 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-reporter-mongodb-3.11.0.zip

11:19:57.823 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-reporter-mongodb-3.11.0/gravitee-am-reporter-mongodb-3.11.0.jar

11:19:57.827 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-repository-jdbc-3.11.0.zip

11:19:58.109 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-repository-jdbc-3.11.0/gravitee-am-repository-jdbc-3.11.0.jar

11:19:58.112 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-repository-mongodb-3.11.0.zip

11:19:58.314 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-repository-mongodb-3.11.0/gravitee-am-repository-mongodb-3.11.0.jar

11:19:58.316 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-resource-infobip-3.11.0.zip

11:19:58.509 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-resource-infobip-3.11.0/gravitee-am-resource-infobip-3.11.0.jar

11:19:58.515 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-resource-smtp-3.11.0.zip

11:19:58.521 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-resource-smtp-3.11.0/gravitee-am-resource-smtp-3.11.0.jar

11:19:58.524 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-am-resource-twilio-3.11.0.zip

11:19:58.982 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-am-resource-twilio-3.11.0/gravitee-am-resource-twilio-3.11.0.jar

11:19:59.008 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-cockpit-connectors-ws-2.0.0.zip

11:19:59.023 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-cockpit-connectors-ws-2.0.0/gravitee-cockpit-connectors-ws-2.0.0.jar

11:19:59.023 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-notifier-email-1.3.0.zip

11:19:59.140 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-notifier-email-1.3.0/gravitee-notifier-email-1.3.0.jar

11:19:59.144 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-notifier-webhook-1.1.0.zip

11:19:59.148 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-notifier-webhook-1.1.0/gravitee-notifier-webhook-1.1.0.jar

11:19:59.150 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-policy-callout-http-1.13.0.zip

11:19:59.157 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-policy-callout-http-1.13.0/gravitee-policy-callout-http-1.13.0.jar

11:19:59.159 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-policy-groovy-1.14.1.zip

11:19:59.601 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-policy-groovy-1.14.1/gravitee-policy-groovy-1.14.1.jar

11:19:59.607 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-policy-ipfiltering-1.8.0.zip

11:19:59.639 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-policy-ipfiltering-1.8.0/gravitee-policy-ipfiltering-1.8.0.jar

11:19:59.713 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Loading plugin from /opt/graviteeio-am-management-api/plugins/gravitee-policy-request-validation-1.12.0.zip

11:19:59.813 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - A plugin manifest has been loaded from: /opt/graviteeio-am-management-api/plugins/.work/gravitee-policy-request-validation-1.12.0/gravitee-policy-request-validation-1.12.0.jar

11:19:59.819 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available bot_detection: 

11:19:59.837 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > google-recaptcha-v3-am-bot-detection [3.11.0] has been loaded

11:19:59.838 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available certificate: 

11:19:59.838 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > javakeystore-am-certificate [3.11.0] has been loaded

11:19:59.838 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > pkcs12-am-certificate [3.11.0] has been loaded

11:19:59.839 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available extension_grant: 

11:19:59.839 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > jwtbearer-am-extension-grant [3.11.0] has been loaded

11:19:59.840 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available factor: 

11:19:59.840 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > email-am-factor [3.11.0] has been loaded

11:19:59.840 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > otp-am-factor [3.11.0] has been loaded

11:19:59.841 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > sms-am-factor [3.11.0] has been loaded

11:19:59.841 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available identity_provider: 

11:19:59.842 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > azure-ad-am-idp [3.11.0] has been loaded

11:19:59.842 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > facebook-am-idp [3.11.0] has been loaded

11:19:59.842 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > franceconnect-am-idp [3.11.0] has been loaded

11:19:59.842 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > github-am-idp [3.11.0] has been loaded

11:19:59.842 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > google-am-idp [3.11.0] has been loaded

11:19:59.842 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > gravitee-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > http-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > inline-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > jdbc-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > ldap-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > linkedin-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > mongo-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > oauth2-generic-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > salesforce-am-idp [3.11.0] has been loaded

11:19:59.843 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > twitter-am-idp [3.11.0] has been loaded

11:19:59.900 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available service: 

11:19:59.902 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > purge-service [3.11.0] has been loaded

11:19:59.907 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > sync-service [3.11.0] has been loaded

11:19:59.907 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available policy: 

11:19:59.908 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > policy-am-enrich-auth-flow [3.11.0] has been loaded

11:19:59.908 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > policy-am-enrich-profile [3.11.0] has been loaded

11:19:59.908 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > policy-http-callout [1.13.0] has been loaded

11:19:59.909 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > groovy [1.14.1] has been loaded

11:19:59.917 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > ip-filtering [1.8.0] has been loaded

11:19:59.918 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > policy-request-validation [1.12.0] has been loaded

11:19:59.918 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available am-reporter: 

11:19:59.918 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > reporter-am-file [3.11.0] has been loaded

11:19:59.919 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > reporter-am-jdbc [3.11.0] has been loaded

11:19:59.920 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > reporter-am-kafka [3.11.0] has been loaded

11:19:59.920 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > mongodb [3.11.0] has been loaded

11:19:59.921 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available repository: 

11:19:59.921 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > repository-am-jdbc [3.11.0] has been loaded

11:19:59.921 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > repository-am-mongodb [3.11.0] has been loaded

11:19:59.921 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available resource: 

11:19:59.921 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > infobip-am-resource [3.11.0] has been loaded

11:19:59.924 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > smtp-am-resource [3.11.0] has been loaded

11:19:59.925 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > twilio-verify-am-resource [3.11.0] has been loaded

11:19:59.925 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available cockpit: 

11:19:59.925 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > cockpit-connectors-ws [2.0.0] has been loaded

11:19:59.933 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - List of available notifier: 

11:19:59.936 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > email-notifier [1.3.0] has been loaded

11:19:59.937 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl -   > webhook-notifier [1.1.0] has been loaded

11:19:59.938 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - All plugins have been loaded. Installing...

11:20:00.004 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing repository plugins...

11:20:00.060 [graviteeio-node] [] INFO  i.g.p.r.i.RepositoryPluginHandler - Register a new repository: repository-am-jdbc [io.gravitee.am.repository.jdbc.JdbcRepositoryProvider]

11:20:00.107 [graviteeio-node] [] INFO  i.g.p.r.i.RepositoryPluginHandler - Register a new repository: repository-am-mongodb [io.gravitee.am.repository.mongodb.MongoRepositoryProvider]

11:20:00.112 [graviteeio-node] [] INFO  i.g.p.r.i.RepositoryPluginHandler - Repository [MANAGEMENT] loaded by mongodb

11:20:17.524 [graviteeio-node] [] INFO  i.g.p.r.i.RepositoryPluginHandler - Repository [OAUTH2] loaded by mongodb

11:20:19.500 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing cockpit plugins...

11:20:19.502 [graviteeio-node] [] INFO  i.g.p.cockpit.CockpitPluginHandler - Install plugin: cockpit-connectors-ws [io.gravitee.cockpit.connectors.ws.WebSocketCockpitConnector]

11:20:20.805 [graviteeio-node] [] INFO  i.g.c.c.c.s.MonitoringCollectorService - Starting monitoring collector service

11:20:20.816 [graviteeio-node] [] INFO  i.g.c.c.c.s.AlertManagerService - Starting alert manager service

11:20:20.818 [graviteeio-node] [] INFO  i.g.c.c.ws.WebSocketCockpitConnector - Cockpit connector is disabled.

11:20:20.818 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing service plugins...

11:20:20.818 [graviteeio-node] [] INFO  i.g.n.p.s.h.ServicePluginHandler - Install plugin: purge-service [io.gravitee.am.management.services.purge.ScheduledPurgeService]

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils (file:/opt/graviteeio-am-management-api/lib/ext/spring-core-5.2.14.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

11:20:21.303 [graviteeio-node] [] INFO  i.g.n.p.s.h.ServicePluginHandler - Install plugin: sync-service [io.gravitee.am.management.services.sync.ScheduledSyncService]

11:20:21.424 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing resource plugins...

11:20:21.431 [graviteeio-node] [] INFO  i.g.a.p.r.p.ResourcePluginHandler - Register a new resource plugin: smtp-am-resource [io.gravitee.am.resource.smtp.SmtpResource]

11:20:21.509 [graviteeio-node] [] INFO  i.g.a.p.r.p.ResourcePluginHandler - Register a new resource plugin: infobip-am-resource [io.gravitee.am.resource.infobip.InfobipResource]

11:20:21.512 [graviteeio-node] [] INFO  i.g.a.p.r.p.ResourcePluginHandler - Register a new resource plugin: twilio-verify-am-resource [io.gravitee.am.resource.twilio.TwilioVerifyResource]

11:20:21.513 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing am-reporter plugins...

11:20:21.630 [graviteeio-node] [] INFO  i.g.a.p.r.p.ReporterPluginHandler - Register a new reporter plugin: reporter-am-jdbc [io.gravitee.am.reporter.jdbc.JdbcReporter]

11:20:21.652 [graviteeio-node] [] INFO  i.g.a.p.r.p.ReporterPluginHandler - Register a new reporter plugin: reporter-am-kafka [io.gravitee.am.reporter.kafka.KafkaReporter]

11:20:21.700 [graviteeio-node] [] INFO  i.g.a.p.r.p.ReporterPluginHandler - Register a new reporter plugin: reporter-am-file [io.gravitee.am.reporter.file.FileReporter]

11:20:21.703 [graviteeio-node] [] INFO  i.g.a.p.r.p.ReporterPluginHandler - Register a new reporter plugin: mongodb [io.gravitee.am.reporter.mongodb.MongoReporter]

11:20:21.704 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing factor plugins...

11:20:21.706 [graviteeio-node] [] INFO  i.g.a.p.f.plugin.FactorPluginHandler - Register a new factor plugin: sms-am-factor [io.gravitee.am.factor.sms.SMSFactor]

11:20:21.707 [graviteeio-node] [] INFO  i.g.a.p.f.plugin.FactorPluginHandler - Register a new factor plugin: email-am-factor [io.gravitee.am.factor.email.EmailFactor]

11:20:21.709 [graviteeio-node] [] INFO  i.g.a.p.f.plugin.FactorPluginHandler - Register a new factor plugin: otp-am-factor [io.gravitee.am.factor.otp.OTPFactor]

11:20:21.710 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing identity_provider plugins...

11:20:21.717 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: facebook-am-idp [io.gravitee.am.identityprovider.facebook.FacebookIdentityProvider]

11:20:21.719 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: google-am-idp [io.gravitee.am.identityprovider.google.GoogleIdentityProvider]

11:20:21.721 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: franceconnect-am-idp [io.gravitee.am.identityprovider.franceconnect.FranceConnectIdentityProvider]

11:20:21.722 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: linkedin-am-idp [io.gravitee.am.identityprovider.linkedin.LinkedinIdentityProvider]

11:20:21.723 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: mongo-am-idp [io.gravitee.am.identityprovider.mongo.MongoIdentityProvider]

11:20:21.725 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: azure-ad-am-idp [io.gravitee.am.identityprovider.azure.AzureADIdentityProvider]

11:20:21.802 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: github-am-idp [io.gravitee.am.identityprovider.github.GithubIdentityProvider]

11:20:21.805 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: twitter-am-idp [io.gravitee.am.identityprovider.twitter.TwitterIdentityProvider]

11:20:21.807 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: http-am-idp [io.gravitee.am.identityprovider.http.HttpIdentityProvider]

11:20:21.809 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: oauth2-generic-am-idp [io.gravitee.am.identityprovider.oauth2.OAuth2GenericIdentityProvider]

11:20:21.812 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: inline-am-idp [io.gravitee.am.identityprovider.inline.InlineIdentityProvider]

11:20:21.915 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: salesforce-am-idp [io.gravitee.am.identityprovider.salesforce.SalesForceIdentityProvider]

11:20:21.927 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: ldap-am-idp [io.gravitee.am.identityprovider.ldap.LdapIdentityProvider]

11:20:21.938 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: gravitee-am-idp [io.gravitee.am.identityprovider.gravitee.GraviteeIdentityProvider]

11:20:21.959 [graviteeio-node] [] INFO  i.g.a.p.i.p.IdentityProviderPluginHandler - Register a new identity provider plugin: jdbc-am-idp [io.gravitee.am.identityprovider.jdbc.JdbcIdentityProvider]

11:20:21.963 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing policy plugins...

11:20:21.965 [graviteeio-node] [] INFO  i.g.p.p.internal.PolicyPluginHandler - Install plugin: policy-request-validation [io.gravitee.policy.requestvalidation.RequestValidationPolicy]

11:20:22.158 [graviteeio-node] [] INFO  i.g.p.p.internal.PolicyPluginHandler - Install plugin: groovy [io.gravitee.policy.groovy.GroovyPolicy]

11:20:23.116 [graviteeio-node] [] INFO  i.g.p.p.internal.PolicyPluginHandler - Install plugin: policy-am-enrich-auth-flow [io.gravitee.am.policy.enrich.auth.EnrichAuthFlowPolicy]

11:20:23.121 [graviteeio-node] [] INFO  i.g.p.p.internal.PolicyPluginHandler - Install plugin: ip-filtering [io.gravitee.policy.ipfiltering.IPFilteringPolicy]

11:20:23.151 [graviteeio-node] [] INFO  i.g.p.p.internal.PolicyPluginHandler - Install plugin: policy-am-enrich-profile [io.gravitee.am.policy.enrich.profile.EnrichProfilePolicy]

11:20:23.155 [graviteeio-node] [] INFO  i.g.p.p.internal.PolicyPluginHandler - Install plugin: policy-http-callout [io.gravitee.policy.callout.CalloutHttpPolicy]

11:20:23.181 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing bot_detection plugins...

11:20:23.202 [graviteeio-node] [] INFO  i.g.a.p.b.p.BotDetectionPluginHandler - Register a new bot detection plugin: google-recaptcha-v3-am-bot-detection [io.gravitee.am.botdetection.google.recaptcha.GoogleReCaptchaV3]

11:20:23.209 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing certificate plugins...

11:20:23.214 [graviteeio-node] [] INFO  i.g.a.p.c.p.CertificatePluginHandler - Register a new certificate plugin: javakeystore-am-certificate [io.gravitee.am.certificate.javakeystore.JavaKeyStore]

11:20:23.218 [graviteeio-node] [] INFO  i.g.a.p.c.p.CertificatePluginHandler - Register a new certificate plugin: pkcs12-am-certificate [io.gravitee.am.certificate.pkcs12.PKCS12Certificate]

11:20:23.219 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing notifier plugins...

11:20:23.220 [graviteeio-node] [] INFO  i.g.p.n.i.NotifierPluginHandler - Install plugin: webhook-notifier [io.gravitee.notifier.webhook.WebhookNotifier]

11:20:23.336 [graviteeio-node] [] INFO  i.g.p.n.i.NotifierPluginHandler - Install plugin: email-notifier [io.gravitee.notifier.email.EmailNotifier]

11:20:23.424 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginEventListener - Installing extension_grant plugins...

11:20:23.427 [graviteeio-node] [] INFO  i.g.a.p.e.p.ExtensionGrantPluginHandler - Register a new extension grant plugin: jwtbearer-am-extension-grant [io.gravitee.am.extensiongrant.jwtbearer.JWTBearerExtensionGrant]

11:20:23.441 [graviteeio-node] [] INFO  i.g.p.c.internal.PluginRegistryImpl - Plugins have been loaded and installed.

11:20:23.442 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: ServiceManager

11:20:23.442 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Plugins - Services Manager

11:20:23.442 [graviteeio-node] [] WARN  i.g.a.m.s.p.ScheduledPurgeService - Purge service has been disabled

11:20:23.442 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Sync Service

11:20:23.447 [graviteeio-node] [] INFO  i.g.a.m.s.sync.ScheduledSyncService - Sync service has been initialized with cron [*/5 * * * * *]

11:20:23.558 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: ManagementService

11:20:23.562 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Node Management API service

11:20:23.636 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: NodeMonitoringEventHandler

11:20:23.647 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler

11:20:23.753 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: NodeInfosService

11:20:23.763 [graviteeio-node] [] INFO  i.g.n.m.infos.NodeInfosService - Starting node infos service

11:20:23.763 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Node Infos Service

11:20:23.807 [vert.x-eventloop-thread-1] [] INFO  i.g.n.m.h.v.v.ManagementVerticle - Start HTTP listener for Node Management API

11:20:24.204 [graviteeio-node] [] INFO  i.g.n.m.infos.NodeInfosService - Start node infos service: DONE

11:20:24.205 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: NodeHealthCheckService

11:20:24.233 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Node Health-check service

11:20:24.234 [graviteeio-node] [] INFO  i.g.n.m.healthcheck.ProbeManagerImpl - Loading instances for type io.gravitee.node.api.healthcheck.Probe

11:20:24.524 [graviteeio-node] [] INFO  i.g.n.m.h.v.e.ManagementEndpointManagerImpl - Register a new endpoint for Management API: GET /health [io.gravitee.node.monitoring.healthcheck.NodeHealthCheckManagementEndpoint]

11:20:25.886 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: NodeMonitorService

11:20:25.886 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Node Monitor Service

11:20:26.035 [graviteeio-node] [] INFO  i.g.n.m.monitor.NodeMonitorService - Node monitoring scheduled with fixed delay 5000 MILLISECONDS 

11:20:26.061 [graviteeio-node] [] INFO  i.g.n.m.h.v.e.ManagementEndpointManagerImpl - Register a new endpoint for Management API: GET /monitor [io.gravitee.node.monitoring.monitor.NodeMonitorManagementEndpoint]

11:20:26.061 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: ReporterManager

11:20:26.070 [graviteeio-node] [] INFO  i.g.common.service.AbstractService - Initializing service Reporter service

11:20:26.222 [graviteeio-node] [] INFO  i.g.a.m.s.node.ManagementNode -     Starting component: JettyHttpServer

11:20:26.852 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 3029 ms, time limit is 2000 ms

11:20:26.853 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2613 ms, time limit is 2000 ms

11:20:27.837 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 4029 ms, time limit is 2000 ms

11:20:27.837 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3613 ms, time limit is 2000 ms

11:20:28.908 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 5031 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at java.base@11.0.11/java.lang.ClassLoader.defineClass1(Native Method)

    at java.base@11.0.11/java.lang.ClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.security.SecureClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.security.AccessController.doPrivileged(Native Method)

    at java.base@11.0.11/java.net.URLClassLoader.findClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at io.netty.channel.AbstractChannel.<init>(AbstractChannel.java:52)

    at io.netty.channel.nio.AbstractNioChannel.<init>(AbstractNioChannel.java:80)

    at io.netty.channel.nio.AbstractNioMessageChannel.<init>(AbstractNioMessageChannel.java:42)

    at io.netty.channel.socket.nio.NioDatagramChannel.<init>(NioDatagramChannel.java:151)

    at io.netty.channel.socket.nio.NioDatagramChannel.<init>(NioDatagramChannel.java:119)

    at io.vertx.core.net.impl.transport.Transport.datagramChannel(Transport.java:160)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1.lambda$newResolver$0(DnsResolverProvider.java:122)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1$$Lambda$436/0x0000000100659040.newChannel(Unknown Source)

    at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:310)

    at io.netty.bootstrap.AbstractBootstrap.register(AbstractBootstrap.java:227)

    at io.netty.resolver.dns.DnsNameResolver.<init>(DnsNameResolver.java:497)

    at io.netty.resolver.dns.DnsNameResolverBuilder.build(DnsNameResolverBuilder.java:488)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1$1.newNameResolver(DnsResolverProvider.java:181)

    at io.netty.resolver.dns.DnsAddressResolverGroup.newResolver(DnsAddressResolverGroup.java:92)

    at io.netty.resolver.dns.DnsAddressResolverGroup.newResolver(DnsAddressResolverGroup.java:77)

    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:70)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1.newResolver(DnsResolverProvider.java:185)

    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:70)

    at io.vertx.core.impl.AddressResolver.resolveHostname(AddressResolver.java:90)

    at io.vertx.core.impl.AddressResolver.resolveHostname(AddressResolver.java:83)

    at io.vertx.core.impl.VertxImpl.resolveAddress(VertxImpl.java:837)

    at io.vertx.core.net.impl.AsyncResolveConnectHelper.doBind(AsyncResolveConnectHelper.java:73)

    at io.vertx.core.net.impl.TCPServerBase.listen(TCPServerBase.java:130)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:213)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:143)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:129)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:134)

    at io.gravitee.node.management.http.vertx.verticle.ManagementVerticle.doStart(ManagementVerticle.java:148)

    at io.gravitee.node.management.http.vertx.verticle.ManagementVerticle.start(ManagementVerticle.java:91)

    at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:196)

    at io.vertx.core.impl.DeploymentManager$$Lambda$401/0x0000000100625840.handle(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:59)

    at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:37)

    at io.vertx.core.impl.EventLoopContext$$Lambda$407/0x0000000100627840.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

11:20:28.909 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4615 ms, time limit is 2000 ms

11:20:29.313 [graviteeio-node] [] INFO  org.eclipse.jetty.server.Server - jetty-9.4.36.v20210114; built: 2021-01-14T16:44:28.689Z; git: 238ec6997c7806b055319a6d11f8ae7564adc0de; jvm 11.0.11+9

11:20:29.810 [graviteeio-node] [] INFO  o.e.jetty.server.RequestLogWriter - Opened /opt/graviteeio-am-management-api/logs/gravitee_accesslog_2021_10_04.log

11:20:29.902 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 6031 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at java.base@11.0.11/java.util.ArrayDeque.poll(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile$CleanableResource.getInflater(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile$ZipFileInflaterInputStream.<init>(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile.getInputStream(Unknown Source)

    at java.base@11.0.11/java.util.jar.JarFile.getInputStream(Unknown Source)

    at java.base@11.0.11/jdk.internal.loader.URLClassPath$JarLoader$2.getInputStream(Unknown Source)

    at java.base@11.0.11/jdk.internal.loader.Resource.cachedInputStream(Unknown Source)

    at java.base@11.0.11/jdk.internal.loader.Resource.getByteBuffer(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.security.AccessController.doPrivileged(Native Method)

    at java.base@11.0.11/java.net.URLClassLoader.findClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at io.netty.buffer.ByteBufUtil.<clinit>(ByteBufUtil.java:1615)

    at io.netty.buffer.ByteBufAllocator.<clinit>(ByteBufAllocator.java:24)

    at io.netty.channel.DefaultChannelConfig.<init>(DefaultChannelConfig.java:60)

    at io.netty.channel.socket.DefaultDatagramChannelConfig.<init>(DefaultDatagramChannelConfig.java:55)

    at io.netty.channel.socket.nio.NioDatagramChannelConfig.<init>(NioDatagramChannelConfig.java:117)

    at io.netty.channel.socket.nio.NioDatagramChannel.<init>(NioDatagramChannel.java:152)

    at io.netty.channel.socket.nio.NioDatagramChannel.<init>(NioDatagramChannel.java:119)

    at io.vertx.core.net.impl.transport.Transport.datagramChannel(Transport.java:160)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1.lambda$newResolver$0(DnsResolverProvider.java:122)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1$$Lambda$436/0x0000000100659040.newChannel(Unknown Source)

    at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:310)

    at io.netty.bootstrap.AbstractBootstrap.register(AbstractBootstrap.java:227)

    at io.netty.resolver.dns.DnsNameResolver.<init>(DnsNameResolver.java:497)

    at io.netty.resolver.dns.DnsNameResolverBuilder.build(DnsNameResolverBuilder.java:488)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1$1.newNameResolver(DnsResolverProvider.java:181)

    at io.netty.resolver.dns.DnsAddressResolverGroup.newResolver(DnsAddressResolverGroup.java:92)

    at io.netty.resolver.dns.DnsAddressResolverGroup.newResolver(DnsAddressResolverGroup.java:77)

    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:70)

    at io.vertx.core.impl.resolver.DnsResolverProvider$1.newResolver(DnsResolverProvider.java:185)

    at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:70)

    at io.vertx.core.impl.AddressResolver.resolveHostname(AddressResolver.java:90)

    at io.vertx.core.impl.AddressResolver.resolveHostname(AddressResolver.java:83)

    at io.vertx.core.impl.VertxImpl.resolveAddress(VertxImpl.java:837)

    at io.vertx.core.net.impl.AsyncResolveConnectHelper.doBind(AsyncResolveConnectHelper.java:73)

    at io.vertx.core.net.impl.TCPServerBase.listen(TCPServerBase.java:130)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:213)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:143)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:129)

    at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:134)

    at io.gravitee.node.management.http.vertx.verticle.ManagementVerticle.doStart(ManagementVerticle.java:148)

    at io.gravitee.node.management.http.vertx.verticle.ManagementVerticle.start(ManagementVerticle.java:91)

    at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$5(DeploymentManager.java:196)

    at io.vertx.core.impl.DeploymentManager$$Lambda$401/0x0000000100625840.handle(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:96)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:59)

    at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:37)

    at io.vertx.core.impl.EventLoopContext$$Lambda$407/0x0000000100627840.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

11:20:30.419 [graviteeio-node] [] INFO  o.e.j.s.h.ContextHandler.management - Initializing Spring root WebApplicationContext

11:20:30.923 [vert.x-eventloop-thread-1] [] INFO  i.g.n.m.h.v.e.ManagementEndpointManagerImpl - Register a new endpoint for Management API: GET / [io.gravitee.node.management.http.node.NodeEndpoint]

11:20:30.923 [vert.x-eventloop-thread-1] [] INFO  i.g.n.m.h.v.e.ManagementEndpointManagerImpl - Register a new endpoint for Management API: GET /configuration [io.gravitee.node.management.http.configuration.ConfigurationEndpoint]

11:20:30.924 [vert.x-eventloop-thread-1] [] INFO  i.g.n.m.h.v.v.ManagementVerticle - HTTP listener for Node Management bind to port TCP:18093

11:20:31.514 [vert.x-eventloop-thread-0] [] INFO  i.g.n.r.vertx.ReporterManagerImpl -   There is no reporter to start

11:20:33.409 [graviteeio-node] [] INFO  i.g.a.m.h.m.a.a.m.i.i.IdentityProviderManagerImpl - Register event listener for identity provider events for all organizations

11:20:33.411 [graviteeio-node] [] INFO  i.g.c.event.impl.EventManagerImpl - Register new listener IdentityProviderManagerImpl for event type class io.gravitee.am.common.event.IdentityProviderEvent

11:20:33.411 [graviteeio-node] [] INFO  i.g.a.m.h.m.a.a.m.i.i.IdentityProviderManagerImpl - Initializing identity providers for all organizations

11:20:33.499 [graviteeio-node] [] INFO  i.g.a.m.h.m.a.a.m.i.i.IdentityProviderManagerImpl - Identity providers loaded for all organizations

11:20:34.048 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - 

11:20:34.055 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - ##############################################################

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - #                      SECURITY WARNING                      #

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - ##############################################################

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - 

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - You still use the default jwt secret.

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - This known secret can be used to impersonate anyone.

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - Please change this value, or ask your administrator to do it !

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - 

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - ##############################################################

11:20:34.058 [graviteeio-node] [] WARN  i.g.a.m.h.m.a.a.p.g.JWTGenerator - 

11:20:41.731 [graviteeio-node] [] INFO  o.s.s.web.DefaultSecurityFilterChain - Creating filter chain: Ant [pattern='/swagger.json'], []

11:20:42.314 [graviteeio-node] [] INFO  o.s.s.web.DefaultSecurityFilterChain - Creating filter chain: OrRequestMatcher [requestMatchers=[Ant [pattern='/auth/authorize'], Ant [pattern='/auth/login'], Ant [pattern='/auth/cockpit'], Ant [pattern='/auth/login/callback'], Ant [pattern='/auth/logout'], Ant [pattern='/auth/assets/**']]], [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@1a67bde5, org.springframework.security.web.context.SecurityContextPersistenceFilter@498f1f63, org.springframework.security.web.header.HeaderWriterFilter@4d2f8ee7, org.springframework.web.filter.CorsFilter@5c648e38, org.springframework.security.web.csrf.CsrfFilter@30b29f55, io.gravitee.am.management.handlers.management.api.authentication.filter.CsrfIncludeFilter@6bc8d8bd, org.springframework.security.web.authentication.logout.LogoutFilter@2fd409c1, io.gravitee.am.management.handlers.management.api.authentication.filter.CockpitAuthenticationFilter@395c21ee, io.gravitee.am.management.handlers.management.api.authentication.filter.RecaptchaFilter@1e733bb5, io.gravitee.am.management.handlers.management.api.authentication.filter.CheckRedirectionCookieFilter@669e730d, io.gravitee.am.management.handlers.management.api.authentication.filter.BuiltInAuthenticationFilter@4b3e4253, io.gravitee.am.management.handlers.management.api.authentication.filter.SocialAuthenticationFilter@13b5598b, io.gravitee.am.management.handlers.management.api.authentication.filter.CheckAuthenticationCookieFilter@141c66db, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@5980e29f, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@2a7b4bb6, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@1c76b2fe, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@18f794e, org.springframework.security.web.session.SessionManagementFilter@3a1e5fa7, org.springframework.security.web.access.ExceptionTranslationFilter@79735611, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@3df54fd5]

11:20:42.465 [graviteeio-node] [] INFO  o.s.s.web.DefaultSecurityFilterChain - Creating filter chain: Ant [pattern='/auth/token'], [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@30259c61, org.springframework.security.web.context.SecurityContextPersistenceFilter@5aec151b, org.springframework.security.web.header.HeaderWriterFilter@57f6fc09, org.springframework.security.web.authentication.logout.LogoutFilter@7e577eed, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@5dc8448b, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@74cf2d5f, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@14b9817b, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@33d08a24, org.springframework.security.web.session.SessionManagementFilter@ae5d544, org.springframework.security.web.access.ExceptionTranslationFilter@4e3c0627, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@5bfeb9ba]

11:20:42.472 [graviteeio-node] [] INFO  o.s.s.web.DefaultSecurityFilterChain - Creating filter chain: OrRequestMatcher [requestMatchers=[Ant [pattern='/organizations/**'], Ant [pattern='/user/**'], Ant [pattern='/platform/**']]], [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@5af7e2f, org.springframework.security.web.context.SecurityContextPersistenceFilter@175c26ff, org.springframework.security.web.header.HeaderWriterFilter@1196537d, org.springframework.web.filter.CorsFilter@2df24f19, org.springframework.security.web.csrf.CsrfFilter@11c40a65, io.gravitee.am.management.handlers.management.api.authentication.filter.CsrfIncludeFilter@c76762, org.springframework.security.web.authentication.logout.LogoutFilter@45bbb3e8, io.gravitee.am.management.handlers.management.api.authentication.filter.JWTAuthenticationFilter@5318731f, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@623bdc46, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@411576c7, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@2db82155, org.springframework.security.web.session.SessionManagementFilter@267a8777, org.springframework.security.web.access.ExceptionTranslationFilter@68c3aa38, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@80b70fd]

11:20:43.049 [graviteeio-node] [] INFO  i.g.a.m.h.m.a.a.m.f.i.FormManagerImpl - Register event listener for form events

11:20:43.066 [graviteeio-node] [] INFO  i.g.c.event.impl.EventManagerImpl - Register new listener FormManagerImpl for event type class io.gravitee.am.common.event.FormEvent

11:20:43.072 [graviteeio-node] [] INFO  i.g.a.m.h.m.a.a.m.f.i.FormManagerImpl - Initializing forms

11:20:43.432 [graviteeio-node] [] INFO  o.e.j.s.h.ContextHandler.management - Initializing Spring DispatcherServlet 'org.springframework.web.servlet.DispatcherServlet-362c48c'
andrearossi90 commented 3 years ago

Attaching the complete log gateway.log

leleueri commented 3 years ago

Problem seems to come from the DNS resolution. Could you please try to disable the vertx DNSRresolver using the following JVM option : -Dvertx.disableDnsResolver=true

gateway:
  env:
    - name: JAVA_OPTS
      value: -Dvertx.disableDnsResolver=true 
api:
  env:
    - name: JAVA_OPTS
      value: -Dvertx.disableDnsResolver=true
andrearossi90 commented 3 years ago

Hi, I'm using this command and the attached file helm upgrade --values am/values.yaml graviteeio-am graviteeio/am

values.yaml.txt

Edited for better clarity @leleueri

andrearossi90 commented 3 years ago

Edited for better clarity @leleueri , here the logs gateway3.log management3.log

In the last one can't catch the exception, it resets before I can download the log. Howerber we can see that it takes JAVA_OPTS correctly. Still not working.

=========================================================================

  Gravitee.io - Access Management - Management API
  GRAVITEE_HOME: /opt/graviteeio-am-management-api
  GRAVITEE_OPTS: 
  JAVA: /opt/java/openjdk/bin/java
  JAVA_OPTS: -Dvertx.disableDnsResolver=true -Djava.net.preferIPv4Stack=true -Xms256m -Xmx256m -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8
  CLASSPATH: /opt/graviteeio-am-management-api/lib/gravitee-am-management-api-standalone-bootstrap-3.11.0.jar

=========================================================================
=========================================================================

  Gravitee.io - Access Management - Gateway
  GRAVITEE_HOME: /opt/graviteeio-am-gateway
  GRAVITEE_OPTS: 
  JAVA: /opt/java/openjdk/bin/java
  JAVA_OPTS: -Dvertx.disableDnsResolver=true -Djava.net.preferIPv4Stack=true -Xms256m -Xmx256m -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8
  CLASSPATH: /opt/graviteeio-am-gateway/lib/gravitee-am-gateway-standalone-bootstrap-3.11.0.jar

=========================================================================

Gateway:

16:43:38.739 [cluster-ClusterId{value='615b2f31523437448262fe4e', description='null'}-graviteeio-am-mongodb-replicaset-2.graviteeio-am-mongodb-replicaset.default.svc.cluster.local:27017] [] INFO  org.mongodb.driver.cluster - Exception in monitor thread while connecting to server graviteeio-am-mongodb-replicaset-2.graviteeio-am-mongodb-replicaset.default.svc.cluster.local:27017
com.mongodb.MongoSocketWriteException: Exception sending message
    at com.mongodb.internal.connection.InternalStreamConnection.translateWriteException(InternalStreamConnection.java:659)
    at com.mongodb.internal.connection.InternalStreamConnection.sendMessage(InternalStreamConnection.java:536)
    at com.mongodb.internal.connection.InternalStreamConnection.sendCommandMessage(InternalStreamConnection.java:367)
    at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:317)
    at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:88)
    at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:36)
    at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:129)
    at com.mongodb.internal.connection.InternalStreamConnectionInitializer.startHandshake(InternalStreamConnectionInitializer.java:71)
    at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:167)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:195)
    at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:151)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.channels.ClosedChannelException: null
    at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.handshake(TlsChannelImpl.java:524)
    at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.write(TlsChannelImpl.java:378)
    at com.mongodb.internal.connection.tlschannel.ClientTlsChannel.write(ClientTlsChannel.java:184)
    at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.writeHandlingTasks(AsynchronousTlsChannelGroup.java:540)
    at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doWrite(AsynchronousTlsChannelGroup.java:498)
    at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processWrite$4(AsynchronousTlsChannelGroup.java:459)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    ... 1 common frames omitted
andrearossi90 commented 3 years ago

@leleueri Using these JAVA_OPTS=-Dvertx.disableDnsResolver=true -Djdk.tls.client.protocols=TLSv1.2 and connection uri mongodb+srv://graviteeio-am-mongodb-replicaset.default.svc.cluster.local/?**tls=false**&ssl=false&serverSelectionTimeoutMS=5000&connectTimeoutMS=5000&socketTimeoutMS=5000&maxIdleTimeMS=30000 there are no more DNS o connection issues. The Thread blocked came back however.

Gateway:

17:42:17.310 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5722 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at java.base@11.0.11/java.lang.ClassLoader.defineClass1(Native Method)

    at java.base@11.0.11/java.lang.ClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.security.SecureClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.security.AccessController.doPrivileged(Native Method)

    at java.base@11.0.11/java.net.URLClassLoader.findClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at io.gravitee.am.repository.mongodb.management.MongoNodeMonitoringRepository.findByNodeIdAndType(MongoNodeMonitoringRepository.java:58)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/java.lang.reflect.Method.invoke(Unknown Source)

    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)

    at com.sun.proxy.$Proxy20.findByNodeIdAndType(Unknown Source)

    at io.gravitee.node.monitoring.NodeMonitoringService.createOrUpdate(NodeMonitoringService.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfos(NodeMonitoringEventHandler.java:84)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfosMessage(NodeMonitoringEventHandler.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler$$Lambda$287/0x00000001004e8c40.handle(Unknown Source)

    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:49)

    at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:167)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.dispatch(MessageConsumerImpl.java:177)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.next(HandlerRegistration.java:163)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.dispatch(HandlerRegistration.java:128)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$303/0x00000001004ffc40.run(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:107)

    at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:104)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.deliver(MessageConsumerImpl.java:183)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.doReceive(MessageConsumerImpl.java:168)

    at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:54)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$302/0x00000001004f9c40.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

17:42:18.486 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 6745 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at io.gravitee.am.repository.mongodb.management.MongoNodeMonitoringRepository.findByNodeIdAndType(MongoNodeMonitoringRepository.java:58)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/java.lang.reflect.Method.invoke(Unknown Source)

    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)

    at com.sun.proxy.$Proxy20.findByNodeIdAndType(Unknown Source)

    at io.gravitee.node.monitoring.NodeMonitoringService.createOrUpdate(NodeMonitoringService.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfos(NodeMonitoringEventHandler.java:84)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfosMessage(NodeMonitoringEventHandler.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler$$Lambda$287/0x00000001004e8c40.handle(Unknown Source)

    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:49)

    at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:167)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.dispatch(MessageConsumerImpl.java:177)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.next(HandlerRegistration.java:163)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.dispatch(HandlerRegistration.java:128)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$303/0x00000001004ffc40.run(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:107)

    at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:104)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.deliver(MessageConsumerImpl.java:183)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.doReceive(MessageConsumerImpl.java:168)

    at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:54)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$302/0x00000001004f9c40.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

17:42:19.599 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 7753 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at java.base@11.0.11/java.io.RandomAccessFile.readBytes(Native Method)

    at java.base@11.0.11/java.io.RandomAccessFile.read(Unknown Source)

    at java.base@11.0.11/java.io.RandomAccessFile.readFully(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile$Source.readFullyAt(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile$ZipFileInputStream.initDataOffset(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile$ZipFileInputStream.read(Unknown Source)

    at java.base@11.0.11/java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(Unknown Source)

    at java.base@11.0.11/java.util.zip.InflaterInputStream.read(Unknown Source)

    at java.base@11.0.11/jdk.internal.loader.Resource.getBytes(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader.defineClass(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.net.URLClassLoader$1.run(Unknown Source)

    at java.base@11.0.11/java.security.AccessController.doPrivileged(Native Method)

    at java.base@11.0.11/java.net.URLClassLoader.findClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at io.gravitee.am.repository.mongodb.management.MongoNodeMonitoringRepository.findByNodeIdAndType(MongoNodeMonitoringRepository.java:58)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/java.lang.reflect.Method.invoke(Unknown Source)

    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)

    at com.sun.proxy.$Proxy20.findByNodeIdAndType(Unknown Source)

    at io.gravitee.node.monitoring.NodeMonitoringService.createOrUpdate(NodeMonitoringService.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfos(NodeMonitoringEventHandler.java:84)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfosMessage(NodeMonitoringEventHandler.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler$$Lambda$287/0x00000001004e8c40.handle(Unknown Source)

    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:49)

    at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:167)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.dispatch(MessageConsumerImpl.java:177)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.next(HandlerRegistration.java:163)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.dispatch(HandlerRegistration.java:128)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$303/0x00000001004ffc40.run(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:107)

    at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:104)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.deliver(MessageConsumerImpl.java:183)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.doReceive(MessageConsumerImpl.java:168)

    at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:54)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$302/0x00000001004f9c40.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

17:42:20.353 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 8768 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at java.base@11.0.11/java.lang.Throwable.fillInStackTrace(Native Method)

    at java.base@11.0.11/java.lang.Throwable.fillInStackTrace(Unknown Source)

    at java.base@11.0.11/java.lang.Throwable.<init>(Unknown Source)

    at java.base@11.0.11/java.lang.Exception.<init>(Unknown Source)

    at java.base@11.0.11/java.lang.ReflectiveOperationException.<init>(Unknown Source)

    at java.base@11.0.11/java.lang.ClassNotFoundException.<init>(Unknown Source)

    at java.base@11.0.11/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at java.base@11.0.11/java.lang.ClassLoader.loadClass(Unknown Source)

    at io.reactivex.Observable.elementAt(Observable.java:8341)

    at io.reactivex.Observable.firstElement(Observable.java:8440)

    at io.gravitee.am.repository.mongodb.management.MongoNodeMonitoringRepository.findByNodeIdAndType(MongoNodeMonitoringRepository.java:59)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.base@11.0.11/java.lang.reflect.Method.invoke(Unknown Source)

    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)

    at com.sun.proxy.$Proxy20.findByNodeIdAndType(Unknown Source)

    at io.gravitee.node.monitoring.NodeMonitoringService.createOrUpdate(NodeMonitoringService.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfos(NodeMonitoringEventHandler.java:84)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfosMessage(NodeMonitoringEventHandler.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler$$Lambda$287/0x00000001004e8c40.handle(Unknown Source)

    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:49)

    at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:167)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.dispatch(MessageConsumerImpl.java:177)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.next(HandlerRegistration.java:163)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.dispatch(HandlerRegistration.java:128)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$303/0x00000001004ffc40.run(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:107)

    at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:104)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.deliver(MessageConsumerImpl.java:183)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.doReceive(MessageConsumerImpl.java:168)

    at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:54)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$302/0x00000001004f9c40.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

Management

17:46:05.963 [vertx-blocked-thread-checker] [] WARN  i.v.core.impl.BlockedThreadChecker - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5767 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

    at java.base@11.0.11/jdk.internal.misc.Unsafe.unpark(Native Method)

    at java.base@11.0.11/java.util.concurrent.locks.LockSupport.unpark(Unknown Source)

    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(Unknown Source)

    at java.base@11.0.11/java.util.concurrent.locks.AbstractQueuedSynchronizer.release(Unknown Source)

    at java.base@11.0.11/java.util.concurrent.locks.ReentrantLock.unlock(Unknown Source)

    at java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(Unknown Source)

    at java.base@11.0.11/java.util.concurrent.LinkedBlockingQueue.offer(Unknown Source)

    at java.base@11.0.11/java.util.AbstractQueue.add(Unknown Source)

    at com.mongodb.internal.connection.DefaultConnectionPool$AsyncWorkManager.enqueue(DefaultConnectionPool.java:1178)

    at com.mongodb.internal.connection.DefaultConnectionPool.getAsync(DefaultConnectionPool.java:179)

    at com.mongodb.internal.connection.DefaultServer.getConnectionAsync(DefaultServer.java:105)

    at com.mongodb.internal.binding.AsyncClusterBinding$AsyncClusterBindingConnectionSource.getConnection(AsyncClusterBinding.java:153)

    at com.mongodb.internal.async.client.ClientSessionBinding$SessionBindingAsyncConnectionSource.getConnection(ClientSessionBinding.java:179)

    at com.mongodb.internal.operation.OperationHelper.withAsyncConnectionSource(OperationHelper.java:731)

    at com.mongodb.internal.operation.OperationHelper.access$200(OperationHelper.java:69)

    at com.mongodb.internal.operation.OperationHelper$AsyncCallableWithConnectionAndSourceCallback.onResult(OperationHelper.java:751)

    at com.mongodb.internal.operation.OperationHelper$AsyncCallableWithConnectionAndSourceCallback.onResult(OperationHelper.java:739)

    at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)

    at com.mongodb.internal.async.client.ClientSessionBinding$WrappingCallback.onResult(ClientSessionBinding.java:267)

    at com.mongodb.internal.async.client.ClientSessionBinding$WrappingCallback.onResult(ClientSessionBinding.java:255)

    at com.mongodb.internal.binding.AsyncClusterBinding$1.onResult(AsyncClusterBinding.java:119)

    at com.mongodb.internal.binding.AsyncClusterBinding$1.onResult(AsyncClusterBinding.java:113)

    at com.mongodb.internal.connection.BaseCluster$ServerSelectionRequest.onResult(BaseCluster.java:430)

    at com.mongodb.internal.connection.BaseCluster.handleServerSelectionRequest(BaseCluster.java:294)

    at com.mongodb.internal.connection.BaseCluster.selectServerAsync(BaseCluster.java:155)

    at com.mongodb.internal.connection.AbstractMultiServerCluster.selectServerAsync(AbstractMultiServerCluster.java:50)

    at com.mongodb.internal.binding.AsyncClusterBinding.getAsyncClusterBindingConnectionSource(AsyncClusterBinding.java:113)

    at com.mongodb.internal.binding.AsyncClusterBinding.getReadConnectionSource(AsyncClusterBinding.java:98)

    at com.mongodb.internal.async.client.ClientSessionBinding.getReadConnectionSource(ClientSessionBinding.java:62)

    at com.mongodb.internal.operation.OperationHelper.withAsyncReadConnection(OperationHelper.java:678)

    at com.mongodb.internal.operation.FindOperation.executeAsync(FindOperation.java:692)

    at com.mongodb.reactivestreams.client.internal.OperationExecutorImpl.lambda$execute$3(OperationExecutorImpl.java:74)

    at com.mongodb.reactivestreams.client.internal.OperationExecutorImpl$$Lambda$468/0x00000001006bac40.accept(Unknown Source)

    at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)

    at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)

    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)

    at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:67)

    at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)

    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1812)

    at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:61)

    at reactor.core.publisher.Mono.subscribe(Mono.java:4252)

    at io.reactivex.internal.operators.observable.ObservableFromPublisher.subscribeActual(ObservableFromPublisher.java:31)

    at io.reactivex.Observable.subscribe(Observable.java:12284)

    at io.reactivex.internal.operators.observable.ObservableElementAtMaybe.subscribeActual(ObservableElementAtMaybe.java:32)

    at io.reactivex.Maybe.subscribe(Maybe.java:4290)

    at io.reactivex.internal.operators.maybe.MaybeMap.subscribeActual(MaybeMap.java:40)

    at io.reactivex.Maybe.subscribe(Maybe.java:4290)

    at io.reactivex.internal.operators.maybe.MaybeFlatten.subscribeActual(MaybeFlatten.java:42)

    at io.reactivex.Maybe.subscribe(Maybe.java:4290)

    at io.reactivex.internal.operators.maybe.MaybeSwitchIfEmptySingle.subscribeActual(MaybeSwitchIfEmptySingle.java:45)

    at io.reactivex.Single.subscribe(Single.java:3666)

    at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)

    at io.reactivex.Single.subscribe(Single.java:3666)

    at io.reactivex.internal.operators.single.SingleDoFinally.subscribeActual(SingleDoFinally.java:44)

    at io.reactivex.Single.subscribe(Single.java:3666)

    at io.reactivex.Single.subscribe(Single.java:3652)

    at io.reactivex.Single.subscribe(Single.java:3565)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfos(NodeMonitoringEventHandler.java:84)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler.handleNodeInfosMessage(NodeMonitoringEventHandler.java:69)

    at io.gravitee.node.monitoring.handler.NodeMonitoringEventHandler$$Lambda$399/0x000000010061cc40.handle(Unknown Source)

    at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:49)

    at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:167)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.dispatch(MessageConsumerImpl.java:177)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.next(HandlerRegistration.java:163)

    at io.vertx.core.eventbus.impl.HandlerRegistration$InboundDeliveryContext.dispatch(HandlerRegistration.java:128)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$411/0x000000010062d040.run(Unknown Source)

    at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:107)

    at io.vertx.core.eventbus.impl.HandlerRegistration.dispatch(HandlerRegistration.java:104)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.deliver(MessageConsumerImpl.java:183)

    at io.vertx.core.eventbus.impl.MessageConsumerImpl.doReceive(MessageConsumerImpl.java:168)

    at io.vertx.core.eventbus.impl.HandlerRegistration.lambda$receive$0(HandlerRegistration.java:54)

    at io.vertx.core.eventbus.impl.HandlerRegistration$$Lambda$408/0x000000010062c440.run(Unknown Source)

    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

    at java.base@11.0.11/java.lang.Thread.run(Unknown Source)

I'm attaching the full logs, what can I try?

Thanks in advance

gateway4.log management4.log

andrearossi90 commented 3 years ago

I switched to a clean environment. At the moment it works with:

Thank you for your support