UI is stuck on the "Loading..." screen

jpaxtonrs commented 1 year ago

Describe the bug When I open the browser and navigate to ReportPortal's UI, it is stuck in the Loading... screen.

Expected behavior It should direct the user to the login screen prompting for user credentials.

Additional context I am deploying ReportPortal into AWS Fargate using CloudFormation. Here is a breakdown of how I am deploying the infrastructure:

namespace
- This CloudFormation Stack is responsible for deploying out Namespaces for Service Discovery so that the Fargate containers can communicate with each other and store their hostnames in the Container's Environment Variable prior to deployment. For example, the api Container needs to store RP_AMQP_HOST variable, which means it needs to know the rabbitmq host before deployment.
efs
- This CloudFormation Stack is responsible for deploying out Elastic Filesystems so that the Fargate containers can share their file systems and can mount volumes for persistent data. Both elasticsearch and minio are leveraging this.
postgres
- This CloudFormation Stack is responsible for deploying out a RDS instance running Postgres.
cluster
- This CloudFormation Stack is responsible for deploying out an ECS Cluster that will hold and group all Fargate services and tasks that make up ReportPortal.
migrations
- This CloudFormation Stack is responsible for deploying out a Fargate task that pulls the db-scripts image. Also, this Stack has a Lambda function that will invoke/run the db-scripts task only once to populate the Postgres DB instance.
alb
- This CloudFormation Stack is responsible for deploying out an ALB that has multiple target groups for the UI and API services (api, jobs, and uat) with the appropriate path-pattern rules and health-checks. For example, the api listener rule is looking for the path-pattern of /api and will forward traffic to the api target group which is on port 8585 and runs a health-check on the/health endpoint (all the other services are very similar with the exception of the UI as this is the default forwarding action). The ALB allows us to communicate with ReportPortal internally through the browser for the UI and through integration (i.e. Playwright) for API calls.
app
- This CloudFormation Stack is responsible for deploying out all the Fargate services and tasks that all make up ReportPortal. Most of the Fargate services and tasks are configured very similarly to the docker-compose.

Currently, all the services and tasks are up and running and healthy; however, the UI is stuck at the Loading... screen, which tells me there is something that is either incorrect with the application and/or the infrastructure setup.

⚠️ I decided to not use traefik and index services as I am using an ALB and Namespace for service discovery and container communication.

Screenshots

Loading... screen issue:
Fargate services all healthy:

Logs I removed some of the noise surrounding the warnings/errors found in the logs. Here is a list of warnings/errors. I know that some of these logs are not related; however, it might be better to put as much information here as possible:

Analyzer
- WARNING: you are running uWSGI as root !!! (use the --uid flag)
- /venv/lib/python3.7/site-packages/sklearn/linear_model/least_angle.py:1442: DeprecationWarning: np.float is a deprecated alias for the builtin float. To silence this warning, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
API
- Unable to create an RSA verifier from verifierKey (ignoreable if using MAC)
Elasticsearch
- JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-12671927430012896513, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]
- max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Metrics
- WARNING: you are running uWSGI as root !!! (use the --uid flag)
RabbitMQ
- [error] <0.1900.0> closing AMQP connection <0.1900.0>
- [warning] <0.709.0> Message store "CCCKRGUCI5BC4D7Q237IU7AY5/msg_store_persistent": rebuilding indices from scratch
UAT
- 2023-07-12 20:56:52.370 ERROR 1 --- [ main] c.e.t.r.f.distributed.s3.S3DataStore : Unable to find file 'keystore/secret-integration-salt'
- Unable to create an RSA verifier from verifierKey (ignoreable if using MAC)

Versions:

Linux/X86_64, Chrome and Firefox (Loading screen issue shows for both browsers)
Not able to get to the login screen. However, here are all the image versions I am using (all 5.7.4, except metrics which is 5.7.3):
- reportportal/service-auto-analyzer:5.7.4
- reportportal/service-auto-analyzer:5.7.4
- reportportal/service-api:5.7.4
- reportportal/service-jobs:5.7.4
- reportportal/service-metrics-gatherer:5.7.3
- minio/minio:RELEASE.2020-10-27T04-03-55Z
- rabbitmq:3.7.16-management
- reportportal/service-authorization:5.7.4
- reportportal/service-ui:5.7.4

Please, let me know if I need to provide anything else. Any feedback would be greatly appreciated.

Thanks!

jpaxtonrs commented 1 year ago

Following up on this issue. Thanks!

DzmitryHumianiuk commented 1 year ago

@jpaxtonrs looks like web app (JS bundle in UI service) can't see the API service for initial calls to represent the login screen.

KenKingson commented 1 year ago

Hey, We are facing the same issue, usually this state comes after the night, when the developer tries to open the report portal in the morning, he gets this endless loading. The only workaround we found was to restart the containers.

DzmitryHumianiuk commented 1 year ago

@KenKingson could you please examine the logs in the service-api container prior to restart? This should shed some light on the issue leading up to the service being unavailable.

MikitaKarbanovich commented 1 year ago

Hi everyone. @DzmitryHumianiuk I was faced with the same issue. Restarts help to fix it, but only for a few hours. I use the latest RP version. Before updating to 23.1 everything was good, but after update we have such behavior.

Endless logs in the service-api :

2023-08-05 04:08:07   ... 31 common frames omitted
2023-08-05 04:08:07 Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30001ms.
2023-08-05 04:08:07   at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
2023-08-05 04:08:07   at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
2023-08-05 04:08:07   at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
2023-08-05 04:08:07   at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
2023-08-05 04:08:07   at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
2023-08-05 04:08:07   at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
2023-08-05 04:08:07   at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
2023-08-05 04:08:07   ... 35 common frames omitted
2023-08-05 04:08:07
2023-08-05 04:09:00 2023-08-05 01:09:00.684 ERROR 1 --- [_MisfireHandler] o.s.s.quartz.LocalDataSourceJobStore   : MisfireHandler: Error handling misfires: Failed to obtain DB connection from data source ‘springNonTxDataSource.reportportal’: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-08-05 04:09:00
2023-08-05 04:09:00 org.quartz.JobPersistenceException: Failed to obtain DB connection from data source ‘springNonTxDataSource.reportportal’: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-08-05 04:09:00   at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
2023-08-05 04:09:00   at org.quartz.impl.jdbcjobstore.JobStoreSupport.doRecoverMisfires(JobStoreSupport.java:3246)
2023-08-05 04:09:00   at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.manage(JobStoreSupport.java:4012)
2023-08-05 04:09:00   at org.quartz.impl.jdbcjobstore.JobStoreSupport$MisfireHandler.run(JobStoreSupport.java:4033)
2023-08-05 04:09:00 Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-08-05 04:09:00   at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
2023-08-05 04:09:00   at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
2023-08-05 04:09:00   at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
2023-08-05 04:09:00   at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
2023-08-05 04:09:00   at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:140)
2023-08-05 04:09:00   at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
2023-08-05 04:09:00   at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
2023-08-05 04:09:00   ... 3 common frames omitted
2023-08-05 04:09:00
2023-08-05 04:09:35 2023-08-05 01:09:35.305 WARN 1 --- [ult-task-sched3] o.h.engine.jdbc.spi.SqlExceptionHelper  : SQL Error: 0, SQLState: null
2023-08-05 04:09:35 2023-08-05 01:09:35.305 ERROR 1 --- [ult-task-sched3] o.h.engine.jdbc.spi.SqlExceptionHelper  : HikariPool-1 - Connection is not available, request timed out after 30001ms.
2023-08-05 04:09:35 2023-08-05 01:09:35.305 ERROR 1 --- [ult-task-sched3] o.s.s.s.TaskUtils$LoggingErrorHandler  : Unexpected error occurred in scheduled task
2023-08-05 04:09:35
2023-08-05 04:09:35 org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
2023-08-05 04:09:35   at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467)
2023-08-05 04:09:35   at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManag

irynakozak2 commented 1 year ago

@jpaxtonrs @MikitaKarbanovich Hello everyone, could you please share the total amount of PostgreSQL connections by executing the following query while connecting to the DB: show max_connections;

Please also share the running queries/active sessions from the database. It can be checked like that: SELECT * FROM pg_stat_activity;

Please also take a look at this documentation regarding API connection pool tuning and try to follow the instructions and increase the connection pool for the API service, it could be a quick solution for that issue.

Waiting for your reply.

jpaxtonrs commented 1 year ago

@jpaxtonrs @MikitaKarbanovich Hello everyone, could you please share the total amount of PostgreSQL connections by executing the following query while connecting to the DB: show max_connections;

Please also share the running queries/active sessions from the database. It can be checked like that: SELECT * FROM pg_stat_activity;

Please also take a look at this documentation regarding API connection pool tuning and try to follow the instructions and increase the connection pool for the API service, it could be a quick solution for that issue.

Waiting for your reply.

Screenshot 2023-08-08 at 10 26 41 AM

DzmitryHumianiuk commented 1 year ago

@jpaxtonrs and SELECT * FROM pg_stat_activity; ??

yevheniilavrenchuk commented 1 year ago

Faced this as well today, after upgrading to 5.8.0. UI just hangs on Loading... API container restart helps though,

service-api container throws:

2023-08-09 11:26:24.519 ERROR 1 --- [_ClusterManager] o.s.s.quartz.LocalDataSourceJobStore     : ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'springNonTxDataSource.reportportal': java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.reportportal': java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3322)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3935)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3972)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:140)
        at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
        ... 3 common frames omitted

On the other hand, postgres is alive and the only logs its writing are:

PostgreSQL Database directory appears to contain a database; Skipping initialization

2023-08-09 06:30:48.845 UTC [1] LOG:  starting PostgreSQL 12.6 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
2023-08-09 06:30:48.845 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-08-09 06:30:48.845 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-08-09 06:30:48.879 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-08-09 06:30:49.254 UTC [22] LOG:  database system was shut down at 2023-08-09 06:30:41 UTC
2023-08-09 06:30:49.321 UTC [1] LOG:  database system is ready to accept connections

yevheniilavrenchuk commented 1 year ago

For some reason service-api stops communication with rebitmq as well:

2023-08-09 11:44:17.385 ERROR 1 --- [-listener-exec4] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.epam.ta.reportportal.core.events.listener.LaunchFinishedEventListener.onApplicationEvent(com.epam.ta.reportportal.core.events.activity.LaunchFinishedEvent)

com.epam.ta.reportportal.exception.ReportPortalException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://rabbitmq@172.18.0.8:5672/analyzer,5) (#5) timed out
        at com.epam.ta.reportportal.core.analyzer.auto.impl.LogIndexerService.indexLaunchLogs(LogIndexerService.java:109)
        at com.epam.ta.reportportal.core.analyzer.auto.impl.LogIndexerService$$FastClassBySpringCGLIB$$f53c9b3b.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
        at com.epam.ta.reportportal.core.analyzer.auto.impl.LogIndexerService$$EnhancerBySpringCGLIB$$c843e7cf.indexLaunchLogs(<generated>)
        at com.epam.ta.reportportal.core.analyzer.auto.starter.decorator.IndexingAutoAnalysisStarter.start(IndexingAutoAnalysisStarter.java:46)
        at com.epam.ta.reportportal.core.analyzer.auto.starter.decorator.IndexingAutoAnalysisStarter$$FastClassBySpringCGLIB$$6c5f0b93.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
        at com.epam.ta.reportportal.core.analyzer.auto.starter.decorator.IndexingAutoAnalysisStarter$$EnhancerBySpringCGLIB$$7c282fd7.start(<generated>)
        at com.epam.ta.reportportal.core.analyzer.auto.starter.decorator.ExistingAnalyzerStarter.start(ExistingAnalyzerStarter.java:46)
        at com.epam.ta.reportportal.core.events.handler.launch.LaunchAutoAnalysisRunner.handle(LaunchAutoAnalysisRunner.java:51)
        at com.epam.ta.reportportal.core.events.handler.launch.LaunchAutoAnalysisRunner.handle(LaunchAutoAnalysisRunner.java:34)
        at com.epam.ta.reportportal.core.events.subscriber.impl.delegate.ProjectConfigDelegatingSubscriber.lambda$handleEvent$0(ProjectConfigDelegatingSubscriber.java:44)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at com.epam.ta.reportportal.core.events.subscriber.impl.delegate.ProjectConfigDelegatingSubscriber.handleEvent(ProjectConfigDelegatingSubscriber.java:44)
        at com.epam.ta.reportportal.core.events.subscriber.impl.delegate.ProjectConfigDelegatingSubscriber.handleEvent(ProjectConfigDelegatingSubscriber.java:30)
        at com.epam.ta.reportportal.core.events.listener.LaunchFinishedEventListener.lambda$onApplicationEvent$0(LaunchFinishedEventListener.java:45)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at com.epam.ta.reportportal.core.events.listener.LaunchFinishedEventListener.onApplicationEvent(LaunchFinishedEventListener.java:45)
        at com.epam.ta.reportportal.core.events.listener.LaunchFinishedEventListener$$FastClassBySpringCGLIB$$81f8adcc.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

MikitaKarbanovich commented 1 year ago

Hello @irynakozak2. Unfortunately cannot provide this information anymore as I fully reinstalled RP as such behaviour was a blocker for me and I didn't have time to investigate it fully. But the good news - after full reinstallation RP works well.

irynakozak2 commented 1 year ago

@MikitaKarbanovich Thank you for the information and updates. We will check with the Back End team as we discovered this behavior was reproducible occasionally in the local environment (Docker installation). In our Kubernetes environments, we haven't seen that anywhere.

irynakozak2 commented 1 year ago

@jpaxtonrs Thank you. I assume the issue can be with the default connection pool initialization or the lack of the default connections count (27). However, it's difficult to prove without understanding what sessions/queries in the database are running and acquiring the connections. Please follow the instructions in the documentation mentioned in the comment above. I assume the issue will be resolved after increasing the default connection pool for the API.

jpaxtonrs commented 1 year ago

@irynakozak2 / @DzmitryHumianiuk I am currently redeploying the entire CloudFormation stack. Once, all the Resources are up and running, I'll run the query. Stay tuned! Thanks.

DzmitryHumianiuk commented 1 year ago

@irynakozak2 it seems like raising the default values in the default setup we distribute could be a sensible decision. Are there any potential downsides or drawbacks to this change that we should consider?

luckhiem commented 1 year ago

I still facing the same issue "UI still loading" and Couldn't start RabbitMQ in background.


2023-08-10 10:20:56 rabbitmq 03:20:56.68 Welcome to the Bitnami rabbitmq container
2023-08-10 10:20:56 rabbitmq 03:20:56.71 Subscribe to project updates by watching https://github.com/bitnami/containers
2023-08-10 10:20:56 rabbitmq 03:20:56.76 Submit issues and feature requests at https://github.com/bitnami/containers/issues
2023-08-10 10:20:56 rabbitmq 03:20:56.80 
2023-08-10 10:20:56 rabbitmq 03:20:56.83 INFO  ==> ** Starting RabbitMQ setup **
2023-08-10 10:20:57 rabbitmq 03:20:57.10 INFO  ==> Validating settings in RABBITMQ_* env vars..
2023-08-10 10:20:57 rabbitmq 03:20:57.60 INFO  ==> Initializing RabbitMQ...
2023-08-10 10:20:57 rabbitmq 03:20:57.94 INFO  ==> Generating random cookie
2023-08-10 10:20:58 rabbitmq 03:20:58.52 INFO  ==> Starting RabbitMQ in background...
2023-08-10 10:20:59 /opt/bitnami/scripts/libos.sh: line 344:   148 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:10 /opt/bitnami/scripts/libos.sh: line 344:   246 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:21 /opt/bitnami/scripts/libos.sh: line 344:   294 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:31 /opt/bitnami/scripts/libos.sh: line 344:   391 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:42 /opt/bitnami/scripts/libos.sh: line 344:   439 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:53 /opt/bitnami/scripts/libos.sh: line 344:   487 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:03 /opt/bitnami/scripts/libos.sh: line 344:   584 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:14 /opt/bitnami/scripts/libos.sh: line 344:   632 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:25 /opt/bitnami/scripts/libos.sh: line 344:   680 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:35 /opt/bitnami/scripts/libos.sh: line 344:   777 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:45 rabbitmq 03:22:45.71 ERROR ==> Couldn't start RabbitMQ in background.

DzmitryHumianiuk commented 1 year ago

@luckhiem did you raise the default connections parameter?

luckhiem commented 1 year ago

@DzmitryHumianiuk How can we raise the default connections parameter? Please help

yevheniilavrenchuk commented 1 year ago

Guys, faced the same thing today.

Suddenly service-api starting to write these logs and never restarts itself, but marked as unhealthy. Rabbitmq on the other hand is healthy.

2023-08-11 06:29:35.183  WARN 1 --- [io-8585-exec-35] o.s.b.a.amqp.RabbitHealthIndicator       : Rabbit health check failed

org.springframework.amqp.AmqpIOException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://rabbitmq@172.18.0.5:5672/analyzer,192) (#192) timed out

After service-api restart, it successfully connects to rebbitmq:

2023-08-11 06:31:03.571  INFO 1 --- [           main] o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: rabbitmq:5672
2023-08-11 06:31:03.699  INFO 1 --- [           main] o.s.a.r.c.CachingConnectionFactory       : Created new connection: connectionFactory#6b35b0f4:0/SimpleConnection@5292d417 [delegate=amqp://rabbitmq@172.18.0.5:5672/, localPort= 53186]
2023-08-11 06:31:03.966  INFO 1 --- [           main] o.s.amqp.rabbit.core.RabbitAdmin         : Auto-declaring a non-durable or auto-delete Exchange (broadcast.events) durable:false, auto-delete:false. It will be deleted by the broker if it shuts down, and can be redeclared by closing and reopening the connection.
2023-08-11 06:31:03.967  INFO 1 --- [           main] o.s.amqp.rabbit.core.RabbitAdmin         : Auto-declaring a non-durable, auto-delete, or exclusive Queue (broadcast.events.xcGrhtJWRjSJR8wUJllSKg) durable:false, auto-delete:true, exclusive:true. It will be redeclared if the broker stops and is restarted while the connection factory is alive, but all messages will be lost.
2023-08-11 06:31:04.049  INFO 1 --- [           main] c.e.t.r.c.c.r.ReportingConfiguration     : Consumer is created, current consumers count is 1
2023-08-11 06:31:04.050  INFO 1 --- [           main] c.e.t.r.c.c.r.ReportingConfiguration     : Consumer is created, current consumers count is 2

irynakozak2 commented 1 year ago

@DzmitryHumianiuk Not much, but it's necessary to investigate the issue on our side to determine the exact root cause. We need to decide why sometimes the default pool isn't enough to start the API service and profile the connection utilization in case of any potential fix we need to do on our side. As far as we didn't face such behavior in our small instances in Kubernetes and observed it on Docker local setup, maybe the API service requires more CPU or RAM resources to start in some cases.

irynakozak2 commented 1 year ago

@luckhiem Hi, could you please specify the CPU and RAM resources for the API service? And regarding raising the default connection pool, please check this link. Basically, you should add the environment variable RP_DATASOURCE_MAXIMUMPOOLSIZE=100 for the service-API and check that the PostgreSQL database has more max_connections.

I still facing the same issue "UI still loading" and Couldn't start RabbitMQ in background.

2023-08-10 10:20:56 rabbitmq 03:20:56.68 Welcome to the Bitnami rabbitmq container
2023-08-10 10:20:56 rabbitmq 03:20:56.71 Subscribe to project updates by watching https://github.com/bitnami/containers
2023-08-10 10:20:56 rabbitmq 03:20:56.76 Submit issues and feature requests at https://github.com/bitnami/containers/issues
2023-08-10 10:20:56 rabbitmq 03:20:56.80 
2023-08-10 10:20:56 rabbitmq 03:20:56.83 INFO  ==> ** Starting RabbitMQ setup **
2023-08-10 10:20:57 rabbitmq 03:20:57.10 INFO  ==> Validating settings in RABBITMQ_* env vars..
2023-08-10 10:20:57 rabbitmq 03:20:57.60 INFO  ==> Initializing RabbitMQ...
2023-08-10 10:20:57 rabbitmq 03:20:57.94 INFO  ==> Generating random cookie
2023-08-10 10:20:58 rabbitmq 03:20:58.52 INFO  ==> Starting RabbitMQ in background...
2023-08-10 10:20:59 /opt/bitnami/scripts/libos.sh: line 344:   148 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:10 /opt/bitnami/scripts/libos.sh: line 344:   246 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:21 /opt/bitnami/scripts/libos.sh: line 344:   294 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:31 /opt/bitnami/scripts/libos.sh: line 344:   391 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:42 /opt/bitnami/scripts/libos.sh: line 344:   439 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:21:53 /opt/bitnami/scripts/libos.sh: line 344:   487 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:03 /opt/bitnami/scripts/libos.sh: line 344:   584 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:14 /opt/bitnami/scripts/libos.sh: line 344:   632 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:25 /opt/bitnami/scripts/libos.sh: line 344:   680 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:35 /opt/bitnami/scripts/libos.sh: line 344:   777 Segmentation fault      "$@" > /dev/null 2>&1
2023-08-10 10:22:45 rabbitmq 03:22:45.71 ERROR ==> Couldn't start RabbitMQ in background.

jpaxtonrs commented 1 year ago

@jpaxtonrs and SELECT * FROM pg_stat_activity; ??

datid	datname	pid	usesysid	usename	application_name	client_addr	client_port	backend_start	xact_start	query_start	state_change	wait_event_type	wait_event	state	backend_xmin	query	backend_type
		395	10	rdsadmin				2023-06-30 02:33:41.568806+00				Activity	LogicalLauncherMain				logical replication launcher
		392						2023-06-30 02:33:41.569456+00				Activity	AutoVacuumMain				autovacuum launcher
16384	rdsadmin	10801	10	rdsadmin	PostgreSQL JDBC Driver		-1	2023-07-29 05:34:34.956147+00		2023-08-11 14:55:31.644025+00	2023-08-11 14:55:31.644054+00	Client	ClientRead	idle		SELECT value FROM rds_heartbeat2	client backend
16396	reportportal	30938	16394	reportportal	TablePlus	XXX.XX.XX.XXX	52937	2023-08-11 14:54:43.729941+00	2023-08-11 14:55:39.2058+00	2023-08-11 14:55:39.2058+00	2023-08-11 14:55:39.205804+00			active	73252	SELECT * FROM pg_stat_activity;	client backend
		390						2023-06-30 02:33:41.567964+00				Activity	BgWriterHibernate				background writer
		389						2023-06-30 02:33:41.56744+00				Activity	CheckpointerMain				checkpointer
		391						2023-06-30 02:33:41.570068+00				Activity	WalWriterMain				walwriter

irynakozak2 commented 1 year ago

Seems like there is a normal state of the PG activities and no issues or lots of idle connections from the ReportPortal. Did you manage to fix the issue after the redeploying CloudFormation stack?

@jpaxtonrs and SELECT * FROM pg_stat_activity; ??

datid datname pid usesysid usename application_name client_addr client_hostname client_port backend_start xact_start query_start state_change wait_event_type wait_event state backend_xid backend_xmin query backend_type 395 10 rdsadmin 2023-06-30 02:33:41.568806+00 Activity LogicalLauncherMain logical replication launcher 392 2023-06-30 02:33:41.569456+00 Activity AutoVacuumMain autovacuum launcher 16384 rdsadmin 10801 10 rdsadmin PostgreSQL JDBC Driver -1 2023-07-29 05:34:34.956147+00 2023-08-11 14:55:31.644025+00 2023-08-11 14:55:31.644054+00 Client ClientRead idle SELECT value FROM rds_heartbeat2 client backend 16396 reportportal 30938 16394 reportportal TablePlus XXX.XX.XX.XXX 52937 2023-08-11 14:54:43.729941+00 2023-08-11 14:55:39.2058+00 2023-08-11 14:55:39.2058+00 2023-08-11 14:55:39.205804+00 active 73252 SELECT * FROM pg_stat_activity; client backend 390 2023-06-30 02:33:41.567964+00 Activity BgWriterHibernate background writer 389 2023-06-30 02:33:41.56744+00 Activity CheckpointerMain checkpointer 391 2023-06-30 02:33:41.570068+00 Activity WalWriterMain walwriter

irynakozak2 commented 1 year ago

@KenKingson @MikitaKarbanovich @luckhiem @jpaxtonrs Please check the Network tab in your browser on the requests while opening the login page. I'm currently debugging the issue and didn't manage to reproduce it locally with docker-compose (with connections), but I faced almost the same behavior on other running instance.

KenKingson commented 1 year ago

@irynakozak2 here is the screenshot

traj279 commented 1 year ago

I Have been facing this same issue while trying to setup with Kubernetes + Docker Desktop + Minikube . But when I tried with docker compose as mentioned by @irynakozak2, initially I got the same loading error, but I tried to increase the CPUs to 16 and RAM to 64 GB (initial config 8 CPU and 64 GB RAM) to I was able to deploy successfully. (https://github.com/reportportal/reportportal/blob/master/docker-compose.yml).

However, the Kubernetes docker Desktop still didn't work after increasing the system resources.

DzmitryHumianiuk commented 1 year ago

@traj279 good to know that it works. but it seems to be a very beefy setup for the tool :/

jpaxtonrs commented 1 year ago

Seems like there is a normal state of the PG activities and no issues or lots of idle connections from the ReportPortal. Did you manage to fix the issue after the redeploying CloudFormation stack?

@jpaxtonrs and SELECT * FROM pg_stat_activity; ??

datid datname pid usesysid usename application_name client_addr client_hostname client_port backend_start xact_start query_start state_change wait_event_type wait_event state backend_xid backend_xmin query backend_type 395 10 rdsadmin 2023-06-30 02:33:41.568806+00 Activity LogicalLauncherMain logical replication launcher 392 2023-06-30 02:33:41.569456+00 Activity AutoVacuumMain autovacuum launcher 16384 rdsadmin 10801 10 rdsadmin PostgreSQL JDBC Driver -1 2023-07-29 05:34:34.956147+00 2023-08-11 14:55:31.644025+00 2023-08-11 14:55:31.644054+00 Client ClientRead idle SELECT value FROM rds_heartbeat2 client backend 16396 reportportal 30938 16394 reportportal TablePlus XXX.XX.XX.XXX 52937 2023-08-11 14:54:43.729941+00 2023-08-11 14:55:39.2058+00 2023-08-11 14:55:39.2058+00 2023-08-11 14:55:39.205804+00 active 73252 SELECT * FROM pg_stat_activity; client backend 390 2023-06-30 02:33:41.567964+00 Activity BgWriterHibernate background writer 389 2023-06-30 02:33:41.56744+00 Activity CheckpointerMain checkpointer 391 2023-06-30 02:33:41.570068+00 Activity WalWriterMain walwriter

No, I ran into the same issue. I'm still thinking that there is some sort of networking issue. It's just not showing in the logs. Is there anyway that we can add additional debug log lines whenever this problem occurs?

venom0107 commented 1 year ago

@jpaxtonrs @irynakozak2 I'm also facing same issue. Have tried restarting service-api multiple times. After restarting the service-api container, everything works well. But as soon as we start sending data from our automation tests, after some 300-400 cases execution, the service-api starts responding with 405's. And then the report portal UI gets stuck on Loading... page.

The EC2 instance we are using is t3a.xlarge [ t3a.xlarge | CPU - 4 | RAM - 16.0 ] Here's my max connection:

 max_connections
-----------------
 821
(1 row)

And here's my response for SELECT * FROM pg_stat_activity;


 datid | datname  |  pid  | leader_pid | usesysid |  usename   |    application_name    | client_addr | client_hostname | client_port |         backend_start   |          xact_start           |          query_start          |         state_change          | wait_event_type |     wait_event      |        state        |backend_xid | backend_xmin |    query                                                                                                                                                    |         backend_type
-------+----------+-------+------------+----------+------------+------------------------+-------------+-----------------+-------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+---------------------+---------------------+-------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------
       |          |   397 |            |       10 | rdsadmin   |                        |             |                 |             | 2023-10-04 11:57:19.620348+00 |                               |                               |                               | Activity        | LogicalLauncherMain |                     |            |              |                                                                                                                                                    | logical replication launcher
       |          |   394 |            |          |            |                        |             |                 |             | 2023-10-04 11:57:19.624208+00 |                               |                               |                               | Activity        | AutoVacuumMain      |                     |            |              |                                                                                                                                                    | autovacuum launcher
 16398 | reportdb | 17375 |            |    16396 | reportuser | PostgreSQL JDBC Driver | x.x.x.x  |                 |       51730 | 2023-10-04 16:11:38.532594+00 |                               | 2023-10-04 16:11:38.543139+00 | 2023-10-04 16:11:38.543152+00 | Client          | ClientRead          | idle                |            |              | SET application_name = 'PostgreSQL JDBC Driver'                                                                                                                                                    | client backend
 16398 | reportdb | 17353 |            |    16396 | reportuser | PostgreSQL JDBC Driver | x.x.x.x  |                 |       54864 | 2023-10-04 16:11:14.934408+00 |                               | 2023-10-04 16:11:14.944382+00 | 2023-10-04 16:11:14.944395+00 | Client          | ClientRead          | idle                |            |              | SET application_name = 'PostgreSQL JDBC Driver'                                                                                                                                                    | client backend
 16398 | reportdb | 17287 |            |    16396 | reportuser | PostgreSQL JDBC Driver | x.x.x.x  |                 |       54944 | 2023-10-04 16:10:07.68402+00  |                               | 2023-10-04 16:35:10.382199+00 | 2023-10-04 16:35:10.382203+00 | Client          | ClientRead          | idle                |            |              | ```

venom0107 commented 1 year ago

Here are the error logs from our reportportal-api-1 (reportportal/service-api:5.9.2) for DB Connection Issue:

023-10-04 15:11:53.006 ERROR 1 --- [_ClusterManager] o.s.s.quartz.LocalDataSourceJobStore     : ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'springNonTxDataSource.reportportal': java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.reportportal': java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3322)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3935)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3972)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:140)
        at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
        ... 3 common frames omitted

2023-10-04 15:11:58.233  WARN 1 --- [ult-task-sched3] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: null
2023-10-04 15:11:58.233 ERROR 1 --- [ult-task-sched3] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-10-04 15:11:58.233 ERROR 1 --- [ult-task-sched3] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy270.findAll(Unknown Source)
        at com.epam.ta.reportportal.job.CleanOutdatedPluginsJob.execute(CleanOutdatedPluginsJob.java:77)
        at jdk.internal.reflect.GeneratedMethodAccessor244.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
        at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:480)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:152)
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421)
        ... 26 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
        at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
        ... 30 common frames omitted

2023-10-04 15:11:58.234  WARN 1 --- [ult-task-sched1] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: null
2023-10-04 15:11:58.234 ERROR 1 --- [ult-task-sched1] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-10-04 15:11:58.234 ERROR 1 --- [ult-task-sched1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy270.findAll(Unknown Source)
        at com.epam.ta.reportportal.job.service.impl.PluginLoaderServiceImpl.getNotLoadedPluginsInfo(PluginLoaderServiceImpl.java:70)
        at com.epam.ta.reportportal.job.service.impl.PluginLoaderServiceImpl$$FastClassBySpringCGLIB$$d76672c2.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
        at com.epam.ta.reportportal.job.service.impl.PluginLoaderServiceImpl$$EnhancerBySpringCGLIB$$d124f1ab.getNotLoadedPluginsInfo(<generated>)
        at com.epam.ta.reportportal.job.LoadPluginsJob.execute(LoadPluginsJob.java:76)
        at jdk.internal.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
        at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:480)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:152)
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421)
        ... 31 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
        at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
        ... 35 common frames omitted

2023-10-04 15:13:28.234  WARN 1 --- [ult-task-sched3] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: null
2023-10-04 15:13:28.234 ERROR 1 --- [ult-task-sched3] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-10-04 15:13:28.235 ERROR 1 --- [ult-task-sched3] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy270.findAll(Unknown Source)
        at com.epam.ta.reportportal.job.CleanOutdatedPluginsJob.execute(CleanOutdatedPluginsJob.java:77)
        at jdk.internal.reflect.GeneratedMethodAccessor244.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
        at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:480)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:152)
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421)
        ... 26 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
        at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
        ... 30 common frames omitted

2023-10-04 15:13:28.235  WARN 1 --- [ult-task-sched5] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: null
2023-10-04 15:13:28.235 ERROR 1 --- [ult-task-sched5] o.h.engine.jdbc.spi.SqlExceptionHelper   : HikariPool-1 - Connection is not available, request timed out after 30000ms.
2023-10-04 15:13:28.235 ERROR 1 --- [ult-task-sched5] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:467)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:595)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:382)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:174)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy270.findAll(Unknown Source)
        at com.epam.ta.reportportal.job.service.impl.PluginLoaderServiceImpl.getNotLoadedPluginsInfo(PluginLoaderServiceImpl.java:70)
        at com.epam.ta.reportportal.job.service.impl.PluginLoaderServiceImpl$$FastClassBySpringCGLIB$$d76672c2.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
        at com.epam.ta.reportportal.job.service.impl.PluginLoaderServiceImpl$$EnhancerBySpringCGLIB$$d124f1ab.getNotLoadedPluginsInfo(<generated>)
        at com.epam.ta.reportportal.job.LoadPluginsJob.execute(LoadPluginsJob.java:76)
        at jdk.internal.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
        at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:111)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
        at org.hibernate.internal.SessionImpl.connection(SessionImpl.java:480)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:152)
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:421)
        ... 31 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
        at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
        ... 35 common frames omitted

2023-10-04 15:14:53.040 ERROR 1 --- [_ClusterManager] o.s.s.quartz.LocalDataSourceJobStore     : ClusterManager: Error managing cluster: Failed to obtain DB connection from data source 'springNonTxDataSource.reportportal': java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

org.quartz.JobPersistenceException: Failed to obtain DB connection from data source 'springNonTxDataSource.reportportal': java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:168)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport.doCheckin(JobStoreSupport.java:3322)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.manage(JobStoreSupport.java:3935)
        at org.quartz.impl.jdbcjobstore.JobStoreSupport$ClusterManager.run(JobStoreSupport.java:3972)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
        at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
        at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
        at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
        at org.springframework.scheduling.quartz.LocalDataSourceJobStore$2.getConnection(LocalDataSourceJobStore.java:140)
        at org.quartz.utils.DBConnectionManager.getConnection(DBConnectionManager.java:108)
        at org.quartz.impl.jdbcjobstore.JobStoreCMT.getNonManagedTXConnection(JobStoreCMT.java:165)
        ... 3 common frames omitted

venom0107 commented 1 year ago

Here's logs for rmq issue too. I observed both (rmq and db conn issue) on my setup.

Logs from reportportal-rabbitmq-1 (bitnami/rabbitmq:3.12.2-debian-11-r8):

2023-10-04 16:50:00.147821+00:00 [warning] <0.13847.0> closing AMQP connection <0.13847.0> (x.x.x.x:51556 -> x.x.x.x:5672, vhost: 'analyzer', user: 'rabbitmq'):
2023-10-04 16:50:00.147821+00:00 [warning] <0.13847.0> client unexpectedly closed TCP connection
2023-10-04 16:50:00.148391+00:00 [warning] <0.13887.0> closing AMQP connection <0.13887.0> (x.x.x.x:39338 -> x.x.x.x:5672, vhost: 'analyzer', user: 'rabbitmq'):
2023-10-04 16:50:00.148391+00:00 [warning] <0.13887.0> client unexpectedly closed TCP connection
2023-10-04 16:50:00.148584+00:00 [warning] <0.13926.0> closing AMQP connection <0.13926.0> (x.x.x.x:60198 -> x.x.x.x:5672, vhost: 'analyzer', user: 'rabbitmq'):
2023-10-04 16:50:00.148584+00:00 [warning] <0.13926.0> client unexpectedly closed TCP connection
2023-10-04 16:50:00.317770+00:00 [info] <0.13967.0> accepting AMQP connection <0.13967.0> (x.x.x.x:48270 -> x.x.x.x:5672)
2023-10-04 16:50:00.322514+00:00 [info] <0.13967.0> connection <0.13967.0> (x.x.x.x:48270 -> x.x.x.x:5672): user 'rabbitmq' authenticated and granted access to vhost 'analyzer'
2023-10-04 16:51:00.596138+00:00 [info] <0.14012.0> accepting AMQP connection <0.14012.0> (x.x.x.x:50544 -> x.x.x.x:5672)
2023-10-04 16:51:00.599461+00:00 [info] <0.14012.0> connection <0.14012.0> (x.x.x.x:50544 -> x.x.x.x:5672): user 'rabbitmq' authenticated and granted access to vhost 'analyzer'
2023-10-04 16:52:00.876648+00:00 [info] <0.14046.0> accepting AMQP connection <0.14046.0> (x.x.x.x:36950 -> x.x.x.x:5672)
2023-10-04 16:52:00.880531+00:00 [info] <0.14046.0> connection <0.14046.0> (x.x.x.x:36950 -> x.x.x.x:5672): user 'rabbitmq' authenticated and granted access to vhost 'analyzer'
2023-10-04 16:53:01.007498+00:00 [warning] <0.13967.0> closing AMQP connection <0.13967.0> (x.x.x.x:48270 -> x.x.x.x:5672, vhost: 'analyzer', user: 'rabbitmq'):
2023-10-04 16:53:01.007498+00:00 [warning] <0.13967.0> client unexpectedly closed TCP connection
2023-10-04 16:53:01.007519+00:00 [warning] <0.14012.0> closing AMQP connection <0.14012.0> (x.x.x.x:50544 -> x.x.x.x:5672, vhost: 'analyzer', user: 'rabbitmq'):
2023-10-04 16:53:01.007519+00:00 [warning] <0.14012.0> client unexpectedly closed TCP connection
2023-10-04 16:53:01.007801+00:00 [warning] <0.14046.0> closing AMQP connection <0.14046.0> (x.x.x.x:36950 -> x.x.x.x:5672, vhost: 'analyzer', user: 'rabbitmq'):
2023-10-04 16:53:01.007801+00:00 [warning] <0.14046.0> client unexpectedly closed TCP connection
2023-10-04 16:53:01.187140+00:00 [info] <0.14099.0> accepting AMQP connection <0.14099.0> (x.x.x.x:59224 -> x.x.x.x:5672)
2023-10-04 16:53:01.189736+00:00 [info] <0.14099.0> connection <0.14099.0> (x.x.x.x:59224 -> x.x.x.x:5672): user 'rabbitmq' authenticated and granted access to vhost 'analyzer'

Logs from reportportal-api-1 (reportportal/service-api:5.9.2):

023-10-04 16:48:34.527  WARN 1 --- [nio-8585-exec-3] o.s.b.a.amqp.RabbitHealthIndicator       : Rabbit health check failed

org.springframework.amqp.AmqpIOException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://rabbitmq@x.x.x.x:5672/analyzer,18) (#18) timed out
        at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70)
        at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:69)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:1408)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$200(CachingConnectionFactory.java:1394)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:696)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:679)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:649)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:539)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1600(CachingConnectionFactory.java:99)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1413)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2181)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2148)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2128)
        at org.springframework.boot.actuate.amqp.RabbitHealthIndicator.getVersion(RabbitHealthIndicator.java:49)
        at org.springframework.boot.actuate.amqp.RabbitHealthIndicator.doHealthCheck(RabbitHealthIndicator.java:44)
        at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
        at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:85)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:44)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
        at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:71)
        at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:60)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
        at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
        at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
        at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:308)
        at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:413)
        at jdk.internal.reflect.GeneratedMethodAccessor250.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:670)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:125)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:121)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:115)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:182)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://rabbitmq@x.x.x.x:5672/analyzer,18) (#18) timed out
        at com.rabbitmq.client.impl.AMQChannel.wrapTimeoutException(AMQChannel.java:313)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:295)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141)
        at com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:128)
        at com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:182)
        at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:618)
        at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:58)
        ... 129 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
        at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)
        at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293)
        ... 134 common frames omitted

2023-10-04 16:49:05.477  WARN 1 --- [ult-task-sched5] .l.DirectReplyToMessageListenerContainer : basicConsume failed, scheduling consumer for queue amq.rabbitmq.reply-to for restart

org.springframework.amqp.AmqpIOException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://rabbitmq@x.x.x.x:5672/analyzer,19) (#19) timed out
        at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70)
        at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:69)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:1408)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$200(CachingConnectionFactory.java:1394)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:696)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:679)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:649)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:539)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1600(CachingConnectionFactory.java:99)
        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1413)
        at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.consume(DirectMessageListenerContainer.java:760)
        at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:731)
        at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.adjustConsumers(DirectMessageListenerContainer.java:344)
        at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.setConsumersPerQueue(DirectMessageListenerContainer.java:171)
        at org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer.getChannelHolder(DirectReplyToMessageListenerContainer.java:175)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceiveWithDirect(RabbitTemplate.java:1959)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceive(RabbitTemplate.java:1850)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceiveRaw(RabbitTemplate.java:1818)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceiveAsType(RabbitTemplate.java:1788)
        at org.springframework.amqp.rabbit.core.RabbitOperations.convertSendAndReceiveAsType(RabbitOperations.java:371)
        at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceiveAsType(RabbitTemplate.java:1730)
        at com.epam.ta.reportportal.core.analyzer.auto.client.impl.IndexerServiceClientImpl.lambda$indexDefectsUpdate$2(IndexerServiceClientImpl.java:91)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
        at com.epam.ta.reportportal.core.analyzer.auto.client.impl.IndexerServiceClientImpl.indexDefectsUpdate(IndexerServiceClientImpl.java:97)
        at com.epam.ta.reportportal.core.analyzer.auto.impl.LogIndexerService.indexDefectsUpdate(LogIndexerService.java:153)
        at com.epam.ta.reportportal.core.analyzer.auto.impl.LogIndexerService$$FastClassBySpringCGLIB$$f53c9b3b.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://rabbitmq@x.x.x.x:5672/analyzer,19) (#19) timed out
        at com.rabbitmq.client.impl.AMQChannel.wrapTimeoutException(AMQChannel.java:313)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:295)
        at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141)
        at com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:128)
        at com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:182)
        at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:618)
        at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:58)
        ... 41 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
        at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)
        at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
        at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)
        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293)
        ... 46 common frames omitted

venom0107 commented 1 year ago

Also attaching result for docker ps command, for reference, just in case:

DzmitryHumianiuk commented 1 year ago

@venom0107 could you please try latest 23.2 release.

Just delete everything and start from scratch. Please also make sure that you have at least 6GB allocated for the docker machine.

DzmitryHumianiuk commented 1 year ago

If this still remains a concern or if there are new developments, please feel free to reopen or create a new issue.

Thank you for your understanding and contributions to the project!

pankajrawat1996 commented 1 month ago

@DzmitryHumianiuk @venom0107 I am facing same issue API service is intermittently going unhealthy then I have to restart the service. below are the logs

2024-09-29 06:36:54.093 ERROR 1 --- [io-8585-exec-27] c.e.r.r.c.e.rest.DefaultErrorResolver    : com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<queue.declare>(ticket=0, queue=amq.rabbitmq.reply-to, passive=true, durable=false, exclusive=true, auto-delete=true, nowait=false, arguments=null) on channel AMQChannel(amqp://rabbitmq@172.23.0.5:5672/analyzer,1) (#1) timed out

org.springframework.amqp.AmqpIOException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<queue.declare>(ticket=0, queue=amq.rabbitmq.reply-to, passive=true, durable=false, exclusive=true, auto-delete=true, nowait=false, arguments=null) on channel AMQChannel(amqp://rabbitmq@172.23.0.5:5672/analyzer,1) (#1) timed out
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70)
    at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2202)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2148)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:2128)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.useDirectReplyTo(RabbitTemplate.java:1018)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.evaluateFastReplyTo(RabbitTemplate.java:990)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doSendAndReceive(RabbitTemplate.java:1844)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceiveRaw(RabbitTemplate.java:1818)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceiveAsType(RabbitTemplate.java:1788)
    at org.springframework.amqp.rabbit.core.RabbitOperations.convertSendAndReceiveAsType(RabbitOperations.java:371)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.convertSendAndReceiveAsType(RabbitTemplate.java:1730)
    at com.epam.ta.reportportal.core.analyzer.auto.client.impl.AnalyzerServiceClientImpl.searchLogs(AnalyzerServiceClientImpl.java:98)
    at com.epam.ta.reportportal.core.analyzer.auto.impl.SearchLogServiceImpl.processRequest(SearchLogServiceImpl.java:156)
    at com.epam.ta.reportportal.core.analyzer.auto.impl.SearchLogServiceImpl.lambda$search$3(SearchLogServiceImpl.java:118)
    at java.base/java.util.Optional.map(Optional.java:260)
    at com.epam.ta.reportportal.core.analyzer.auto.impl.SearchLogServiceImpl.search(SearchLogServiceImpl.java:117)
    at com.epam.ta.reportportal.core.analyzer.auto.impl.SearchLogServiceImpl$$FastClassBySpringCGLIB$$fb44ce98.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
    at com.epam.ta.reportportal.core.analyzer.auto.impl.SearchLogServiceImpl$$EnhancerBySpringCGLIB$$6e2b3366.search(<generated>)
    at com.epam.ta.reportportal.ws.controller.LogController.searchLogs(LogController.java:309)
    at com.epam.ta.reportportal.ws.controller.LogController$$FastClassBySpringCGLIB$$f7673969.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
    at com.epam.ta.reportportal.ws.controller.LogController$$EnhancerBySpringCGLIB$$b6c7edfa.searchLogs(<generated>)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:903)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:555)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:125)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117)
    at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:182)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<queue.declare>(ticket=0, queue=amq.rabbitmq.reply-to, passive=true, durable=false, exclusive=true, auto-delete=true, nowait=false, arguments=null) on channel AMQChannel(amqp://rabbitmq@172.23.0.5:5672/analyzer,1) (#1) timed out
    at com.rabbitmq.client.impl.AMQChannel.wrapTimeoutException(AMQChannel.java:313)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:295)
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:141)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1012)
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:46)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1157)
    at jdk.proxy2/jdk.proxy2.$Proxy315.queueDeclarePassive(Unknown Source)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.lambda$useDirectReplyTo$2(RabbitTemplate.java:1019)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.invokeAction(RabbitTemplate.java:2237)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:2196)
    ... 133 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
    at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77)
    at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:502)
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:293)
    ... 143 common frames omitted

2024-09-29 06:36:54.094  WARN 1 --- [io-8585-exec-27] c.e.r.r.c.e.rest.RestExceptionHandler    : Resolved [org.springframework.amqp.AmqpIOException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<queue.declare>(ticket=0, queue=amq.rabbitmq.reply-to, passive=true, durable=false, exclusive=true, auto-delete=true, nowait=false, arguments=null) on channel AMQChannel(amqp://rabbitmq@172.23.0.5:5672/analyzer,1) (#1) timed out]

DzmitryHumianiuk commented 1 month ago

@pankajrawat1996 your rabbitMQ is unreachable. here is what GPT is saying on this:

The error logs you've provided indicate a timeout issue when your application is attempting to communicate with the RabbitMQ server. Here's a detailed analysis of the problem and potential solutions:

Problem Analysis

Error Message:

com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<queue.declare>(ticket=0, queue=amq.rabbitmq.reply-to, passive=true, durable=false, exclusive=true, auto-delete=true, nowait=false, arguments=null) on channel AMQChannel(amqp://rabbitmq@172.23.0.5:5672/analyzer,1) (#1) timed out

Key Points:

Timeout Exception: The ChannelContinuationTimeoutException suggests that the RabbitMQ client did not receive a response within the expected timeframe when trying to declare the amq.rabbitmq.reply-to queue.
Passive Queue Declaration: The passive=true parameter means the client is attempting to check if the queue exists without creating it. If the queue doesn't exist, it would typically result in a ChannelException, but here it's a timeout, indicating no response.
Connection Details: The client is trying to connect to RabbitMQ at 172.23.0.5:5672 using the analyzer virtual host.

Possible Causes

RabbitMQ Server Unreachable:
- The RabbitMQ server might be down or not reachable at the specified IP and port.
Network Issues:
- Network partitions, firewall rules, or routing issues could prevent the application from reaching the RabbitMQ server.
Resource Constraints on RabbitMQ:
- The server might be overloaded, lacking sufficient resources (CPU, memory), or hitting limits on channels or connections.
Misconfiguration:
- Incorrect RabbitMQ connection settings in the application (e.g., wrong hostname, port, virtual host, or credentials).
Firewall or Security Groups:
- Firewalls or security groups might be blocking traffic to RabbitMQ's port (5672).
DNS Issues:
- If using a hostname instead of an IP, DNS resolution problems could prevent successful connections.
Version Incompatibility:
- Incompatibility between the RabbitMQ client library version and the RabbitMQ server version.

Additional Recommendations

Implement Retry Logic:
- Incorporate retry mechanisms in your application to handle transient connectivity issues gracefully.
Use Heartbeats:
- Configure heartbeats between the client and RabbitMQ to detect and handle broken connections promptly.
Load Balancing and High Availability:
- If applicable, set up RabbitMQ in a clustered or high-availability configuration to mitigate single points of failure.
Consult RabbitMQ Documentation:
- Refer to the official RabbitMQ documentation for best practices and advanced configurations.

Conclusion

The ChannelContinuationTimeoutException typically points to connectivity or responsiveness issues between your application and the RabbitMQ server. By systematically verifying the server's status, network connectivity, application configurations, and resource utilization, you can identify and resolve the root cause of the timeout. Implementing monitoring and resilience strategies will also help prevent similar issues in the future.

pankajrawat1996 commented 1 month ago

@DzmitryHumianiuk Others are also facing the same issue ..check below link https://github.com/reportportal/reportportal/issues/2273#issuecomment-2384736877 @akshayamaldhure

DzmitryHumianiuk commented 1 month ago

@DzmitryHumianiuk Others are also facing the same issue ..check below link #2273 (comment) @akshayamaldhure

seems like not relevant for 24.2 check the thread #2273 (comment)

reportportal / reportportal