search

ibm-mas / ansible-devops

Ansible collection supporting devops for IBM Maximo Application Suite
https://ibm-mas.github.io/ansible-devops/
Eclipse Public License 2.0
49 stars 87 forks source link

store-api-deployment on Analyticsproxy pod crash on oneclick Installation. #1073

Closed aswindo23 closed 1 year ago

aswindo23 commented 1 year ago

Hello,

I have some issue when install using ansible oneclick_core.

First Error when i check on deployment store-api-deployment. connection to instrumentationdb-primary.ibm-common-services.svc:5432 refused. 

 /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.7.5)
2023-10-24 08:44:32.304 INFO 1 --- [ main] com.ibm.edge.StoreApiApplication : Starting StoreApiApplication v2.0.8 using Java 11.0.18 on store-api-deployment-794c7dc65c-x6tjh with PID 1 (/usr/local/lib/store-api.jar started by store-api in /)
2023-10-24 08:44:32.401 INFO 1 --- [ main] com.ibm.edge.StoreApiApplication : No active profile set, falling back to 1 default profile: "default"
2023-10-24 08:44:54.199 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2023-10-24 08:44:55.402 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1089 ms. Found 3 JPA repository interfaces.
2023-10-24 08:44:55.695 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2023-10-24 08:44:56.201 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 505 ms. Found 3 JPA repository interfaces.
2023-10-24 08:45:27.196 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8443 (https)
2023-10-24 08:45:27.501 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-10-24 08:45:27.502 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.68]
2023-10-24 08:45:29.594 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-10-24 08:45:29.594 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 55798 ms

2023-10-24 08:45:47.404 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2023-10-24 08:45:49.893 ERROR 1 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
org.postgresql.util.PSQLException: Connection to instrumentationdb-primary.ibm-common-services.svc:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:319) ~[postgresql-42.4.1.jar!/:42.4.1]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.4.1.jar!/:42.4.1]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:247) ~[postgresql-42.4.1.jar!/:42.4.1]
at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.4.1.jar!/:42.4.1]
at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.4.1.jar!/:42.4.1]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115) ~[HikariCP-4.0.3.jar!/:na]

after i checking on the instrumentationdb-primary.ibm-common-services.svc services, this services cannot attached any IP Address.

but when i check the Crunchy Postgres for Kubernetes Operator on Postgres Cluster, deployment of instrumentationdb-instance1-deployment have some error connection to instrumentationdb-ha 

2023-10-24 08:49:59,138 INFO: starting after demotion in progress
/tmp/postgres:5432 - rejecting connections
/tmp/postgres:5432 - rejecting connections
/tmp/postgres:5432 - rejecting connections
/tmp/postgres:5432 - rejecting connections
/tmp/postgres:5432 - rejecting connections
/tmp/postgres:5432 - rejecting connections
/tmp/postgres:5432 - accepting connections
2023-10-24 08:50:05,523 INFO: Lock owner: instrumentationdb-instance1-qqjd-0; I am instrumentationdb-instance1-qqjd-0
2023-10-24 08:50:05,523 INFO: establishing a new patroni connection to the postgres cluster
2023-10-24 08:50:05,590 ERROR: Permission denied
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 914, in _update_leader_with_retry
    return self._patch_or_create(self.leader_path, annotations, resource_version, ips=ips, retry=_retry)
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 868, in _patch_or_create
    ret = retry(func, self._namespace, body) if retry else func(self._namespace, body)
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 911, in _retry
    return retry(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/patroni/utils.py", line 334, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 468, in wrapper
    return getattr(self._core_v1_api, func)(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 404, in wrapper
    return self._api_client.call_api(method, path, headers, body, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 373, in call_api
    return self._handle_server_response(response, _preload_content)
  File "/usr/local/lib/python3.6/site-packages/patroni/dcs/kubernetes.py", line 203, in _handle_server_response
    raise k8s_client.rest.ApiException(http_resp=response)
patroni.dcs.kubernetes.K8sClient.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '4d88e4ff-89e2-4258-9603-f72474d0324f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'X-Kubernetes-Pf-Flowschema-Uid': '72d70c50-7e86-4e02-b7d7-ac0bbe0d43f4', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'd4a1531c-bd6a-4b1c-880a-af927cbbd54b', 'Date': 'Tue, 24 Oct 2023 08:50:05 GMT', 'Content-Length': '267'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"endpoints \\"instrumentationdb-ha\\" is forbidden: endpoint address 10.129.3.28 is not allowed","reason":"Forbidden","details":{"name":"instrumentationdb-ha","kind":"endpoints"},"code":403}\n'

2023-10-24 08:50:05,591 ERROR: failed to update leader lock
2023-10-24 08:50:05,713 INFO: not promoting because failed to update leader lock in DCS
2023-10-24 08:50:16,112 INFO: promoted self to leader by acquiring session lock
server promoting
2023-10-24 08:50:16,141 INFO: cleared rewind state after becoming the leader

This issue indicate, from UDS store-api-deployment connection to Crunchy Postgres for Kubernetes instrumentationdb

Thanks.

aswindo23 commented 1 year ago

I have resolved the issue. Thanks.