Closed FranziskusWild closed 5 months ago
onesaitplatform
commented
8 months ago Hi,
I suppose those logs are from the standard output, but for finding the problem, you should better check the ones you can obtain from the directory /var/log/platform-logs in the pod. If you can paste here those logs, it would be appreciated. Also, is the pod marked as ready (1/1) at anytime?
Best Regards, Onesait Platform Team
FranziskusWild
commented
8 months ago Thanks for answering so swiftly :)
Find the log attached: CC-log.txt
Update: (this time I catched the errors that actually lead to the shutdown, I think) CC-log.txt
FranziskusWild
commented
8 months ago ERROR com.hazelcast.instance.impl.Node - [10.244.2.100]:5701 [onesaitplatform] [5.1.3] Could not join cluster. Shutting down now!
Looks like Hazelcast is the culprit, right?
onesaitplatform
commented
8 months ago Hi,
it seems that it is a Hazelcast problem, the cacheservice and zookeeper modules are up and running?
Best Regards, Onesait Platform Team
FranziskusWild
commented
8 months ago Hi there,
Yep, the Helm Charts sets the controlcenter to you use zookeeper and it failed to connect. I set HZ_SERVICE_DISCOVERY_STRATEGY to service and that made it starts up correctly.
Zookeeper and cacheservice where running.
onesaitplatform-control-panel 07:46:07.510 [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/10.0.72.7:2181, session id = 0x1002a2a22010000, negotiated timeout = 60000 -> it seems to be connected to zookeeper, but unable 'to join the cluster' ...
FranziskusWild
commented
8 months ago So to sum up:
service configured it's workingzookeeper configured it runs into that Could not join cluster error. But it is able to talk to zookeeper: `Session establishment complete on server zookeeperDo you have an idea on what might be the problem? `
onesaitplatform
commented
8 months ago Hi,
it sounds quite strange, because we have deployed it in an Openshift cluster and in a Rancher managed cluster and we had no problems with the connection. Where are you deploying the helm charts? Have you changed anything in the values.yml?
Best Regards, Onesait Platform Team
FranziskusWild
commented
8 months ago Hi again,
we are deploying in an AKS Cluster. We rendered the helm chart and are using the CRDs directly, so that we have more control (f.e. using StatefullSets instead of Deployments for DBs later). But we started out with the original values.
Here are the values provided to the ConrtolPanel:
HZ_SERVICE_DISCOVERY_STRATEGY: "zookeeper"
SCHEDULERDBBURL: "jdbc:mysql://configdb:3306/onesaitplatform_scheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useSSL=false&autoReconnect=true"
SCHEDULERDBBUSER: "root"
SCHEDULERDBBDRIVER: "com.mysql.cj.jdbc.Driver"
SCHEDULERDBBDIALECT: "org.hibernate.dialect.MySQL5Dialect"
QUARTZ_DRIVER: "org.quartz.impl.jdbcjobstore.StdJDBCDelegate"
HZ_ZOOKEEPER_URL: "zookeeper:2181"
CAPTCHA_ON: "false"
CLOUD_GATEWAY: "https://localhost"
HTTP_LOG_LEVEL: "TRACE"
KAFKAENABLED: "false"
KAFKAHOST: "kafkaservice"
KSQLENABLE: "false"
SERVER_NAME: "localhost"
STREAMSETS_SDC_VERSION: "3.18.1"
TWOFA_ENABLED: "false"
MULTITENANCY_ENABLED: "false"
AUTH_PROVIDER: "configdb"
LDAP_ADMIN_DN: ""
LDAP_ADMIN_PASSWORD: ""
LDAP_ADMIN_WHITELIST: "administrator,admin"
LDAP_AUTH_REGEX: "(.*)"
LDAP_BASE_DN: "OU=''"
LDAP_DEFAULT_ROLE: "ROLE_USER"
LDAP_GROUP_WHITELIST: ""
LDAP_IGNORE_PARTIAL_RESULT: "false"
LDAP_IMPORT_USERS: "true"
LDAP_ROLES_MEMBER_ATT: ""
LDAP_URL: ""
AUDITGLOBALNOTIFY: "false"
ELASTICSEARCH_ENABLED: "false"
CONFIGDBMASTERURL: "jdbc:mysql://configdb:3306/onesaitplatform_master_config?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useSSL=false&autoReconnect=true"
CONFIGDBURL: "jdbc:mysql://configdb:3306/onesaitplatform_config?createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true&useSSL=true&autoReconnect=true&nullDatabaseMeansCurrent=true"
CONFIGDBUSER: "root"
CONFIGDB_DRIVER: "com.mysql.cj.jdbc.Driver"
CONFIGDB_DIALECT: "org.hibernate.dialect.MySQL5InnoDBDialect"
REALTIMEDBUSER: "platformadmin"
REALTIMEDBSERVERS: "realtimedb:27017"
ONESAIT_PROPERTIES: "{\"spring\":{\"profile\":{\"active\":\"dev\"}},\"onesaitplatform\":{\"webproject\":{\"rootfolder\":{\"path\":\"/opt/bitnami/nginx/webprojects/\"}}}}"
NOTEBOOKPLATFORMAUTH: "true"And it seems to be the same problem with the cacher server. It connects first, but the can't join the cluster.
Is it maybe a configuration problem with the zookeeper instanz. We have no explicit env vars in the Deploment, is that correct?
And also, would it be possible to get the missing Dockerfiles for zookeeper (and kafka, mariadb, elasticsearch), to better understand the changes you made to the original images?
Thanks a lot
onesaitplatform
commented
8 months ago Hi,
so the problem is with the zookeeper connection, because cacheserver must be started too. I think there might be some configuration issue, because zookeeper has some variables defined in the chart. This one should be set ZOOKEEPER_CLIENT_PORT to 2181. Maybe it depends on the kubernetes version, so there is no proper communication with zookeeper. Which version do you have in AKS?
Sorry, but Dockerfiles are only available in the enterprise version.
Best Regards, Onesait Platform Team
FranziskusWild
commented
8 months ago Hi there,
kubernetes version is 1.29.
Cacheservice is running with:
- name: HZ_SERVICE_DISCOVERY_STRATEGY
value: "zookeeper"
- name: HZ_ZOOKEEPER_URL
value: "zookeeper:2181"and zookepper is running with:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"The reason I asked for the Dockerfiles is that, for example with mariadb we thought about using the operator, but we would need to now what configs are needed. Is that something you could share?
FranziskusWild
commented
8 months ago Ok, I got it to run. I cleaned everything up and startet everything piece by piece. And now the controlcenter works. Unfortunately, I am not sure what the problem actually was. Maybe some of the services can't handle a situation where there dependencies are not up yet.
I want to thank you for your prompt and helpful support. Very appreciated.
I am using the basic helm chart and managed to let everything start up. but the moment traffic hits the controlpanel it restarts. Unfortunately there are no relevant logs before that happens, so I have no clue what might be the problem.
But I saw that when starting up there are some errors in the log the might hint to something. But I would need some help to understand what they mean:
Some help would be apriciated.