Failed to start MDS

[ongtsg]

Tried to run ./script/start.sh to start cp-demo but encountered the following error.. please advise. Thanks. Waiting up to 120 seconds for MDS to start ........................ERROR: Failed after 120 seconds. Please troubleshoot and run again. For troubleshooting instructions see https://docs.confluent.io/current/tutorials/cp-demo/docs/index.html#troubleshooting % docker-compose logs kafka1
Attaching to kafka1 kafka1 | ERROR: Did not find SSL certificates in /etc/kafka/secrets/ (did you remember to run ./scripts/start.sh instead of docker-compose up -d?) % docker-compose ps Name Command State Ports

kafka1 bash -c if [ ! -f /etc/kaf ... Exit 1 kafka2 bash -c if [ ! -f /etc/kaf ... Exit 1 openldap /container/tool/run --copy ... Up 0.0.0.0:389->389/tcp, 636/tcp tools /bin/bash Up zookeeper /etc/confluent/docker/run Up 0.0.0.0:2181->2181/tcp, 0.0.0.0:2182->2182/tcp, 2888/tcp, 3888/tcp

[awalther28]

Hi @ongtsg thank you for bringing this to our attention. At first glance I don't have an answer for you but will investigate and report back.

[awalther28]

@ongtsg After looking at this further, it looks like maybe the certs weren't generated correctly on your side. I say this because that error you are reporting occurs when kafka.kafka1.keystore.jks is not found. This file should be autogenerated for you when using the start script. You should be able to find that file in cp-demo/scripts/security after running the start script. Can you please try rerunning with CLEAN=true ./scripts/start.sh? This will force the certs to be regenerated. If still failing, please check the cp-demo/scripts/security folder for kafka.kafka1.keystore.jks. I would say please list all of the files in the security folder but mine has 188 :)

Your output from running CLEAN=true ./scripts/start.sh should have something like the following near the beginning.

Generate keys and certificates used for SSL (see /Users/awalther/Documents/devx/cp-demo/scripts/helper/security)
Generating a 2048 bit RSA private key
..............................................................+++
...................................................................................................+++
writing new private key to 'snakeoil-ca-1.key'
-----
Creating certificates
Created certificates for client
Created certificates for controlCenterAndKsqlDBServer
Created certificates for appSA
Created certificates for clientListen
Created certificates for kafka1
Created certificates for connectorSA
Created certificates for kafka2
Created certificates for mds
Created certificates for connect
Created certificates for zookeeper
Created certificates for badapp
Created certificates for schemaregistry
Created certificates for ksqlDBUser
Created certificates for restproxy
Creating certificates completed
Generating public and private keys for token signing
Generating RSA private key, 2048 bit long modulus

For the record, I did not have any issues with the 6.0.1 ./script/start.sh.

[ongtsg]

THanks Allison, below are the output of the CLEAN=true start.sh command as well as files listing in security directory..I am still getting the same MDS error. Thanks for advise..

output of start.sh ongt@Terences-MacBook-Pro cp-demo % CLEAN=true ./scripts/start.sh Stopping zookeeper ... done Removing tools ... done Removing kafka1 ... done Removing kafka2 ... done Removing zookeeper ... done Removing openldap ... done Removing network cp-demo_default CLEAN=true -> deleting existing certificates and local Connect Docker image generated by cp-demo Error: No such image: localbuild/connect:6.0.1-6.0.1

Environment parameters REPOSITORY=confluentinc CONNECTOR_VERSION=6.0.1 CLEAN=true VIZ=true C3_KSQLDB_HTTPS=false

Generate keys and certificates used for SSL (see /Users/ongt/cp-demo/scripts/helper/security) Generating a 2048 bit RSA private key .+++ .....+++ writing new private key to 'snakeoil-ca-1.key'

Creating certificates Creating certificates completed Generating public and private keys for token signing Generating RSA private key, 2048 bit long modulus ...........................................................................................................+++ ............................................................................................+++ e is 65537 (0x10001) writing RSA key Setting insecure permissions on some files in /Users/ongt/cp-demo/scripts/helper/../security for demo purposes

Creating network "cp-demo_default" with the default driver Creating openldap ... done openldap is up-to-date Creating zookeeper ... done Creating kafka2 ... done Creating kafka1 ... done Creating tools ... done Waiting up to 120 seconds for MDS to start ........................ERROR: Failed after 120 seconds. Please troubleshoot and run again. For troubleshooting instructions see https://docs.confluent.io/current/tutorials/cp-demo/docs/index.html#troubleshooting

Files in cp-demo/script/security after running start.sh script ongt@Terences-MacBook-Pro security % ls -ltr total 592 -rw-r--r-- 1 ongt staff 251 Jan 11 19:34 appSA.config -rw-r--r-- 1 ongt staff 427 Jan 11 19:34 badclient_without_interceptors.config -rw-r--r-- 1 ongt staff 491 Jan 11 19:34 broker_jaas.conf -rwxr-xr-x 1 ongt staff 239 Jan 11 19:34 certs-clean.sh -rwxr-xr-x 1 ongt staff 3594 Jan 11 19:34 certs-create-per-user.sh -rwxr-xr-x 1 ongt staff 1507 Jan 11 19:34 certs-create.sh -rwxr-xr-x 1 ongt staff 650 Jan 11 19:34 certs-verify.sh -rw-r--r-- 1 ongt staff 822 Jan 11 19:34 clientListen_with_interceptors.config -rw-r--r-- 1 ongt staff 194 Jan 11 19:34 client_sasl_plain.config -rw-r--r-- 1 ongt staff 794 Jan 11 19:34 client_with_interceptors.config -rw-r--r-- 1 ongt staff 421 Jan 11 19:34 client_without_interceptors.config -rw-r--r-- 1 ongt staff 265 Jan 11 19:34 client_without_interceptors_ssl.config -rw-r--r-- 1 ongt staff 263 Jan 11 19:34 connectorSA_without_interceptors_ssl.config -rw-r--r-- 1 ongt staff 261 Jan 11 19:34 ksqlDBUser_without_interceptors_ssl.config drwxr-xr-x 21 ongt staff 672 Jan 11 19:34 ldap_users -rw-r--r-- 1 ongt staff 435 Jan 11 19:34 unknownclient_without_interceptors.config -rw-r--r-- 1 ongt staff 144 Jan 11 19:34 zookeeper_jaas.conf drwxr-xr-x 4 ongt staff 128 Jan 11 19:49 keypair -rw-r--r-- 1 ongt staff 1858 Jan 12 06:59 snakeoil-ca-1.key -rw-r--r-- 1 ongt staff 1237 Jan 12 06:59 snakeoil-ca-1.crt -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 kafka1_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 kafka1_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 kafka1_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 kafka2_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 kafka2_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 kafka2_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 client_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 client_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 client_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 restproxy_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 restproxy_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 restproxy_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 schemaregistry_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 schemaregistry_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 schemaregistry_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 connectorSA_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 connectorSA_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 connectorSA_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 mds_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 mds_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 mds_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 connect_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 connect_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 connect_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 appSA_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 controlCenterAndKsqlDBServer_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 appSA_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 controlCenterAndKsqlDBServer_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 appSA_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 controlCenterAndKsqlDBServer_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 ksqlDBUser_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 ksqlDBUser_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 zookeeper_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 ksqlDBUser_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 zookeeper_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 zookeeper_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 clientListen_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 clientListen_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 clientListen_truststore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 badapp_sslkey_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 badapp_keystore_creds -rw-r--r-- 1 ongt staff 10 Jan 12 06:59 badapp_truststore_creds -rw-r--r-- 1 ongt staff 1612 Jan 12 06:59 certs-create-kafka1.log -rw-r--r-- 1 ongt staff 1612 Jan 12 06:59 certs-create-kafka2.log -rw-r--r-- 1 ongt staff 1612 Jan 12 06:59 certs-create-client.log -rw-r--r-- 1 ongt staff 1627 Jan 12 06:59 certs-create-restproxy.log -rw-r--r-- 1 ongt staff 1652 Jan 12 06:59 certs-create-schemaregistry.log -rw-r--r-- 1 ongt staff 1637 Jan 12 06:59 certs-create-connectorSA.log -rw-r--r-- 1 ongt staff 1597 Jan 12 06:59 certs-create-mds.log -rw-r--r-- 1 ongt staff 1617 Jan 12 06:59 certs-create-connect.log -rw-r--r-- 1 ongt staff 1722 Jan 12 06:59 certs-create-controlCenterAndKsqlDBServer.log -rw-r--r-- 1 ongt staff 1607 Jan 12 06:59 certs-create-appSA.log -rw-r--r-- 1 ongt staff 1632 Jan 12 06:59 certs-create-ksqlDBUser.log -rw-r--r-- 1 ongt staff 1627 Jan 12 06:59 certs-create-zookeeper.log -rw-r--r-- 1 ongt staff 1642 Jan 12 06:59 certs-create-clientListen.log -rw-r--r-- 1 ongt staff 1612 Jan 12 06:59 certs-create-badapp.log

[awalther28]

@ongtsg can you print the contents of scripts/security/certs-create-kafka1.log andscripts/security/certs-create-kafka2.log?

Mine (kafka1) contains the following:

Signature ok
subject=/C=US/ST=Ca/L=PaloAlto/O=CONFLUENT/OU=TEST/CN=kafka1
Getting CA Private Key
Certificate was added to keystore
Certificate reply was installed in keystore
Certificate was added to keystore
Certificate stored in file <kafka1.der>
Importing keystore kafka.kafka1.keystore.jks to kafka1.keystore.p12...
Entry for alias kafka1 successfully imported.
Entry for alias snakeoil-caroot successfully imported.
Import command completed:  2 entries successfully imported, 0 entries failed or cancelled
MAC verified OK

[ongtsg]

Thanks Allison.. from the scripts/security/certs-create-kafka1.log, it shows missing Java runtime. I have since installed the Java as required and this problem is resolved. Thank you so much for your prompt support.

---

DONE! Connect to Confluent Control Center at http://localhost:9021 (login as xxx for full access)

---