[K8s] Problems with tables creation and "Timeout expired while fetching topic metadata"

[pulnara]

Im running this application in Kubernetes (GKE cluster). When trying to create a restaurant in Swagger UI, I'm getting the following error:

When I log in to the ftgo-mysql service, I can see that the table message doesn't exist indeed.

So I've added it (and some others lacking) manually, using the contents of ./mysql/template:

After that, adding a restaurant works:

But the Order Service isn't able to see the new restaurant. After looking into the ftgo-cdc-service logs, it turned out that we're still lacking a table, this time eventuate.cdc_monitoring. So once again, I created it manually using the contents of ./mysql/common-schema.sql.

After that, I've looked into the ftgo-cdc-service logs once again, and got information about successful reconnection to database. However, now I'm getting a problem about fetching information from the Kafka topic:

10:21:44.485 [Timer-1484] INFO  io.eventuate.local.common.DaoUtils - Reconnected to database
Fri May 07 10:21:50 UTC 2021 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
10:22:17.747 [Curator-LeaderSelector-0] ERROR i.e.l.d.l.c.DatabaseOffsetKafkaStore - Timeout expired while fetching topic metadata
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
10:22:57.847 [Curator-LeaderSelector-0] ERROR i.e.l.d.l.c.DatabaseOffsetKafkaStore - Timeout expired while fetching topic metadata
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
10:23:37.947 [Curator-LeaderSelector-0] ERROR i.e.l.d.l.c.DatabaseOffsetKafkaStore - Timeout expired while fetching topic metadata
org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata

It seems that there's some problem with Kafka configuration. I also guess the automatic creation of all the tables should be fixed.

[eedizy]

I have the exact same issue

[pulnara]

@cer could you please let us know how you've overcome these issues in your deployment?

[pulnara]

Okay, it seems I've fixed this problem by changing this port: https://github.com/microservices-patterns/ftgo-application/blob/master/deployment/kubernetes/cdc-service/ftgo-cdc-service.yml#L40 to 9092. Now the error about Kafka doesn't appear in the ftgo-cdc-service. However, the CDC Service still seems to not post any data into the topic.

[pulnara]

I've provided the fix for this problem in https://github.com/microservices-patterns/ftgo-application/pull/131. Looks like the problem laid in database misconfigurations. The application appears to work correctly now, I am able to create a customer, restaurant and order.

[cer]

@pulnara It's great that you fixed this. Sorry for not responding - I had a crazy busy couple of weeks.