search

airyhq / airy

💬 Open Source App Framework to build streaming apps with real-time data - 💎 Build real-time data pipelines and make real-time data universally accessible - 🤖 Join historical and real-time data in the stream to create smarter ML and AI applications. - ⚡ Standardize complex data ingestion and stream data to apps with pre-built connectors
https://airy.co/docs/core
Apache License 2.0
369 stars 44 forks source link

API Communication crashes with default replication factor error #3510

Closed steffh closed 2 years ago

steffh commented 2 years ago

Describe the bug

As an Airy user with Airy instance vs. 0.47.0, installed via Helm with no changes to default configuration (embedded Kafka with 1 broker), activating Chat Plugin via Control Center, API Communication pod crashes with the following error:

[~/code/k8s-confluent-v1] $kubectl logs api-communication-6c8c75cf87-8z7fs

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.6.4)

[2022-07-31 20:51:27,971] [INFO] [AirySpringBootApplication] {} Starting AirySpringBootApplication using Java 11.0.6 on api-communication-6c8c75cf87-8z7fs with PID 1 (/app_springboot.jar started by root in /) 
[2022-07-31 20:51:27,985] [INFO] [AirySpringBootApplication] {} No active profile set, falling back to 1 default profile: "default" 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/app_springboot.jar!/BOOT-INF/lib/slf4j-nop-1.7.32.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/app_springboot.jar!/BOOT-INF/lib/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
[2022-07-31 20:51:34,804] [INFO] [Version] {} HV000001: Hibernate Validator 6.1.7.Final 
[2022-07-31 20:51:36,466] [INFO] [AirySpringBootApplication] {} Started AirySpringBootApplication in 9.501 seconds (JVM running for 11.327) 
Exception in thread "api.CommunicationStores-StreamThread-2" org.apache.kafka.streams.errors.StreamsException: Could not create topic api.CommunicationStores-KSTREAM-AGGREGATE-STATE-STORE-0000000017-repartition.
    at org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:165)
    at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.prepareRepartitionTopics(StreamsPartitionAssignor.java:498)
    at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:352)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:687)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1000(AbstractCoordinator.java:111)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:597)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:560)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1173)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1148)
    at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206)
    at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169)
    at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801)
    at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:763)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:614)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

To Reproduce

Steps to reproduce the behavior:

see above

Expected behavior

Should not crash, should be configurable in control center

ljupcovangelski commented 2 years ago

Resolved with #3506.