LGouellec / kafka-streams-dotnet

.NET Stream Processing Library for Apache Kafka 🚀
https://lgouellec.github.io/kafka-streams-dotnet/
MIT License
455 stars 74 forks source link

Readme example fails if 'localhost' is used instead of ip address #146

Closed Assoillyng closed 1 year ago

Assoillyng commented 2 years ago

Description

Basic example from readme fails when creating the stream, if 'localhost' is used instead of an ip address. No errors are shown, no exceptions thrown, the app just shuts down.

How to reproduce

Create the following method:

        public async Task Start()
        {
            var config = new StreamConfig<StringSerDes, StringSerDes>();
            config.ApplicationId = "test-app";
            config.BootstrapServers = "localhost:9092";

            StreamBuilder builder = new StreamBuilder();

            var kstream = builder.Stream<string, string>("stream");
            var ktable = builder.Table("table", InMemory<string, string>.As("table-store"));

            kstream.Join(ktable, (v, v1) => $"{v}-{v1}")
                   .To("join-topic");

            Topology t = builder.Build();
            KafkaStream stream = new KafkaStream(t, config);

            Console.CancelKeyPress += (o, e) => {
                stream.Dispose();
            };

            await stream.StartAsync();
        }

The only difference between this example and that in the readme is BootstrapServers. The app will exit with no errors at the line: KafkaStream stream = new KafkaStream(t, config);

Change localhost to the ip address of your local machine, and it no longer happens.

OS: Windows 10 Streamiz.Kafka.Net 1.2.2 Kafka 2.8.1 on docker

Checklist

Please provide the following information:

Assoillyng commented 2 years ago

Scratch that, it still fails, but now at:

stream.StartAsync()

I am finding this difficult to use because I am having various crashes at different points (I also have it crash when trying to do builder.Table(...).ToStream()) but with no information about what has gone wrong. Is there logging somewhere that I am missing? I just see the app crashing with no explanation.

LGouellec commented 2 years ago

Hi @Assoillyng,

How do you bootstrap your Kafka cluster ?

Could you provide your stack configuration (docker-compose, server.properties etc ...) please ?

Kr,

Assoillyng commented 2 years ago

Hi @LGouellec,

My docker-compose.yaml is:

version: '3'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

docker hub: https://hub.docker.com/r/wurstmeister/kafka Github: https://github.com/wurstmeister/kafka-docker

I run this and it just works out of the box. I have written some other .NET6 producers and consumers that are able to communicate with kafka just fine, using Confluent.Kafka 1.8.2.

Assoillyng commented 2 years ago

Hi Again @LGouellec ,

I have found a solution, the problem seems to have nothing to do with Streamiz.Kafka.Net. When setting up my .NET6 project, I checked the 'use docker' option. When I debug via docker I get random crashes, when I change to the application itself, most of the crashes do not occur, and I can see thrown exceptions:

image

I am not sure why exactly this happens, perhaps I have misconfigured docker somehow?

LGouellec commented 2 years ago

Hi @Assoillyng,

Can you use my setup docker-compose here please : https://github.com/LGouellec/kafka-streams-dotnet/blob/develop/environment/docker-compose.yml

Kr,

LGouellec commented 1 year ago

Close this issue due to a lack of time response.