LGouellec / kafka-streams-dotnet

.NET Stream Processing Library for Apache Kafka 🚀
https://lgouellec.github.io/kafka-streams-dotnet/
MIT License
455 stars 74 forks source link

Not possible to join two streams with complex key type #71

Closed lichutin-st closed 3 years ago

lichutin-st commented 3 years ago

Description

Hello! Looks like it's impossible to join two streams with complex key types.

I tried to do so and got an exception [StreamsException] The serializer is not compatible to the actual key (Key type: Program.UserKey). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters(using the DSL)

I debugged through Streamiz.Kafka.Net library and found this: https://github.com/LGouellec/kafka-streams-dotnet/blob/develop/core/Stream/StreamJoinProps.cs#L221 Looks like Key and Value SerDes are ignored when StreamJoinProps is being built from another StreamJoinProps (and that operations is performed during the Join node building

Nuget version: 1.1.3

How to reproduce

class UserKey
{
    public string Id { get; set; }
    public string CustomerId { get; set; }
}

IKStream<UserKey, object> stream1;
IKStream<UserKey, object> stream2;

stream1.Join(
    stream2,
    (entry1, entry) => new { entry1, entry },
    JoinWindowOptions.Of(TimeSpan.FromMinutes(1)),
    StreamJoinProps.With(
        keySerde: new JsonSerializer<UserKey>(),
        valueSerde: new JsonSerializer<object>(),
        otherValueSerde: new JsonSerializer<object>()
    )
)

Checklist

Please provide the following information:

LGouellec commented 3 years ago

Hi @lichutin-st

Thanks for your issue. Exactly, your analysis is correct. I'll try to fix a new release 1.1.5 ASAP.

Regards,

LGouellec commented 3 years ago

Added unit test to specify this use case see