confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client

https://github.com/confluentinc/confluent-kafka-dotnet/wiki

Apache License 2.0

80 stars 866 forks source link

how to stream the file or folder to one machine to another machine using Kafka.net #819

Closed dhavalsp closed 5 years ago

dhavalsp commented 5 years ago

Description

I want to stream the file or folder using the kafka.net in one machine to another machine. Is there any functionality to do that. I want to track the process of file streaming like how many percentages of data transferred already.

Checklist

Please provide the following information:

[1.0 ] Confluent.Kafka nuget version:
[2.12 ] Apache Kafka version:
[ ] Client configuration:
[ ] Operating system:
[ ] Provide logs (with "debug" : "..." as necessary in configuration)
[ ] Provide broker log excerpts
[ ] Critical issue

mhowlett commented 5 years ago

there's no built in functionality for that - you'd need to break up the files, send the pieces as messages and reconstruct at the other end. you'd need logic for handling failure scenarios. idempotent producer (available in 1.0) will be useful. Note that Kafka is optimized for smaller message sizes - IIRC there's a notable drop off in throughput @ around 30Mb. Preferably keep message sizes to around 1Mb or less.