search

apache / camel-kafka-connector

Camel Kafka Connector allows you to use all Camel components as Kafka Connect connectors
https://camel.apache.org
Apache License 2.0
151 stars 99 forks source link

camel-hdfs-kafka-connector sink configuration with compression #1382

Open vitofico opened 2 years ago

vitofico commented 2 years ago

I have the following .json for creating the hdfs-kafka connector

`{ "name":"CamelHdfsSinkConnector",

"config":{

    "connector.class":"org.apache.camel.kafkaconnector.hdfs.CamelHdfsSinkConnector",
    "task.max":2,

    "key.converter":"org.apache.kafka.connect.storage.StringConverter",
    "value.converter":"org.apache.kafka.connect.storage.StringConverter",

    "transforms": "CamelTypeConverterTransform",
    "transforms.CamelTypeConverterTransform.type": "org.apache.camel.kafkaconnector.transforms.CamelTypeConverterTransform$Value",
    "transforms.CamelTypeConverterTransform.target.type":"java.lang.String",

    "topics":"xxxx",
    "camel.sink.path.hostName": "xxxx",
    "camel.sink.endpoint.namedNodes": "namenode-1:8020,namenode-2:8020",
    "camel.sink.endpoint.splitStrategy": "IDLE:100000,BYTES:132120576",
    "camel.sink.endpoint.compressionCodec": "BZIP2",
    "camel.sink.endpoint.compressionType": "RECORD",
    "camel.sink.path.path": "stream_compression"

}

}`

I don't get any error, but no stream is produced on hadoop. Without the compressionCodec and compressionType lines, it works just fine.

Any hint?

DanielGarcia117 commented 2 years ago

Hello! I'm also having this issue when trying to compress records sent from Kafka to HDFS. In my case the stream is created, but there is no compression of the files in the endpoint. Any help would be welcome.