Open daigorowhite opened 6 years ago
I've seen this workaround: log into Hive Metastore DB, then run alter table SERDE_PARAMS MODIFY PARAM_VALUE VARCHAR(40000);
Probably want to pick a realistic size, though.
It is one way to avoid this problem, but I think it is depended Hive Setting side and could be shorten again by Hive upgrade.
How about just use avro.schema.url
to have un-limited size .avro
?
@daigorowhite yes, using avro.schema.url
is fine as long as it can be stored somewhere. HDFS is an obvious place, and that requires non-trivial code changes to handle all of the cases (e.g., the schema changes in a consumed topic and needs to be stored somewhere).
I only mentioned the workaround mentioned because it works today and requires no code changes. It may not be ideal, and it may not work for everyone.
See HIVE-12274.
You could manually apply the upgrade script for Mysql
Thanks for sharing it! 👍
Duplicates #145
Hi, team.
With hive.integration=true
, how can kafka-connect-sink
use avro.schema.url
instead of avro.schema.literal
. Or I need alter table manually in Hive?
Hi team, I have one issue with long Avro schema with
kafka-connect-hdfs
hive integration.When I try to sink long schema table into HDFS with
kafka-connect-hdfs
.Success to put data into HDFS , but I got this error when I throw query
I investigated root cause of this, and it is caused the Hive meta data param
varchar
size. https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/avro/AvroHiveUtil.java#L69 https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/avro/AvroHiveUtil.java#L95Do you have any solution/idea for this in
kafka-connect-hdfs
?