rmoff
commented
6 years ago I've tried this too with a STRING key ("key.converter": "org.apache.kafka.connect.storage.StringConverter"), and get the same error from KSQL
{
"name": "twitter_source_avro_02",
"config": {
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081",
"kafka.delete.topic": "twitter_deletes_avro_02",
"twitter.oauth.consumerKey": "XXXX",
"twitter.oauth.consumerSecret": "XXXX",
"twitter.oauth.accessToken": "XXXX",
"twitter.oauth.accessTokenSecret": "XXXX",
"kafka.status.topic": "twitter_avro_02",
"connector.class": "com.github.jcustenborder.kafka.connect.twitter.TwitterSourceConnector",
"process.deletes": true,
"filter.keywords": "rickastley,rmoff,ksql,confluent,jaykreps,gwenshap,apachekafka,nehanarkhede,kafka streams,kafka connect,kafkasummit,kafka,bacon,aws,ilkley"
}
}Sample message (note STRING key):
ksql> print 'twitter_avro_02' from beginning;
Format:AVRO
06/07/18 10:19:46 BST, Struct{Id=1015163384315240448}, {"CreatedAt": 1530868786000, "Id": 1015163384315240448, "Text": "Will be doing lamb burgers with feta and tzaziki\nCrumbed chicken burgers with Asian pickles \nAnd will revive the beef burger with the bacon, blue cheese, bourbon poached pears", "Source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>", "Truncated": true, "InReplyToStatusId": 1015163372000759808, "InReplyToUserId": 2586931947, "InReplyToScreenName": "cheftakura", "GeoLocation": null, "Place": null, "Favorited": false, "Retweeted": false, "FavoriteCount": 0, "User": {"Id": 2586931947, "Name": "Hotelier ��", "ScreenName": "cheftakura", "Location": "Bvumba and Mutare ", "Description": "Wine, food, cricket, economics, music. Probably in that order", "ContributorsEnabled": false, "ProfileImageURL": "http://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO_normal.jpg", "BiggerProfileImageURL": "http://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO_bigger.jpg", "MiniProfileImageURL": "http://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO_mini.jpg", "OriginalProfileImageURL": "http://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO.jpg", "ProfileImageURLHttps": "https://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO_normal.jpg", "BiggerProfileImageURLHttps": "https://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO_bigger.jpg", "MiniProfileImageURLHttps": "https://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO_mini.jpg", "OriginalProfileImageURLHttps": "https://pbs.twimg.com/profile_images/977536986041987073/71-SAVTO.jpg", "DefaultProfileImage": false, "URL": null, "Protected": false, "FollowersCount": 1990, "ProfileBackgroundColor": "8B542B", "ProfileTextColor": "333333", "ProfileLinkColor": "9D582E", "ProfileSidebarFillColor": "EADEAA", "ProfileSidebarBorderColor": "D9B17E", "ProfileUseBackgroundImage": true, "DefaultProfile": false, "ShowAllInlineMedia": false, "FriendsCount": 1503, "CreatedAt": 1402058406000, "FavouritesCount": 57899, "UtcOffset": -1, "TimeZone": null, "ProfileBackgroundImageURL": "http://abs.twimg.com/images/themes/theme8/bg.gif", "ProfileBackgroundImageUrlHttps": "https://abs.twimg.com/images/themes/theme8/bg.gif", "ProfileBannerURL": "https://pbs.twimg.com/profile_banners/2586931947/1493054747/web", "ProfileBannerRetinaURL": "https://pbs.twimg.com/profile_banners/2586931947/1493054747/web_retina", "ProfileBannerIPadURL": "https://pbs.twimg.com/profile_banners/2586931947/1493054747/ipad", "ProfileBannerIPadRetinaURL": "https://pbs.twimg.com/profile_banners/2586931947/1493054747/ipad_retina", "ProfileBannerMobileURL": "https://pbs.twimg.com/profile_banners/2586931947/1493054747/mobile", "ProfileBannerMobileRetinaURL": "https://pbs.twimg.com/profile_banners/2586931947/1493054747/mobile_retina", "ProfileBackgroundTiled": false, "Lang": "en", "StatusesCount": 13593, "GeoEnabled": false, "Verified": false, "Translator": false, "ListedCount": 18, "FollowRequestSent": false, "WithheldInCountries": []}, "Retweet": false, "Contributors": [], "RetweetCount": 0, "RetweetedByMe": false, "CurrentUserRetweetId": -1, "PossiblySensitive": false, "Lang": "en", "WithheldInCountries": [], "HashtagEntities": [], "UserMentionEntities": [], "MediaEntities": [], "SymbolEntities": [], "URLEntities": []}STREAM still fails to create:
ksql> create stream twitter with (kafka_topic='twitter_avro_02',value_format='avro');
Unable to verify the AVRO schema is compatible with KSQL. Map key must be of type STRING
ksql>
rodesai
allenansari174
KSQL 5.0.0-SNAPSHOT (build 50)
With a view to updating this blog to use the new nested Avro support, I tried this out but it fails:
Kafka Connect as the source of data, using @jcustenborder's twitter source.
Kafka Connect config:
Sample message:
Value Schema: