search

lensesio / fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
https://lenses.io
Apache License 2.0
2.01k stars 330 forks source link

Twitter Source connector is not working #152

Open parthagrawal9 opened 4 years ago

parthagrawal9 commented 4 years ago

Properties :

name=source-twitter-distributed
connector.class=com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
tasks.max=1
topic=demo-3-twitter
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
twitter.consumerkey=*****
twitter.consumersecret=****
twitter.token=****
twitter.secret=***
track.terms=programming,java,kafka,scala
language=en

Create button is disabled. 

Error: lines is null
$onInit/<@http://localhost:3030/kafka-connect-ui/src/combined.js?rel=c0d1d01eec:2867:25
hh</this.$$parseAndValidate@http://localhost:3030/kafka-connect-ui/bower_components/angular/angular.min.js:291:290
hh</this.$commitViewValue@http://localhost:3030/kafka-connect-ui/bower_components/angular/angular.min.js:291:126
hh</this.$$debounceViewValueCommit/J<@http://localhost:3030/kafka-connect-ui/bower_components/angular/angular.min.js:293:32
f/q<@http://localhost:3030/kafka-connect-ui/bower_components/angular/angular.min.js:160:52
e@http://localhost:3030/kafka-connect-ui/bower_components/angular/angular.min.js:45:487
og/k.defer/c<@http://localhost:3030/kafka-connect-ui/bower_components/angular/angular.min.js:48:415

Getting the above error on browser console.

Baresse commented 4 years ago

@parthagrawal9

It seems this connector is outdated which make the UI crash. If you try to use the Kafka Connect API like this : 

curl -X POST \
  http://localhost:8083/connectors \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -d '{
  "name": "source-twitter-distributed",
  "config": {
    "connector.class": "com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector",
    "tasks.max": "1",
    "topic": "demo-3-twitter",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "true",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "true",
    "twitter.consumerkey": "*******",
    "twitter.consumersecret": "*******",
    "twitter.token": "*******",
    "twitter.secret": "*******",
    "track.terms": "programming,java,kafka,scala",
    "language": "en"
  }
}'

You will have the following error message :

{"error_code":500,"message":"Must configure one of topics or topics.regex"}

There is still a workaround, you have to add a topics attribute like this : 

curl -X POST \
  http://localhost:8083/connectors \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json' \
  -d '{
  "name": "source-twitter-distributed",
  "config": {
    "connector.class": "com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector",
    "tasks.max": "1",
    "topic": "demo-3-twitter",
    "topics": "demo-3-twitter",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "true",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "true",
    "twitter.consumerkey": "*******",
    "twitter.consumersecret": "*******",
    "twitter.token": "*******",
    "twitter.secret": "*******",
    "track.terms": "programming,java,kafka,scala",
    "language": "en"
  }
}'

May be lenseio should stop to bundle this connector since it crashes the UI and may be add a new one like this : https://github.com/jcustenborder/kafka-connect-twitter

If you want to give a try to another Twitter connect you just have to update you docker compose file like this : 

version: '3'

services:
  # this is our kafka cluster.
  kafka-cluster:
    image: lensesio/fast-data-dev:2.5.0
    environment:
      ADV_HOST: 127.0.0.1         # Change to 192.168.99.100 if using Docker Toolbox
      RUNTESTS: 0                 # Disable Running tests so the cluster starts faster
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect ports
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker 
    volumes:
      - ./connectors/jcustenborder-kafka-connect-twitter-0.3.33:/connectors/jcustenborder-kafka-connect-twitter
vskubsad commented 3 years ago

I have tried with topics..not working... It is an error from the angular app.. Pleas fix... Also, I tried enabling the create button from removing the disabled attribute.. it is still not working... Below is the screenshot of the error...

Error-1 Error-2
aaelius commented 3 years ago

Same issues here, trying to complete a Udemy Kafka tutorial but I'm stuck in this step.