search

confluentinc / ksql

The database purpose-built for stream processing applications.
https://ksqldb.io
Other
118 stars 1.04k forks source link

can not load data in grafana #300

Closed kuncao closed 7 years ago

kuncao commented 7 years ago

os:suse 12 sp2 NKG1000114449:/home/ksql/ksql-0.1.x/ksql-clickstream-demo/demo # ./ksql-tables-to-grafana.sh Loading Clickstream-Demo TABLES to Confluent-Connect => Elastic => Grafana datasource Logging to: /tmp/ksql-connect.log Charting CLICK_USER_SESSIONS_TS Charting USER_IP_ACTIVITY_TS Charting CLICKSTREAM_STATUS_CODES_TS Charting ENRICHED_ERROR_CODES_TS Charting ERRORS_PER_MIN_ALERT_TS Charting ERRORS_PER_MIN_TS Charting EVENTS_PER_MIN_MAX_AVG_TS Charting EVENTS_PER_MIN_TS Charting PAGES_PER_MIN_TS Navigate to http://localhost:3000/dashboard/db/click-stream-analysis NKG1000114449:/home/ksql/ksql-0.1.x/ksql-clickstream-demo/demo # ./clickstream-analysis-dashboard.sh Loading Grafana ClickStream Dashboard {"slug":"click-stream-analysis","status":"success","version":5}

ksql> SELECT * FROM EVENTS_PER_MIN_TS LIMIT 5; 1505526580000 | 4^�� | 1505526580000 | 4 | 1 1505526580000 | 27^�� | 1505526580000 | 27 | 2 1505526580000 | 2^�� | 1505526580000 | 2 | 1 1505526580000 | 15^�� | 1505526580000 | 15 | 1 1505526580000 | 39^�`� | 1505526580000 | 39 | 1 LIMIT reached for the partition. Query terminated

it seem there is no index in elasticsearch NKG1000114449:/home/ksql/ksql-0.1.x/ksql-clickstream-demo/demo # curl -XGET 'localhost:9200/_cat/indices?v&pretty' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size

kuncao commented 7 years ago

NKG1000114449:/home/ksql/ksql-0.1.x/ksql-clickstream-demo/demo # cat /tmp/ksql-connect.log PROCESSING UPLOAD Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 2841 78338 --:--:-- --:--:-- --:--:-- 82714 {"acknowledged":true}================================================================== Charting CLICK_USER_SESSIONS_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 411 100 411 0 0 55450 0 --:--:-- --:--:-- --:--:-- 58714 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"click_user_sessions_ts","index_uuid":"na","index":"click_user_sessions_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"click_user_sessions_ts","index_uuid":"na","index":"click_user_sessions_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 221 0 --:--:-- --:--:-- --:--:-- 221 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana click_user_sessions_ts Connecting: click_user_sessions_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1619 44655 --:--:-- --:--:-- --:--:-- 48250 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 405 100 71 100 334 896 4215 --:--:-- --:--:-- --:--:-- 4227 {"id":100,"message":"Datasource added","name":"click_user_sessions_ts"}================================================================== Charting USER_IP_ACTIVITY_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 399 100 399 0 0 52041 0 --:--:-- --:--:-- --:--:-- 57000 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"user_ip_activity_ts","index_uuid":"na","index":"user_ip_activity_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"user_ip_activity_ts","index_uuid":"na","index":"user_ip_activity_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 452 0 --:--:-- --:--:-- --:--:-- 458 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana user_ip_activity_ts Connecting: user_ip_activity_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 2669 73598 --:--:-- --:--:-- --:--:-- 82714 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 396 100 68 100 328 848 4093 --:--:-- --:--:-- --:--:-- 4151 {"id":101,"message":"Datasource added","name":"user_ip_activity_ts"}================================================================== Charting CLICKSTREAM_STATUS_CODES_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 431 100 431 0 0 55284 0 --:--:-- --:--:-- --:--:-- 61571 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"clickstream_status_codes_ts","index_uuid":"na","index":"clickstream_status_codes_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"clickstream_status_codes_ts","index_uuid":"na","index":"clickstream_status_codes_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 303 0 --:--:-- --:--:-- --:--:-- 305 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana clickstream_status_codes_ts Connecting: clickstream_status_codes_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1898 52331 --:--:-- --:--:-- --:--:-- 57900 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 420 100 76 100 344 585 2649 --:--:-- --:--:-- --:--:-- 2666 {"id":102,"message":"Datasource added","name":"clickstream_status_codes_ts"}================================================================== Charting ENRICHED_ERROR_CODES_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 415 100 415 0 0 56240 0 --:--:-- --:--:-- --:--:-- 59285 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"enriched_error_codes_ts","index_uuid":"na","index":"enriched_error_codes_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"enriched_error_codes_ts","index_uuid":"na","index":"enriched_error_codes_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 436 0 --:--:-- --:--:-- --:--:-- 440 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana enriched_error_codes_ts Connecting: enriched_error_codes_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1723 47533 --:--:-- --:--:-- --:--:-- 48250 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 408 100 72 100 336 600 2802 --:--:-- --:--:-- --:--:-- 2800 {"id":103,"message":"Datasource added","name":"enriched_error_codes_ts"}================================================================== Charting ERRORS_PER_MIN_ALERT_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 415 100 415 0 0 60725 0 --:--:-- --:--:-- --:--:-- 69166 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"errors_per_min_alert_ts","index_uuid":"na","index":"errors_per_min_alert_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"errors_per_min_alert_ts","index_uuid":"na","index":"errors_per_min_alert_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 372 0 --:--:-- --:--:-- --:--:-- 375 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana errors_per_min_alert_ts Connecting: errors_per_min_alert_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 2479 68350 --:--:-- --:--:-- --:--:-- 72375 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 408 100 72 100 336 968 4520 --:--:-- --:--:-- --:--:-- 4540 {"id":104,"message":"Datasource added","name":"errors_per_min_alert_ts"}================================================================== Charting ERRORS_PER_MIN_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 391 100 391 0 0 57584 0 --:--:-- --:--:-- --:--:-- 65166 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"errors_per_min_ts","index_uuid":"na","index":"errors_per_min_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"errors_per_min_ts","index_uuid":"na","index":"errors_per_min_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 414 0 --:--:-- --:--:-- --:--:-- 417 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana errors_per_min_ts Connecting: errors_per_min_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1864 51407 --:--:-- --:--:-- --:--:-- 52636 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 390 100 66 100 324 1024 5031 --:--:-- --:--:-- --:--:-- 5062 {"id":105,"message":"Datasource added","name":"errors_per_min_ts"}================================================================== Charting EVENTS_PER_MIN_MAX_AVG_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 423 100 423 0 0 61824 0 --:--:-- --:--:-- --:--:-- 70500 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"events_per_min_max_avg_ts","index_uuid":"na","index":"events_per_min_max_avg_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"events_per_min_max_avg_ts","index_uuid":"na","index":"events_per_min_max_avg_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 481 0 --:--:-- --:--:-- --:--:-- 485 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana events_per_min_max_avg_ts Connecting: events_per_min_max_avg_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1820 50195 --:--:-- --:--:-- --:--:-- 52636 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 414 100 74 100 340 947 4353 --:--:-- --:--:-- --:--:-- 4358 {"id":106,"message":"Datasource added","name":"events_per_min_max_avg_ts"}================================================================== Charting EVENTS_PER_MIN_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 391 100 391 0 0 60386 0 --:--:-- --:--:-- --:--:-- 65166 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"events_per_min_ts","index_uuid":"na","index":"events_per_min_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"events_per_min_ts","index_uuid":"na","index":"events_per_min_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 261 0 --:--:-- --:--:-- --:--:-- 261 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana events_per_min_ts Connecting: events_per_min_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1735 47847 --:--:-- --:--:-- --:--:-- 52636 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 390 100 66 100 324 727 3572 --:--:-- --:--:-- --:--:-- 3600 {"id":107,"message":"Datasource added","name":"events_per_min_ts"}================================================================== Charting PAGES_PER_MIN_TS % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 387 100 387 0 0 63840 0 --:--:-- --:--:-- --:--:-- 77400 {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"pages_per_min_ts","index_uuid":"na","index":"pages_per_min_ts"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"pages_per_min_ts","index_uuid":"na","index":"pages_per_min_ts"},"status":404} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 33 100 33 0 0 323 0 --:--:-- --:--:-- --:--:-- 326 {"message":"Data source deleted"}\n\nConnecting KSQL->Elastic->Grafana pages_per_min_ts Connecting: pages_per_min_ts Loading Elastic Dynamic Template to ensure _TS fields are used for TimeStamp % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 600 100 21 100 579 1822 50247 --:--:-- --:--:-- --:--:-- 52636 {"acknowledged":true}Adding Elastic Source\n\n % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8083: Connection refused

Adding Grafana Source % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 387 100 65 100 322 1106 5483 --:--:-- --:--:-- --:--:-- 5551 {"id":108,"message":"Datasource added","name":"pages_per_min_ts"}

kuncao commented 7 years ago

please help to check what's wrong? thanks a lot

kuncao commented 7 years ago

according curl: (7) Failed to connect to localhost port 8083: Connection refused message netstat -anp|grep 8083 there is no record restart confluent ./confluent start zookeeper is already running. Try restarting if needed kafka is already running. Try restarting if needed schema-registry is already running. Try restarting if needed kafka-rest is already running. Try restarting if needed Starting connect connect is [UP]----note here , kafka connect is up check 8083 port netstat -tulpn | grep 8083 tcp 0 0 :::8083 :::* LISTEN 125511/java

run ./ksql-tables-to-grafana.sh again the data can be show in grafana