search

jaegertracing / spark-dependencies

Spark job for dependency links
http://jaegertracing.io/
Apache License 2.0
126 stars 70 forks source link

spark-dependencies can not run normally in docker #90

Open yangwang888 opened 4 years ago

yangwang888 commented 4 years ago

I'm running spark-dependencies in a docker container, it seems that it can't run normally which will exit after 5 seconds, my command is docker run -d --env STORAGE=elasticsearch --env ES_NODES=http://192.167.0.x: 9600 jaegertracing/spark-dependencies:latest

and the log is : 20/06/04 06:59:02 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 20/06/04 06:59:03 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2020-06-04T00:00Z, reading from jaeger-span-2020-06-04 index, result storing to jaeger-dependencies-2020-06-04 20/06/04 06:59:04 INFO ElasticsearchDependenciesJob: Done, 0 dependency objects created

i use this command to verify data: curl http://localhost:9600/jaeger-span-2020-06-04/_search?pretty=true&q=*:*, and it gives me this : "took" : 19, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 46, "max_score" : 1.0, "hits" : [ { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "U0SefnIBHXsa-__vuvXR", "_score" : 1.0, "_source" : { "traceID" : "c75708f073c4caf0", "spanID" : "c75708f073c4caf0", "flags" : 1, "operationName" : "grpc-request",

my jaeger config is: cat docker-compose.yaml

version: '2' services: jaeger-collector: image: jaegertracing/jaeger-collector command:

` please help me with this,thank you~

pavolloffay commented 4 years ago

The address of the ES in curl is different than the address in the docker run for spark dependencies.

Could you run the curl with the same address? 

curl http://192.167.0.x:9600/jaeger-span-2020-06-04/_search?pretty=true&q=*:*

or run the docker with --net=host and use localhost as the ES address.

yangwang888 commented 4 years ago

The address of the ES in curl is different than the address in the docker run for spark dependencies.

Could you run the curl with the same address?

curl http://192.167.0.x:9600/jaeger-span-2020-06-04/_search?pretty=true&q=*:*

or run the docker with --net=host and use localhost as the ES address.

yes, curl http://192.167.0.103:9600/jaeger-span-2020-06-04/_search?pretty=true&q=*:* |head -n 5 "took" : 2, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 46, "max_score" : 1.0, "hits" : [ { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "U0SefnIBHXsa-__vuvXR", "_score" : 1.0, "_source" : { "traceID" : "c75708f073c4caf0",

pavolloffay commented 4 years ago

The spark job can connect to the ES otherwise it would fail. Maybe the data in the index does not contain enough information to create the links.

Could you please share some of the data?

yangwang888 commented 4 years ago

the data is here,please check "took" : 9, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 46, "max_score" : 1.0, "hits" : [ { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "U0SefnIBHXsa-__vuvXR", "_score" : 1.0, "_source" : { "traceID" : "c75708f073c4caf0", "spanID" : "c75708f073c4caf0", "flags" : 1, "operationName" : "grpc-request", "references" : [ ], "startTime" : 1591262230521000, "startTimeMillis" : 1591262230521, "duration" : 12000, "tags" : [ { "key" : "sampler.type", "type" : "string", "value" : "const" }, { "key" : "sampler.param", "type" : "bool", "value" : "true" }, { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "grpc.service", "type" : "string", "value" : "BeepPurchase" }, { "key" : "grpc.method", "type" : "string", "value" : "getBeepOrder" }, { "key" : "request.id", "type" : "string", "value" : "f5327723-58e8-4f99-ae05-1e3c1c467c7f" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "purchase-app-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "7879aa61-29bc-4a78-8134-2da341fe7e98" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "VESefnIBHXsa-__vu_V2", "_score" : 1.0, "_source" : { "traceID" : "c75708f073c4caf0", "spanID" : "9f3894845f637d3a", "flags" : 1, "operationName" : "database-query", "references" : [ { "refType" : "CHILD_OF", "traceID" : "c75708f073c4caf0", "spanID" : "8fca16ae82f77279" } ], "startTime" : 1591262230522000, "startTimeMillis" : 1591262230522, "duration" : 10000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "persistence.repo", "type" : "string", "value" : "OnlineTransactionRepo" }, { "key" : "persistence.method", "type" : "string", "value" : "getOneByReceiptNumber" }, { "key" : "request.id", "type" : "string", "value" : "f5327723-58e8-4f99-ae05-1e3c1c467c7f" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "transaction-domain-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "d1e39359-a0b2-4b9f-9399-b82501ca21b3" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "VUSefnIBHXsa-__vu_Wf", "_score" : 1.0, "_source" : { "traceID" : "c75708f073c4caf0", "spanID" : "8fca16ae82f77279", "flags" : 1, "operationName" : "grpc-request", "references" : [ { "refType" : "CHILD_OF", "traceID" : "c75708f073c4caf0", "spanID" : "c75708f073c4caf0" } ], "startTime" : 1591262230522000, "startTimeMillis" : 1591262230522, "duration" : 11000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "grpc.service", "type" : "string", "value" : "OnlineOrder" }, { "key" : "grpc.method", "type" : "string", "value" : "getOrder" }, { "key" : "request.id", "type" : "string", "value" : "f5327723-58e8-4f99-ae05-1e3c1c467c7f" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "transaction-domain-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "d1e39359-a0b2-4b9f-9399-b82501ca21b3" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "VkSefnIBHXsa-__vvfX3", "_score" : 1.0, "_source" : { "traceID" : "47b8c55b92fda498", "spanID" : "47b8c55b92fda498", "flags" : 1, "operationName" : "http-request", "references" : [ ], "startTime" : 1591262143432000, "startTimeMillis" : 1591262143432, "duration" : 87108000, "tags" : [ { "key" : "sampler.type", "type" : "string", "value" : "const" }, { "key" : "sampler.param", "type" : "bool", "value" : "true" }, { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "http.url", "type" : "string", "value" : "/test?receipt_number=816795832612380" }, { "key" : "http.method", "type" : "string", "value" : "GET" }, { "key" : "requestId", "type" : "string", "value" : "undefined" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "beep-bff-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "76c26727-e3a7-4120-9db9-bc55667704b5" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "V0SpfnIBHXsa-__vIfVX", "_score" : 1.0, "_source" : { "traceID" : "d61e39d6f1de22a1", "spanID" : "812de3d6ceab74bf", "flags" : 1, "operationName" : "grpc-request", "references" : [ { "refType" : "CHILD_OF", "traceID" : "d61e39d6f1de22a1", "spanID" : "d61e39d6f1de22a1" } ], "startTime" : 1591262912208000, "startTimeMillis" : 1591262912208, "duration" : 23000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "grpc.service", "type" : "string", "value" : "BeepPurchase" }, { "key" : "grpc.method", "type" : "string", "value" : "getBeepOrder" }, { "key" : "request.id", "type" : "string", "value" : "f8e96e93-f2e1-4c73-9f4a-7f1bd82ba997" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "purchase-app-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "4e696196-210f-498a-86a2-8b3d3c2ed62b" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "WESpfnIBHXsa-__vIvUf", "_score" : 1.0, "_source" : { "traceID" : "d61e39d6f1de22a1", "spanID" : "a523d72ec4b5214c", "flags" : 1, "operationName" : "database-query", "references" : [ { "refType" : "CHILD_OF", "traceID" : "d61e39d6f1de22a1", "spanID" : "7a2ca5eb174548e6" } ], "startTime" : 1591262912216000, "startTimeMillis" : 1591262912216, "duration" : 9000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "persistence.repo", "type" : "string", "value" : "OnlineTransactionRepo" }, { "key" : "persistence.method", "type" : "string", "value" : "getOneByReceiptNumber" }, { "key" : "request.id", "type" : "string", "value" : "f8e96e93-f2e1-4c73-9f4a-7f1bd82ba997" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "transaction-domain-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "d1e39359-a0b2-4b9f-9399-b82501ca21b3" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "WUSpfnIBHXsa-__vIvUf", "_score" : 1.0, "_source" : { "traceID" : "d61e39d6f1de22a1", "spanID" : "7a2ca5eb174548e6", "flags" : 1, "operationName" : "grpc-request", "references" : [ { "refType" : "CHILD_OF", "traceID" : "d61e39d6f1de22a1", "spanID" : "812de3d6ceab74bf" } ], "startTime" : 1591262912216000, "startTimeMillis" : 1591262912216, "duration" : 17000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "grpc.service", "type" : "string", "value" : "OnlineOrder" }, { "key" : "grpc.method", "type" : "string", "value" : "getOrder" }, { "key" : "request.id", "type" : "string", "value" : "f8e96e93-f2e1-4c73-9f4a-7f1bd82ba997" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "transaction-domain-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "d1e39359-a0b2-4b9f-9399-b82501ca21b3" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "WkSpfnIBHXsa-__vIvXn", "_score" : 1.0, "_source" : { "traceID" : "d61e39d6f1de22a1", "spanID" : "d61e39d6f1de22a1", "flags" : 1, "operationName" : "http-request", "references" : [ ], "startTime" : 1591262912203000, "startTimeMillis" : 1591262912203, "duration" : 33000, "tags" : [ { "key" : "sampler.type", "type" : "string", "value" : "const" }, { "key" : "sampler.param", "type" : "bool", "value" : "true" }, { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "http.url", "type" : "string", "value" : "/test?receipt_number=816795832612380" }, { "key" : "http.method", "type" : "string", "value" : "GET" }, { "key" : "requestId", "type" : "string", "value" : "undefined" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "beep-v1-bff", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "3d6aa51e-d0a5-4545-af80-b6b594510847" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "X0SrfnIBHXsa-__v_vXv", "_score" : 1.0, "_source" : { "traceID" : "f1ec4f787d1ebbe7", "spanID" : "aa38de0d76e9a352", "flags" : 1, "operationName" : "database-query", "references" : [ { "refType" : "CHILD_OF", "traceID" : "f1ec4f787d1ebbe7", "spanID" : "0ce730ab7a8b9d1d" } ], "startTime" : 1591263100178000, "startTimeMillis" : 1591263100178, "duration" : 9000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "persistence.repo", "type" : "string", "value" : "OnlineTransactionRepo" }, { "key" : "persistence.method", "type" : "string", "value" : "getOneByReceiptNumber" }, { "key" : "request.id", "type" : "string", "value" : "14ccaf7a-6b65-4035-a7ff-6c1c11378089" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "transaction-domain-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "d1e39359-a0b2-4b9f-9399-b82501ca21b3" } ] } } }, { "_index" : "jaeger-span-2020-06-04", "_type" : "span", "_id" : "YESrfnIBHXsa-__v_vXv", "_score" : 1.0, "_source" : { "traceID" : "f1ec4f787d1ebbe7", "spanID" : "0ce730ab7a8b9d1d", "flags" : 1, "operationName" : "grpc-request", "references" : [ { "refType" : "CHILD_OF", "traceID" : "f1ec4f787d1ebbe7", "spanID" : "cd5235cb8956ffd8" } ], "startTime" : 1591263100178000, "startTimeMillis" : 1591263100178, "duration" : 14000, "tags" : [ { "key" : "span.kind", "type" : "string", "value" : "server" }, { "key" : "grpc.service", "type" : "string", "value" : "OnlineOrder" }, { "key" : "grpc.method", "type" : "string", "value" : "getOrder" }, { "key" : "request.id", "type" : "string", "value" : "14ccaf7a-6b65-4035-a7ff-6c1c11378089" }, { "key" : "internal.span.format", "type" : "string", "value" : "jaeger" } ], "logs" : [ ], "process" : { "serviceName" : "transaction-domain-svc", "tags" : [ { "key" : "jaeger.version", "type" : "string", "value" : "Node-3.18.0" }, { "key" : "hostname", "type" : "string", "value" : "Fans-MBP.localdomain" }, { "key" : "ip", "type" : "string", "value" : "192.168.1.10" }, { "key" : "client-uuid", "type" : "string", "value" : "d1e39359-a0b2-4b9f-9399-b82501ca21b3" } ] } } } ] } }

pavolloffay commented 4 years ago

The data looks good, although it would be better to verify it in a different way.

I would suggest to run the Jaeger hotrod example https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod and then run the span dependencies jobs. The job should create dependencies links.

I am adding example how to run all bits as docker container, but I would suggest you just run the hotrod example and specify the URL to your deployed collector. The logs from hotrod should not contain messages like

2020-06-15T09:46:01.293Z    ERROR   tracing/init.go:62  error reporting Jaeger span "HTTP GET: /route": Post "http://localhost:14268/api/traces": dial tcp 127.0.0.1:14268: connect: connection refused
docker run -it --rm -e "ES_JAVA_OPTS=-Xms2g -Xmx2g" -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "discovery.type=single-node" --name=elasticsearch docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.4
docker run -it --rm --net=host -e SPAN_STORAGE_TYPE=elasticsearch jaegertracing/all-in-one:1.18.0
docker run -it --rm --net=host -e JAEGER_ENDPOINT=http://localhost:14268/api/traces jaegertracing/example-hotrod all
docker run --rm -it --net=host --env STORAGE=elasticsearch --env ES_NODES=http://localhost:9200 jaegertracing/spark-dependencies:latest

Don't forget to open UI on port 8080 and create a couple of transactions to create that will create spans.

yangwang888 commented 4 years ago

@pavolloffay thanks a lot,i will try this