[Bug]: Docker compose file not working with adaptive Sampling

slayer321 commented 11 months ago

What happened?

I'm trying to use Adaptive Sampling in Jaeger and following the blog I found this docker-compose file where we can run jaeger specifying SAMPLING_CONFIG_TYPE=adaptive env (the only changes I did in the collector env is I added this two env as well SPAN_STORAGE_TYPE=cassandra , SAMPLING_STORAGE_TYPE=cassandra ) After running this docker-compose file when I'm seeing the UI I cannot see any of the services ,on the HotROD application I tried booking some rides so that traces get generated but nothing happened .

I also check the cassendra database and below is the output I'm getting

cqlsh:jaeger_v1_dc1> DESCRIBE TABLES;

dependencies_v2  operation_names_v2      service_name_index       tag_index
duration_index   operation_throughput    service_names            traces   
leases           sampling_probabilities  service_operation_index

cqlsh:jaeger_v1_dc1> select * from service_names;

 service_name
--------------

(0 rows)
cqlsh:jaeger_v1_dc1> select * from service_name_index;

 service_name | bucket | start_time | trace_id
--------------+--------+------------+----------

(0 rows)
cqlsh:jaeger_v1_dc1> select * from traces;

 trace_id | span_id | span_hash | duration | flags | logs | operation_name | parent_id | process | refs | start_time | tags
----------+---------+-----------+----------+-------+------+----------------+-----------+---------+------+------------+------

(0 rows)

After checking the logs of hotROD container below is the error I'm getting

2023/09/20 09:36:12 traces export: Post "http://localhost:4318/v1/traces": dial tcp 127.0.0.1:4318: connect: connection refused
2023/09/20 09:36:13 traces export: Post "http://localhost:4318/v1/traces": dial tcp 127.0.0.1:4318: connect: connection refused
2023/09/20 09:36:13 traces export: Post "http://localhost:4318/v1/traces": dial tcp 127.0.0.1:4318: connect: connection refused
2023/09/20 09:36:13 traces export: Post "http://localhost:4318/v1/traces": dial tcp 127.0.0.1:4318: connect: connection refused

After checking for hotrod docker compose file in example dir I found that there was some recent changes done for hotrod default exporter.I tried making same changes to the above docker compose file .. but it didn't help .

Steps to reproduce

Run docker compose up from docker-compose dir
On the Jaeger UI no services are seen
Try booking ride on hotrod application no traces are seen on the UI

Expected behavior

Once we run docker compose up command all the services should be seen on Jaeger UI and then the traces should also be seen.

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

v1.49.0

SDK

No response

Pipeline

No response

Stogage backend

Cassendra

Operating system

Linux

Deployment model

Docker

Deployment configs

jaeger-docker-compose.yaml

yurishkuro commented 11 months ago

Action items:

[ ] find out why this was not caught in CI, I thought the reason hotrod is included in that docker-compose is because we were validating it somehow
[ ] fix the settings in docker-compose to use OTEL env vars, not the deprecated Jaeger SDK env vars
[ ] validate that adaptive sampling works with OTEL SDK (again, I thought we had a CI test for that)

james-ryans commented 11 months ago

Hi! I've encountered this issue too while trying to tackle the other problem. I don't think this issue is specific to adaptive sampling, we'll get caught into this issue even by running docker compose up without modifying anything. After investigating a bit, the cause of this issue is that COLLECTOR_OTLP_ENABLED=true env was removed at https://github.com/jaegertracing/jaeger/pull/4698/files#diff-299f817cc4ab077ddb763f1e6a023d9d042d714e2fd3736cc40af3f218d44f1eL9 PR. Actually, it is my jaegertracing/all-in-one:latest image is not updated.

Investigated items:

I can't find any integration test between hotrod and all-in-one services. What I found is that hotrod tests only the frontend side scripts/hotrod-integration-test.sh
~~Is it enough if I only bring back the COLLECTOR_OTLP_ENABLED=true env?~~ I'll try to continue investigating this issue.

I don't think hotrod with OTEL SDK works with adaptive sampling. I've tried to add OTEL_TRACES_SAMPLER=jaeger_remote env(found this env here) but output unsupported sampler: jaeger_remote, seems like hotrod using go.opentelemetry.io/otel package while jaeger_remote only supported in go.opentelemetry.io/contrib package

hotrod-hotrod-1  | 2023-10-05T15:49:18.529Z INFO    cobra@v1.6.1/command.go:946 Using expvar as metrics backend
hotrod-hotrod-1  | 2023-10-05T15:49:18.529Z INFO    cobra@v1.6.1/command.go:916 Starting all services
hotrod-hotrod-1  | 2023/10/05 15:49:18 unsupported sampler: jaeger_remote
hotrod-hotrod-1  | 2023/10/05 15:49:18 unsupported sampler: jaeger_remote
hotrod-hotrod-1  | 2023-10-05T15:49:18.530Z INFO    frontend/server.go:74   Starting    {"service": "frontend", "address": "http://0.0.0.0:8080"}
hotrod-hotrod-1  | 2023/10/05 15:49:18 unsupported sampler: jaeger_remote
hotrod-hotrod-1  | 2023-10-05T15:49:18.530Z INFO    customer/server.go:55   Starting    {"service": "customer", "address": "http://0.0.0.0:8081"}
hotrod-hotrod-1  | 2023/10/05 15:49:18 unsupported sampler: jaeger_remote
hotrod-hotrod-1  | 2023/10/05 15:49:18 unsupported sampler: jaeger_remote
hotrod-hotrod-1  | 2023/10/05 15:49:18 unsupported sampler: jaeger_remote
hotrod-hotrod-1  | 2023-10-05T15:49:18.531Z INFO    route/server.go:57  Starting    {"service": "route", "address": "http://0.0.0.0:8083"}

varshith257 commented 3 months ago

@yurishkuro https://github.com/jaegertracing/jaeger/pull/4803#discussion_r1347757510

Can it be closed?

yurishkuro commented 3 months ago

why? None of the checklist items seem to be addressed: https://github.com/jaegertracing/jaeger/issues/4768#issuecomment-1727972225

jaegertracing / jaeger