openzipkin / zipkin

Zipkin is a distributed tracing system
https://zipkin.io/
Apache License 2.0
17.02k stars 3.09k forks source link

ES_HTTP_LOGGING appears broken #3714

Closed reta closed 10 months ago

reta commented 10 months ago

There were basically 2 issues:

Sample logs (request body):

zipkin         | {"index":{"_index":"zipkin-span-2024-01-30","_id":"536225a52dbf288f-a973c94cfea8cb9782098ed69b70ec8e"}}
zipkin         | {"timestamp_millis":1706650740662,"_q":["wr","ws","http.method","http.method=GET","http.path","http.path=/zipkin/api/v2/traces"],"traceId":"536225a52dbf288f","id":"536225a52dbf288f","kind":"SER
VER","name":"get /zipkin/api/v2/traces","timestamp":1706650740662116,"duration":381428,"localEndpoint":{"serviceName":"zipkin-server","ipv4":"172.18.0.3"},"remoteEndpoint":{"ipv4":"172.18.0.1","port":41546},"an
notations":[{"timestamp":1706650740662245,"value":"wr"},{"timestamp":1706650741041160,"value":"ws"}],"tags":{"http.method":"GET","http.path":"/zipkin/api/v2/traces"}}
zipkin         | }
zipkin         | 2024-01-30T21:39:02.175Z  INFO [/] 1 --- [orker-epoll-2-5] c.l.a.c.l.LoggingClient                  : [creqId=4fd0f854, chanId=fbe5b66c, laddr=172.18.0.3:52428, raddr=elasticsearch/172.18.0.2:9
200][http://elasticsearch:9200/_bulk#POST] Response: {startTime=2024-01-30T21:39:02.173Z(1706650742173292), length=313B, duration=1285µs(1285103ns), totalDuration=182ms(182880090ns), headers=[:status=200, x-ela
stic-product=Elasticsearch, content-type=application/json, content-encoding=gzip, content-length=313], content={"errors":false,"took":159,"items":[{"index":{"_index":"zipkin-span-2024-01-30","_id":"536225a52dbf
288f-da49c98df47203e6e632b8d8b392a319","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":34,"_primary_term":17,"status":201}},{"index":{"_index":"zipkin-span-2024-01-30",
"_id":"536225a52dbf288f-22efa2babd65cf0f145f1830e9f96fa0","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":32,"_primary_term":17,"status":201}},{"index":{"_index":"zipki
n-span-2024-01-30","_id":"536225a52dbf288f-ce4c65a171114a4df3020cee57ad5ad3","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":32,"_primary_term":17,"status":201}},{"inde
x":{"_index":"zipkin-span-2024-01-30","_id":"536225a52dbf288f-a973c94cfea8cb9782098ed69b70ec8e","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":36,"_primary_term":17,"s
tatus":201}}]}}
zipkin         | 2024-01-30T21:39:05.447Z  INFO [/] 1 --- [orker-epoll-2-5] c.l.a.c.l.LoggingClient                  : [creqId=3eec49c0, sreqId=95fb6523, chanId=fbe5b66c, laddr=172.18.0.3:52428, raddr=elasticse
arch/172.18.0.2:9200][http://elasticsearch:9200/_cluster/health/zipkin*span-*#GET] Request: {startTime=2024-01-30T21:39:05.443Z(1706650745443233), length=0B, duration=874µs(874386ns), scheme=none+h1c, name=get-
cluster-health, headers=[:method=GET, :path=/_cluster/health/zipkin*span-*, :authority=elasticsearch:9200, b3=8f069f5f99c68239-5951f64cf3e5a4c4-0-8f069f5f99c68239, accept-encoding=gzip,deflate,x-snappy-framed,
user-agent=armeria/1.26.4], content=}
zipkin         | 2024-01-30T21:39:05.448Z  INFO [/] 1 --- [orker-epoll-2-5] c.l.a.c.l.LoggingClient                  : [creqId=3eec49c0, sreqId=95fb6523, chanId=fbe5b66c, laddr=172.18.0.3:52428, raddr=elasticse
arch/172.18.0.2:9200][http://elasticsearch:9200/_cluster/health/zipkin*span-*#GET] Response: {startTime=2024-01-30T21:39:05.446Z(1706650745446854), length=228B, duration=155µs(155915ns), totalDuration=3776µs(37
76869ns), headers=[:status=200, x-elastic-product=Elasticsearch, content-type=application/json, content-encoding=gzip, content-length=228], content={"cluster_name":"docker-cluster","status":"yellow","timed_out"
:false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":5,"active_shards":5,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":5,"delayed_unassigned_shards":0,"number_of_pendi
ng_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.0}}
Gracefully stopping... (press Ctrl+C again to force)
reta commented 10 months ago

The build is failing now due to:

#14 1.347 ERROR: unable to select packages:
#14 1.350   mysql-10.11.6-r0:
#14 1.350     breaks: world[mysql~10.11.5]
#14 1.350   mysql-client-10.11.6-r0:
#14 1.350     breaks: world[mysql-client~10.11.5]
#14 ERROR: process "/bin/sh -c (cd install && ./install.sh) && rm -rf ./install" did not complete successfully: exit code: 2

Probably mysql 10.11.6-r0 is being rolled out now ...

codefromthecrypt commented 10 months ago

good guess. you can bump that here for convenience https://pkgs.alpinelinux.org/packages?name=mysql&branch=edge&repo=&arch=&maintainer=

codefromthecrypt commented 10 months ago

we should do a patch after update to armeria 1.27 which addresses some things we've worked around in other areas of the server.