Closed zhao-kun closed 3 years ago
trace ID: 0afe403643632be1
Reproduce: Enter the our portal: https://km.megaease.cn/megacloud/app (XXXXX) Select [Service -> Records ->Trace ID (Input 0afe403643632be1) -> Select correct timerange (2021/03/29 10:00 - 2021/03/29 11:30) ->click search] Click the second span to check the detail annotations information
According to @xxx7xxxx inputs, he tested /user/1
interface of ease-app-backend
in the container loopback network. Got following results:
curl -w "@curl-format.txt" http://127.0.0.1:39527/users/1
{"id":1,"username":"zhao-kun2","type":null,"email":"zhao-kun2@mysql.com","status":null,"department":"IT"}
time_namelookup: 0.000083s
time_connect: 0.000732s
time_appconnect: 0.000000s
time_pretransfer: 0.000832s
time_redirect: 0.000000s
time_starttransfer: 0.021994s
----------
time_total: 0.024681s
For helping us diagnose the issue, I pick a none database accessing /userStatus/1
interface of mesh-app-performance
to test.
I tested it in two way:
/userStatus/1
interface, got the following result: [41] 🐳 → curl -w "@/root/curl-format.txt" http://127.0.0.1:39527/userStatus/1
user 1 status is normal
time_namelookup: 95
time_connect: 424
time_appconnect: 0
time_pretransfer: 552
time_redirect: 0
time_starttransfer: 12654
----------
time_total: 12757
curl -w "@/home/zhaokun/tmp/curl-format.txt" http://127.0.0.1:39526/userStatus/1
user 1 status is normal
time_namelookup: 0.000062
time_connect: 0.000260
time_appconnect: 0.000000
time_pretransfer: 0.000344
time_redirect: 0.000000
time_starttransfer: 0.005456
----------
time_total: 0.005526
> curl-format.txt
\n time_namelookup: %{time_namelookup}\n time_connect: %{time_connect}\n time_appconnect: %{time_appconnect}\n time_pretransfer: %{time_pretransfer}\n time_redirect: %{time_redirect}\n time_starttransfer: %{time_starttransfer}\n ----------\n time_total: %{time_total}\n \n
curl -w "@/Users/akwei/temp/curl-format.txt" http://127.0.0.1:39526/userStatus/1
user 1 status is normal
time_namelookup: 0.000111
time_connect: 0.000490
time_appconnect: 0.000000
time_pretransfer: 0.000543
time_redirect: 0.000000
time_starttransfer: 0.391746
----------
time_total: 0.391778
method-before:22ms, method-after:255ms
curl -w "@/Users/akwei/temp/curl-format.txt" http://127.0.0.1:39526/userStatus/1
user 1 status is normal
time_namelookup: 0.000119
time_connect: 0.000662
time_appconnect: 0.000000
time_pretransfer: 0.000730
time_redirect: 0.000000
time_starttransfer: 0.010690
----------
time_total: 0.010746
method-before:1ms, method-after:1ms
When I invoke multi times, I found agent time info:
method-before:4ms, method-after:1ms
method-before:1ms, method-after:1ms
method-before:1ms, method-after:2ms
When I invoke use sql,
curl -w "@/Users/akwei/temp/curl-format.txt" http://localhost:39527/front/users/1
{"id":1,"username":"akwei","type":null,"email":"akwei@mail.com","status":null,"department":"ui"}
time_namelookup: 0.002375
time_connect: 0.002727
time_appconnect: 0.000000
time_pretransfer: 0.002772
time_redirect: 0.000000
time_starttransfer: 0.051289
----------
time_total: 0.062715
HikariDataSource$$EnhancerBySpringCGLIB$$33a61729#getConnection method-before:0ms, method-after:0ms
HikariProxyPreparedStatement#execute method-before:2ms, method-after:1ms
It seems that Agent did not consume a lot of time.
method-before:22ms, method-after:255ms
Typos?
What's the meaning of method-before
and method-after
?
Explain: method-before : before method invoke method-after: after method invoke
I will fix it read friendly
Explain: method-before : before method invoke method-after: after method invoke
I will fix it read friendly
Does it means that the agent consumes time is method-before
+ method-after
Yes. This is on my computer.
39527
port
03:37:01.319587 lo In IP 127.0.0.1.58166 > 127.0.0.1.39527: Flags [P.], seq 226:339, ack 465, win 86, options [nop,nop,TS val 354688227 ecr 354687246], length 113
0x0000: 4500 00a5 1f15 4000 4006 1d3c 7f00 0001 E.....@.@..<....
0x0010: 7f00 0001 e336 9a67 a558 fa46 1aa0 91a8 .....6.g.X.F....
04:09:40.512059 lo In IP 127.0.0.1.50222 > 127.0.0.1.39527: Flags [P.], seq 474:596, ack 963, win 86, options [nop,nop,TS val 356647403 ecr 356646443], length 122
04:09:40.538563 lo In IP 127.0.0.1.39527 > 127.0.0.1.50222: Flags [P.], seq 963:1209, ack 596, win 86, options [nop,nop,TS val 356647429 ecr 356647403], length 246
In `mesh-app-backend-v1-5656c6b4f4-pjq5m 2/2 Running 0 43m 10.233.111.105 megaease-km4 <none> ` instance, it seems that local Java process consumes nearly 10 ~ 30 ms to reply the package to backend's Ingress. Is this a normal time consuming for logic processing?
## Analysis cmd:
`megaease-km4:➜ ~ |>sudo docker run -it --net container:1ded7018ce94 nicolaka/netshoot `
”Where Has the Time Gone“
:sweat_smile:
After the java agent's observability worked, we can observe the latency between two services. In our environment, we have the following invocation diagram.
I pick a tracing recording, I found the latency between mesh-app-frontend and mesh-app-backend service is higher.
The first section between the two white spots is the communication latency client service (mesh-app-frontend) to server service (mesh-app-backend) . It's apparently too high, about occupies 50% latency of the request.