Failed to make API calls to external services - Githubissues

metalbear-co / mirrord

Connect your local process and your cloud environment, and run local code in cloud conditions.

https://mirrord.dev

MIT License

3.82k stars 104 forks source link

Failed to make API calls to external services #2644

Closed algobot76 closed 3 months ago

algobot76 commented 3 months ago

Bug Description

My app makes calls to external api endpoints, however the app logs showed that it failed to receive responses when i ran it with mirrored enabled.

Steps to Reproduce

Run a golang app that makes api calls to external services (api.vultr.com in my case).
Read the logs produced by mirrrod agent.

Backtrace

No response

Relevant Logs

[2m2024-08-09T02:43:21.425167Z[0m [34mDEBUG[0m ThreadId(01) [2mmirrord_agent::entrypoint[0m[2m:[0m main -> Initializing mirrord-agent, version 3.112.1.
[2m2024-08-09T02:43:21.425625Z[0m [34mDEBUG[0m ThreadId(01) [2mmirrord_agent::entrypoint[0m[2m:[0m start_iptable_guard -> Initializing iptable-guard.
[2m2024-08-09T02:43:21.425688Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mmirrord_agent::container_handle[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427420Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; idle [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427451Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; connecting [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427721Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; not ready [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427758Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; connecting [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427803Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mhyper::client::conn[0m[2m:[0m client handshake Http2 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427835Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m: [2mh2::client[0m[2m:[0m binding client connection [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427848Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m: [2mh2::client[0m[2m:[0m client connection bound [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427905Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427916Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m send [3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 } [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427923Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::frame::settings[0m[2m:[0m encoding SETTINGS; len=18 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427927Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::frame::settings[0m[2m:[0m encoding setting; val=EnablePush(0) [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427930Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::frame::settings[0m[2m:[0m encoding setting; val=InitialWindowSize(2097152) [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427934Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::frame::settings[0m[2m:[0m encoding setting; val=MaxFrameSize(16384) [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427937Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m encoded settings [3mrem[0m[2m=[0m27 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427942Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m close [3mtime.busy[0m[2m=[0m25.9µs [3mtime.idle[0m[2m=[0m11.5µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }[0m
[2m2024-08-09T02:43:21.427955Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mh2::proto::streams::flow_control[0m[2m:[0m inc_window; sz=65535; old=0; new=65535 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427961Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mh2::proto::streams::flow_control[0m[2m:[0m inc_window; sz=65535; old=0; new=65535 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427965Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m Prioritize::new; flow=FlowControl { window_size: Window(65535), available: Window(65535) } [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.427971Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m:[1mConnection[0m: [2mh2::proto::connection[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.427995Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mh2::proto::streams::recv[0m[2m:[0m set_target_connection_window; target=5242880; available=65535, reserved=0 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428010Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; connected [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428020Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; not ready [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428052Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::connection[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428059Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::connection[0m[2m:[0m [3mconnection.state[0m[2m=[0mOpen [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428073Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m: [2mh2::proto::connection[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428079Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m: [2mh2::proto::connection[0m[2m:[0m close [3mtime.busy[0m[2m=[0m1.74µs [3mtime.idle[0m[2m=[0m4.10µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428087Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428091Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m poll [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428106Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m [3mread.bytes[0m[2m=[0m15 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428112Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m:[1mFramedRead::decode_frame[0m: [2mh2::codec::framed_read[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3moffset[0m[2m=[0m15[0m
[2m2024-08-09T02:43:21.428116Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m:[1mFramedRead::decode_frame[0m: [2mh2::codec::framed_read[0m[2m:[0m decoding frame from 15B [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3moffset[0m[2m=[0m15[0m
[2m2024-08-09T02:43:21.428120Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m:[1mFramedRead::decode_frame[0m: [2mh2::codec::framed_read[0m[2m:[0m [3mframe.kind[0m[2m=[0mSettings [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3moffset[0m[2m=[0m15[0m
[2m2024-08-09T02:43:21.428126Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m:[1mFramedRead::decode_frame[0m: [2mh2::codec::framed_read[0m[2m:[0m close [3mtime.busy[0m[2m=[0m9.55µs [3mtime.idle[0m[2m=[0m3.98µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3moffset[0m[2m=[0m15[0m
[2m2024-08-09T02:43:21.428131Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m received [3mframe[0m[2m=[0mSettings { flags: (0x0), max_frame_size: 16384 } [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428136Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m close [3mtime.busy[0m[2m=[0m45.0µs [3mtime.idle[0m[2m=[0m3.94µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428142Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::connection[0m[2m:[0m recv SETTINGS [3mframe[0m[2m=[0mSettings { flags: (0x0), max_frame_size: 16384 } [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428151Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m: [2mh2::proto::connection[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428155Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x1: ACK) }[0m
[2m2024-08-09T02:43:21.428159Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m send [3mframe[0m[2m=[0mSettings { flags: (0x1: ACK) } [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x1: ACK) }[0m
[2m2024-08-09T02:43:21.428163Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m:[1mFramedWrite::buffer[0m: [2mh2::frame::settings[0m[2m:[0m encoding SETTINGS; len=0 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x1: ACK) }[0m
[2m2024-08-09T02:43:21.428167Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m encoded settings [3mrem[0m[2m=[0m36 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x1: ACK) }[0m
[2m2024-08-09T02:43:21.428171Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m close [3mtime.busy[0m[2m=[0m11.3µs [3mtime.idle[0m[2m=[0m3.99µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mSettings { flags: (0x1: ACK) }[0m
[2m2024-08-09T02:43:21.428176Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m: [2mh2::proto::settings[0m[2m:[0m ACK sent; applying settings [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428185Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpoll_ready[0m: [2mh2::proto::connection[0m[2m:[0m close [3mtime.busy[0m[2m=[0m30.6µs [3mtime.idle[0m[2m=[0m3.57µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428192Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428207Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m poll [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428211Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedRead::poll_next[0m: [2mh2::codec::framed_read[0m[2m:[0m close [3mtime.busy[0m[2m=[0m4.37µs [3mtime.idle[0m[2m=[0m15.3µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428218Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mWindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }[0m
[2m2024-08-09T02:43:21.428221Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m send [3mframe[0m[2m=[0mWindowUpdate { stream_id: StreamId(0), size_increment: 5177345 } [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mWindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }[0m
[2m2024-08-09T02:43:21.428226Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::buffer[0m: [2mh2::frame::window_update[0m[2m:[0m encoding WINDOW_UPDATE; id=StreamId(0) [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mWindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }[0m
[2m2024-08-09T02:43:21.428230Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m encoded window_update [3mrem[0m[2m=[0m49 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mWindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }[0m
[2m2024-08-09T02:43:21.428234Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::buffer[0m: [2mh2::codec::framed_write[0m[2m:[0m close [3mtime.busy[0m[2m=[0m12.7µs [3mtime.idle[0m[2m=[0m3.67µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m [2m[3mframe[0m[2m=[0mWindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }[0m
[2m2024-08-09T02:43:21.428240Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::streams::flow_control[0m[2m:[0m inc_window; sz=5177345; old=65535; new=5242880 [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428246Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mtry_reclaim_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428250Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mtry_reclaim_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m180ns [3mtime.idle[0m[2m=[0m3.86µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428258Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m poll_complete [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428261Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m schedule_pending_open [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428266Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpop_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428271Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mpop_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m308ns [3mtime.idle[0m[2m=[0m3.85µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428276Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::flush[0m: [2mh2::codec::framed_write[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428282Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::flush[0m: [2mh2::codec::framed_write[0m[2m:[0m [3mqueued_data_frame[0m[2m=[0mfalse [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428290Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::flush[0m: [2mh2::codec::framed_write[0m[2m:[0m flushing buffer [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428296Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mFramedWrite::flush[0m: [2mh2::codec::framed_write[0m[2m:[0m close [3mtime.busy[0m[2m=[0m14.7µs [3mtime.idle[0m[2m=[0m5.03µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428300Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mtry_reclaim_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428304Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m:[1mtry_reclaim_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m227ns [3mtime.idle[0m[2m=[0m3.30µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428309Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m:[1mConnection[0m:[1mpoll[0m: [2mh2::proto::connection[0m[2m:[0m close [3mtime.busy[0m[2m=[0m250µs [3mtime.idle[0m[2m=[0m6.89µs [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m [2m[3mpeer[0m[2m=[0mClient[0m
[2m2024-08-09T02:43:21.428333Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; connected [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428336Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; ready [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428389Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtower::buffer::service[0m[2m:[0m sending request to buffer worker [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428397Z[0m [35mTRACE[0m ThreadId(01) [2mtower::buffer::worker[0m[2m:[0m worker polling for next message
[2m2024-08-09T02:43:21.428402Z[0m [35mTRACE[0m ThreadId(01) [2mtower::buffer::worker[0m[2m:[0m processing new request
[2m2024-08-09T02:43:21.428406Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtower::buffer::worker[0m[2m:[0m [3mresumed[0m[2m=[0mfalse worker received request; waiting for service readiness [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428409Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; connected [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428412Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m poll_ready; ready [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428415Z[0m [34mDEBUG[0m ThreadId(01) [1mnew[0m: [2mtower::buffer::worker[0m[2m:[0m [3mservice.ready[0m[2m=[0mtrue processing request [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428421Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtonic::transport::service::reconnect[0m[2m:[0m Reconnect::call [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428425Z[0m [35mTRACE[0m ThreadId(01) [1mnew[0m: [2mtower::buffer::worker[0m[2m:[0m returning response future [2m[3mcontainer[0m[2m=[0mContainerd(ContainerdContainer { container_id: "f89b970b6fbab7d7a552814eac485cba82a610c283eb739da3f472890de7bf29" })[0m
[2m2024-08-09T02:43:21.428430Z[0m [35mTRACE[0m ThreadId(01) [2mtower::buffer::worker[0m[2m:[0m worker polling for next message
[2m2024-08-09T02:43:21.428447Z[0m [35mTRACE[0m ThreadId(01) [2mh2::proto::streams::flow_control[0m[2m:[0m inc_window; sz=2097152; old=0; new=2097152
[2m2024-08-09T02:43:21.428452Z[0m [35mTRACE[0m ThreadId(01) [2mh2::proto::streams::flow_control[0m[2m:[0m inc_window; sz=65535; old=0; new=65535
[2m2024-08-09T02:43:21.428463Z[0m [35mTRACE[0m ThreadId(01) [2mh2::proto::streams::send[0m[2m:[0m send_headers; frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }; init_window=65535
[2m2024-08-09T02:43:21.428471Z[0m [35mTRACE[0m ThreadId(01) [2mh2::proto::streams::store[0m[2m:[0m Queue::push_back
[2m2024-08-09T02:43:21.428474Z[0m [35mTRACE[0m ThreadId(01) [2mh2::proto::streams::store[0m[2m:[0m  -> first entry
[2m2024-08-09T02:43:21.428477Z[0m [35mTRACE[0m ThreadId(01) [1mPrioritize::queue_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428487Z[0m [35mTRACE[0m ThreadId(01) [1mPrioritize::queue_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m6.54µs [3mtime.idle[0m[2m=[0m3.04µs [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428497Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m
[2m2024-08-09T02:43:21.428502Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428505Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m [3mrequested[0m[2m=[0m1 [3madditional[0m[2m=[0m1 [3mbuffered[0m[2m=[0m0 [3mwindow[0m[2m=[0m65535 [3mconn[0m[2m=[0m65535 [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428509Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m assigning [3mcapacity[0m[2m=[0m1 [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428513Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::stream[0m[2m:[0m   assigned capacity to stream; available=1; buffered=0; id=StreamId(1); max_buffer_size=1048576 prev=0 [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428516Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::stream[0m[2m:[0m   notifying task [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428520Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m [3mavailable[0m[2m=[0m1 [3mrequested[0m[2m=[0m1 [3mbuffered[0m[2m=[0m0 [3mhas_unavailable[0m[2m=[0mtrue [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428524Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m19.4µs [3mtime.idle[0m[2m=[0m3.39µs [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428531Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m31.1µs [3mtime.idle[0m[2m=[0m2.90µs [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m1 [3mcurr[0m[2m=[0m0[0m
[2m2024-08-09T02:43:21.428538Z[0m [35mTRACE[0m ThreadId(01) [2mhyper::proto::h2[0m[2m:[0m send body chunk: 71 bytes, eos=false
[2m2024-08-09T02:43:21.428544Z[0m [35mTRACE[0m ThreadId(01) [1msend_data[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3msz[0m[2m=[0m71 [3mrequested[0m[2m=[0m1[0m
[2m2024-08-09T02:43:21.428547Z[0m [35mTRACE[0m ThreadId(01) [1msend_data[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m [3mbuffered[0m[2m=[0m71 [2m[3msz[0m[2m=[0m71 [3mrequested[0m[2m=[0m1[0m
[2m2024-08-09T02:43:21.428551Z[0m [35mTRACE[0m ThreadId(01) [1msend_data[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m [3mavailable[0m[2m=[0m1 [3mbuffered[0m[2m=[0m71 [2m[3msz[0m[2m=[0m71 [3mrequested[0m[2m=[0m1[0m
[2m2024-08-09T02:43:21.428555Z[0m [35mTRACE[0m ThreadId(01) [1msend_data[0m:[1mPrioritize::queue_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3msz[0m[2m=[0m71 [3mrequested[0m[2m=[0m1[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428559Z[0m [35mTRACE[0m ThreadId(01) [1msend_data[0m:[1mPrioritize::queue_frame[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m269ns [3mtime.idle[0m[2m=[0m3.42µs [2m[3msz[0m[2m=[0m71 [3mrequested[0m[2m=[0m1[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428564Z[0m [35mTRACE[0m ThreadId(01) [1msend_data[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m close [3mtime.busy[0m[2m=[0m16.6µs [3mtime.idle[0m[2m=[0m2.90µs [2m[3msz[0m[2m=[0m71 [3mrequested[0m[2m=[0m1[0m
[2m2024-08-09T02:43:21.428569Z[0m [35mTRACE[0m ThreadId(01) [2mh2::proto::streams::counts[0m[2m:[0m transition_after; stream=StreamId(1); state=State { inner: Open { local: Streaming, remote: AwaitingHeaders } }; is_closed=false; pending_send_empty=false; buffered_send_data=71; num_recv=0; num_send=0
[2m2024-08-09T02:43:21.428575Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m72 [3mcurr[0m[2m=[0m71[0m
[2m2024-08-09T02:43:21.428579Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m new [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m72 [3mcurr[0m[2m=[0m71[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428582Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m [3mrequested[0m[2m=[0m72 [3madditional[0m[2m=[0m71 [3mbuffered[0m[2m=[0m71 [3mwindow[0m[2m=[0m65535 [3mconn[0m[2m=[0m65534 [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m72 [3mcurr[0m[2m=[0m71[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)[0m
[2m2024-08-09T02:43:21.428586Z[0m [35mTRACE[0m ThreadId(01) [1mreserve_capacity[0m:[1mtry_assign_capacity[0m: [2mh2::proto::streams::prioritize[0m[2m:[0m assigning [3mcapacity[0m[2m=[0m71 [2m[3mstream.id[0m[2m=[0mStreamId(1) [3mrequested[0m[2m=[0m1 [3meffective[0m[2m=[0m72 [3mcurr[0m[2m=[0m71[0m [2m[3mstream.id[0m[2m=[0mStreamId(1)

Your operating system and version

MacOS 14.2.1

Local process

cim

Local process version

No response

Additional Info

No response

algobot76 commented 3 months ago

full log: mirrord-agent.log

Razz4780 commented 3 months ago

I don't see any failure in the agent logs. Can we get local logs and maybe some code snippet we can use to reproduce the issue? You can enable local logs by:

Setting RUST_LOG=mirrord=trace env for the run. This will dump layer logs to stderr of the application

Enabling intproxy logs in the mirrord config:

{
"internal_proxy": {
"log_level": "mirrord=trace",
"log_destination": "where/to/dump/logs"
}
}

Would be great if we had all of agent/layer/intproxy logs from the same run

aviramha commented 3 months ago

Adjusted @Razz4780 json to be more correct (intproxy -> internal_proxy, added missing })

algobot76 commented 3 months ago

I don't see any failure in the agent logs. Can we get local logs and maybe some code snippet we can use to reproduce the issue? You can enable local logs by:

Setting RUST_LOG=mirrord=trace env for the run. This will dump layer logs to stderr of the application

Enabling intproxy logs in the mirrord config:
{
  "internal_proxy": {
    "log_level": "mirrord=trace",
    "log_destination": "where/to/dump/logs"
   }
}
Would be great if we had all of agent/layer/intproxy logs from the same run

I tried again with the latest release and it was working! It cloud have been fixed in recent releases or my poor network connection made the previous release unusable.

algobot76 commented 3 months ago

Closing the issue as it can no longer be reproduced.