search

hashicorp / boundary

Boundary enables identity-based access management for dynamic infrastructure.
https://boundaryproject.io
Other
3.85k stars 289 forks source link

boundary worker unresponsive after sometime. #2598

Closed swapnil-raj closed 1 year ago

swapnil-raj commented 2 years ago

I am running boundary v0.11.0 on k8s with 3 controllers and 3 workers. When I start boundary controller and worker everything works fine but after sometime workers become unresponsive.

controller config 
disable_mlock = true
controller {
    name = "boundary-controller"
    description = "Boundary Controller"
    public_cluster_addr = "boundary-controller-hl.boundary.svc.cluster.local"
    database {
    url = "postgres url"
}
}
listener "tcp" {
purpose = "api"
address = "0.0.0.0"
tls_disable = true
public_addr = "boundary-controller-hl.boundary.svc.cluster.local"
}
listener "tcp" {
    address = "0.0.0.0"
    purpose = "cluster"
    tls_disable = true
    public_addr = "boundary-controller-hl.boundary.svc.cluster.local"
}
kms "aead" {
    purpose = "root"
    aead_type = "aes-gcm"
    key = "key"
    key_id = "global_root"
    public_addr = "boundary-controller-hl.boundary.svc.cluster.local"
}
kms "aead" {
    purpose = "worker-auth"
    aead_type = "aes-gcm"
    key = "key"
    key_id = "global_worker-auth"
}
kms "aead" {
    purpose = "recovery"
    aead_type = "aes-gcm"
    key = "key="
    key_id = "global_recovery"
}

worker config 

disable_mlock = true
             listener "tcp" {
             purpose = "proxy"
             address = "0.0.0.0"
             tls_disable = true
             }
             worker {
             name = "env://HOSTNAME"
             description = "Boundar k8s worker"
             initial_upstreams = ["boundary-controller-hl.boundary.svc.cluster.local"]
             public_addr = "public DNS"
             tags {
                 region    = ["k8s"]
             }
             }
             # Worker authorization KMS
             # Use a production KMS such as AWS KMS for production installs
             # This key is the same key used in the worker configuration
             kms "aead" {
                 purpose = "worker-auth"
                 aead_type = "aes-gcm"
                 key = "key"
                 key_id = "global_worker-auth"
             }

It generally works for a day and then stops working until i restart the pod.

I see these error logs before worker stops working

{"id":"3HyVunohBm","source":"https://hashicorp.com/boundary/boundary-worker-8qdn8/worker","specversion":"1.0","type":"error","data":{"error":"failed to read protobuf message: failed to get reader: failed to read frame header: EOF","error_fields":{},"id":"e_I9lbsVu4LP","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"msg":"error reading handshake from client"}},"datacontentype":"application/cloudevents","time":"2022-11-07T15:37:06.905432729Z"} {"id":"fl6DyubRiA","source":"https://hashicorp.com/boundary/boundary-worker-8qdn8/worker","specversion":"1.0","type":"error","data":{"error":"failed to close WebSocket: failed to write control frame opClose: WebSocket closed: failed to read frame header: EOF","error_fields":{},"id":"e_8WduHdtyWv","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"msg":"error closing client connection"}},"datacontentype":"application/cloudevents","time":"2022-11-07T15:37:06.906030265Z"}

One more thing that i saw is, when it is working if i do curl -v 0.0.0.0:9202 i immediately get 

*   Trying 0.0.0.0:9202...
* Connected to 0.0.0.0 (127.0.0.1) port 9202 (#0)
> GET / HTTP/1.1
> Host: 0.0.0.0:9202
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

but when its not working the server does not reply anything, it gets stuck on

*   Trying 127.0.0.1:9202...
* Connected to 127.0.0.1 (127.0.0.1) port 9202 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9202
> User-Agent: curl/7.79.1
> Accept: */*
>

What can be the issue?

Thanks in advance.

jefferai commented 2 years ago

That message is coming after initial client connection to the worker; the connection is upgraded to a websocket connection, and then we attempt to read a message from the client over that websocket connection. The curl test is a bit weird here because I wouldn't expect that it hangs.

Do you have any other logs from the worker around that time?

When it hangs, can you send a SIGQUIT to the worker process and get us the stack trace?

swapnil-raj commented 2 years ago

The logs looks normal to me before it hangs up

{"id":"FPOCBfW1aS","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_2quPeel8RB","msg":"connection closed","session_id":"s_QvMJ9yMhlI"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:11:20.930595445Z"}
{"id":"lAH0bU19MT","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_fAZKL2ROzb","msg":"connection closed","session_id":"s_w5Gf9tuPk9"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:11:23.000624187Z"}
{"id":"tuMNy5RAo0","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_LWLwaJCcF3","msg":"connection successfully authorized","session_id":"s_zBbjsrp4Tm"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:19:42.36369556Z"}
{"id":"lPvmnBCKKH","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_sHd9oT2lkT","msg":"connection successfully authorized","session_id":"s_QvMJ9yMhlI"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:20:07.233579524Z"}
{"id":"xf4Y0Z7g0f","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"msg":"session successfully activated","session_id":"s_lJpWLM2G9r"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:21:44.433019019Z"}
{"id":"HvH95AAhKT","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_dR8kKNakxv","msg":"connection successfully authorized","session_id":"s_lJpWLM2G9r"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:21:44.44514018Z"}
{"id":"CXeo3KETld","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"msg":"session successfully activated","session_id":"s_W3DLPYFhYC"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:23:07.375349924Z"}
{"id":"cHdFxVBXe0","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_5Fu7pEItfS","msg":"connection successfully authorized","session_id":"s_W3DLPYFhYC"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:23:07.383856494Z"}
{"id":"crJBgAAkBA","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_5Fu7pEItfS","msg":"connection closed","session_id":"s_W3DLPYFhYC"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:23:14.343205223Z"}
{"id":"tboJWViyEY","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"error","data":{"error":"failed to read protobuf message: failed to get reader: failed to read frame header: EOF","error_fields":{},"id":"e_UiV8VoJ762","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"msg":"error reading handshake from client"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:23:27.83409188Z"}
{"id":"e7BA7zhrjV","source":"https://hashicorp.com/boundary/boundary-worker-bptzf/worker","specversion":"1.0","type":"error","data":{"error":"failed to close WebSocket: failed to write control frame opClose: WebSocket closed: failed to read frame header: EOF","error_fields":{},"id":"e_GQWaa5iBrL","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"msg":"error closing client connection"}},"datacontentype":"application/cloudevents","time":"2022-11-09T05:23:27.834377079Z"}

I will Try sending SIGQUIT when this happens again.

Also, One more thing i observed is, if i leave worker in hanged state it comes back to life after some time.

swapnil-raj commented 1 year ago

SIGQUIT restarted the process with message command terminated with exit code 137 Terminal will auto-close in 15 seconds ... There were no other log messages. Does this mean memory issues? currently we have given 8Gi to each worker.

irenarindos commented 1 year ago

Could you share your controller logs from the time that the worker became unresponsive?

When the worker becomes unresponsive, have you tried just restarting the worker process instead of the pod? Does restarting just the worker resolve the issue?

Thanks so much!

swapnil-raj commented 1 year ago

When I sent SIGQUIT it restarted the process and not the pod and the problem was resolved. Here are controller logs from that time.

  |   | 2022-11-30 22:37:40 | message="{\"id\":\"k5tDV3yz7T\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":16.071746,\"request_info\":{\"id\":\"gtraceid_qk3s3Yg7Se8CZzacJjLS\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_LDBbmp1Bpc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:07:40.565714459Z\",\"status\":200,\"stop\":\"2022-11-30T17:07:40.58178612Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:07:40.581848936Z\"}"
  |   | 2022-11-30 22:38:01 | message="{\"id\":\"jdhLz9p6Sd\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.193444,\"request_info\":{\"id\":\"gtraceid_wXdlkyEFALBjGeBaKdRC\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_Dlx769Wl9P%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:08:01.676891657Z\",\"status\":204,\"stop\":\"2022-11-30T17:08:01.677085115Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:08:01.677118961Z\"}"
  |   | 2022-11-30 22:38:01 | message="{\"id\":\"EQTp7iEfeJ\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":21.861318,\"request_info\":{\"id\":\"gtraceid_oUmettSl35SnztC6Ka0m\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_Dlx769Wl9P%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_GQ0xCHud6e\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:08:01.694166654Z\",\"status\":200,\"stop\":\"2022-11-30T17:08:01.716027987Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:08:01.716082136Z\"}"
  |   | 2022-11-30 22:38:01 | message="{\"id\":\"UEnNnLswdD\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.186007,\"request_info\":{\"id\":\"gtraceid_BgTbB4HJ0EfzgpgFv6UZ\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:08:01.738284336Z\",\"status\":204,\"stop\":\"2022-11-30T17:08:01.738470339Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:08:01.738506154Z\"}"
  |   | 2022-11-30 22:38:01 | message="{\"id\":\"s5zlIZUV80\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":30.964452,\"request_info\":{\"id\":\"gtraceid_WXtWtA4BlEzKpR2aDvPv\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_GQ0xCHud6e\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:08:01.76916419Z\",\"status\":200,\"stop\":\"2022-11-30T17:08:01.800128638Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:08:01.800200316Z\"}"
  |   | 2022-11-30 22:39:46 | message="{\"id\":\"YlUyuUrCZe\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":23.20398,\"request_info\":{\"id\":\"gtraceid_qtw6WQ3uPKTvxpPGax8k\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dd2RgmKlPr%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_MFrmUkeB2t\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:09:46.341341959Z\",\"status\":200,\"stop\":\"2022-11-30T17:09:46.364545922Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:09:46.364572679Z\"}"
  |   | 2022-11-30 22:39:46 | message="{\"id\":\"ofYXNOgbll\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":35.155973,\"request_info\":{\"id\":\"gtraceid_JfVhBVP9lIGIOPLGqtb2\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_MFrmUkeB2t\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:09:46.661105225Z\",\"status\":200,\"stop\":\"2022-11-30T17:09:46.696261203Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:09:46.696294626Z\"}"
  |   | 2022-11-30 22:39:56 | message="{\"id\":\"zgoohPEWNq\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":16.793008,\"request_info\":{\"id\":\"gtraceid_TX1N0CfUWZVO8EcbDVNQ\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dCNOsA4hZ4%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_998XVtB5wA\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:09:56.172694524Z\",\"status\":200,\"stop\":\"2022-11-30T17:09:56.18948755Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:09:56.189548334Z\"}"
  |   | 2022-11-30 22:39:56 | message="{\"id\":\"1UJHVG2LvW\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.088712,\"request_info\":{\"id\":\"gtraceid_PUyxwsxXDQo18h2Rk6ro\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:09:56.658901062Z\",\"status\":204,\"stop\":\"2022-11-30T17:09:56.658989781Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:09:56.659004609Z\"}"
  |   | 2022-11-30 22:39:56 | message="{\"id\":\"idv2OTF6pN\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":31.6109,\"request_info\":{\"id\":\"gtraceid_SHXyrM5NwuP4j4JaIJUF\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_998XVtB5wA\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:09:56.905497101Z\",\"status\":200,\"stop\":\"2022-11-30T17:09:56.937107985Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:09:56.937165945Z\"}"
  |   | 2022-11-30 22:40:07 | message="{\"id\":\"1T85DkQj5y\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"error\",\"data\":{\"error\":\"session.(Repository).LookupSession: failed for s_ocdrV4Vmbe: db.LookupById: record not found, search issue: error #1100: dbw.LookupById: record not found\",\"error_fields\":{\"Code\":1100,\"Msg\":\"failed for s_ocdrV4Vmbe\",\"Op\":\"session.(Repository).LookupSession\",\"Wrapped\":{\"Code\":1100,\"Msg\":\"\",\"Op\":\"db.LookupById\",\"Wrapped\":{}}},\"id\":\"e_OujQQVarCp\",\"version\":\"v0.1\",\"op\":\"session.(Repository).LookupSession\",\"request_info\":{\"id\":\"gtraceid_t7QbLIFBijK3DDnE3XgX\",\"method\":\"/controller.servers.services.v1.SessionService/LookupSession\"}},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:10:07.001086009Z\"}"
  |   | 2022-11-30 22:40:32 | message="{\"id\":\"xQLgIbMLd4\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":277.28068,\"request_info\":{\"id\":\"gtraceid_gLAsaLfBOorA4gtx1wIS\",\"method\":\"POST\",\"path\":\"/v1/auth-methods/ampw_8WPopPNLON:authenticate\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:10:31.762658538Z\",\"status\":200,\"stop\":\"2022-11-30T17:10:32.039939227Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:10:32.040016252Z\"}"
  |   | 2022-11-30 22:40:36 | message="{\"id\":\"F2kWAFBBLr\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":20.615879,\"request_info\":{\"id\":\"gtraceid_ykKkW4g4PFdnApoPl0s0\",\"method\":\"POST\",\"path\":\"/v1/targets/o34-k8s-api:authorize-session\",\"public_id\":\"at_PAKhmVLKG4\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:10:36.869607409Z\",\"status\":200,\"stop\":\"2022-11-30T17:10:36.890223247Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:10:36.890284356Z\"}"
  |   | 2022-11-30 22:42:40 | message="{\"id\":\"uVhQPCWoR7\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.213519,\"request_info\":{\"id\":\"gtraceid_gRJvphNnN0T0BWaMzf6e\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_kve2TEymQ5%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:12:40.695076802Z\",\"status\":204,\"stop\":\"2022-11-30T17:12:40.695290346Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:12:40.695325191Z\"}"
  |   | 2022-11-30 22:42:40 | message="{\"id\":\"eoVv8BWD0E\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":10.855275,\"request_info\":{\"id\":\"gtraceid_pUB2U0nHAaCYuKq3xfmw\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_kve2TEymQ5%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_mChZpUtfcN\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:12:40.982619744Z\",\"status\":200,\"stop\":\"2022-11-30T17:12:40.993474967Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:12:40.993532797Z\"}"
  |   | 2022-11-30 22:42:41 | message="{\"id\":\"bwoorZCXoD\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.186651,\"request_info\":{\"id\":\"gtraceid_Na8NXzmNzXDVaUbCwlXF\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:12:41.301985279Z\",\"status\":204,\"stop\":\"2022-11-30T17:12:41.302171952Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:12:41.302207177Z\"}"
  |   | 2022-11-30 22:42:41 | message="{\"id\":\"tNEDikRkoO\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":14.618589,\"request_info\":{\"id\":\"gtraceid_DDCniXsEnLyotwSUqHRi\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_mChZpUtfcN\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:12:41.574164456Z\",\"status\":200,\"stop\":\"2022-11-30T17:12:41.588783047Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:12:41.588842143Z\"}"
  |   | 2022-11-30 22:43:02 | message="{\"id\":\"83vOr0qDnX\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.186715,\"request_info\":{\"id\":\"gtraceid_8cB1Itkv1tNFlEJ6JzTB\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_Dlx769Wl9P%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:02.660521479Z\",\"status\":204,\"stop\":\"2022-11-30T17:13:02.660708212Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:02.660742103Z\"}"
  |   | 2022-11-30 22:43:02 | message="{\"id\":\"9IDXNszu55\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":14.894401,\"request_info\":{\"id\":\"gtraceid_ZNTnX61ndYR8oelwqb2M\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_Dlx769Wl9P%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_GQ0xCHud6e\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:02.677861066Z\",\"status\":200,\"stop\":\"2022-11-30T17:13:02.692755483Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:02.692809316Z\"}"
  |   | 2022-11-30 22:43:02 | message="{\"id\":\"66jYgaeq3Y\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.181621,\"request_info\":{\"id\":\"gtraceid_Or1IWoYJJs73wJdsQ12I\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:02.717649837Z\",\"status\":204,\"stop\":\"2022-11-30T17:13:02.717831459Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:02.717877095Z\"}"
  |   | 2022-11-30 22:43:02 | message="{\"id\":\"iOG1qW5yaI\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":28.505727,\"request_info\":{\"id\":\"gtraceid_3fY59uyGvy4WzmWFjyJY\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_GQ0xCHud6e\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:02.744133615Z\",\"status\":200,\"stop\":\"2022-11-30T17:13:02.772639361Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:02.772679946Z\"}"
  |   | 2022-11-30 22:43:54 | message="{\"id\":\"06RofMQta4\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.127509,\"request_info\":{\"id\":\"gtraceid_7kp6qQHkcBg7PRGWhq16\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_7HO51SDKhs%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:54.869532729Z\",\"status\":204,\"stop\":\"2022-11-30T17:13:54.869660239Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:54.869683137Z\"}"
  |   | 2022-11-30 22:43:54 | message="{\"id\":\"zu9llvaHw3\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":21.76778,\"request_info\":{\"id\":\"gtraceid_Ou4Owu2y1fskx2euyBF6\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_7HO51SDKhs%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_LDBbmp1Bpc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:54.897193115Z\",\"status\":200,\"stop\":\"2022-11-30T17:13:54.918960901Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:54.91900595Z\"}"
  |   | 2022-11-30 22:43:54 | message="{\"id\":\"oVU1p2Acha\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.203093,\"request_info\":{\"id\":\"gtraceid_QvUq0cQN2afbaIhvtiiS\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:54.964095754Z\",\"status\":204,\"stop\":\"2022-11-30T17:13:54.964298863Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:54.964334906Z\"}"
  |   | 2022-11-30 22:43:55 | message="{\"id\":\"Xci4aX0GKg\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":37.001864,\"request_info\":{\"id\":\"gtraceid_ub5ldcvwOfJ2Rb6uZl2u\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_LDBbmp1Bpc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:13:54.989233213Z\",\"status\":200,\"stop\":\"2022-11-30T17:13:55.026235075Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:13:55.026301841Z\"}"
  |   | 2022-11-30 22:44:46 | message="{\"id\":\"0rR39OSYBL\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.178123,\"request_info\":{\"id\":\"gtraceid_m74dYbqxRrCikwUpVhGL\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dd2RgmKlPr%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:14:46.271878752Z\",\"status\":204,\"stop\":\"2022-11-30T17:14:46.272056876Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:14:46.272096448Z\"}"
  |   | 2022-11-30 22:44:46 | message="{\"id\":\"gyVXLs1H7h\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":17.331376,\"request_info\":{\"id\":\"gtraceid_oszxbuPfNvkLcrR9mhQ7\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dd2RgmKlPr%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_MFrmUkeB2t\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:14:46.603180998Z\",\"status\":200,\"stop\":\"2022-11-30T17:14:46.620512372Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:14:46.620544001Z\"}"
  |   | 2022-11-30 22:44:47 | message="{\"id\":\"1thgFC8ZKh\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.178211,\"request_info\":{\"id\":\"gtraceid_6mEAwiEzSHmWYQtAloLq\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:14:47.554372847Z\",\"status\":204,\"stop\":\"2022-11-30T17:14:47.554551058Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:14:47.55458617Z\"}"
  |   | 2022-11-30 22:44:48 | message="{\"id\":\"bJQcYnnIdb\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":47.722726,\"request_info\":{\"id\":\"gtraceid_SIQZ05YHxYFozDls8ugg\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_MFrmUkeB2t\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:14:48.059719106Z\",\"status\":200,\"stop\":\"2022-11-30T17:14:48.107441806Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:14:48.107510493Z\"}"
  |   | 2022-11-30 22:45:55 | message="{\"id\":\"WRjziR6nZH\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.087881,\"request_info\":{\"id\":\"gtraceid_eU8NZNZYw8QPjHZjrav8\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dCNOsA4hZ4%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:15:55.178757642Z\",\"status\":204,\"stop\":\"2022-11-30T17:15:55.178845538Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:15:55.178861928Z\"}"
  |   | 2022-11-30 22:45:55 | message="{\"id\":\"4FQOXZHgOw\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":13.564322,\"request_info\":{\"id\":\"gtraceid_xHDY1KodW06vBwAHDTea\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dCNOsA4hZ4%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_998XVtB5wA\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:15:55.663527028Z\",\"status\":200,\"stop\":\"2022-11-30T17:15:55.677091345Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:15:55.677113654Z\"}"
  |   | 2022-11-30 22:45:55 | message="{\"id\":\"F4pqEEXsjE\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.170048,\"request_info\":{\"id\":\"gtraceid_eqnsAh492RMBhLtDR9bs\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:15:55.931807727Z\",\"status\":204,\"stop\":\"2022-11-30T17:15:55.931977792Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:15:55.932012724Z\"}"
  |   | 2022-11-30 22:45:56 | message="{\"id\":\"XOcujmqsDx\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":29.119665,\"request_info\":{\"id\":\"gtraceid_vYjer7LxYZI1ih9HX7bL\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_998XVtB5wA\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:15:56.176194347Z\",\"status\":200,\"stop\":\"2022-11-30T17:15:56.205313972Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:15:56.20537442Z\"}"
  |   | 2022-11-30 22:47:08 | message="{\"id\":\"4cTnL0T9iX\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":19.913729,\"request_info\":{\"id\":\"gtraceid_MgCwUPeyLestGY4TwbMa\",\"method\":\"GET\",\"path\":\"/v1/host-catalogs?scope_id=p_AOrrXR4GGp\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:08.452293256Z\",\"status\":200,\"stop\":\"2022-11-30T17:17:08.472206987Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:08.472265334Z\"}"
  |   | 2022-11-30 22:47:14 | message="{\"id\":\"zZ8SLcLUTi\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":12.132358,\"request_info\":{\"id\":\"gtraceid_skeN8lmdryyj09RrRXXO\",\"method\":\"GET\",\"path\":\"/v1/host-catalogs/hcst_VFGukIXHL4\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:14.694670155Z\",\"status\":200,\"stop\":\"2022-11-30T17:17:14.70680253Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:14.706852348Z\"}"
  |   | 2022-11-30 22:47:22 | message="{\"id\":\"Bl2er0gVYP\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":13.802453,\"request_info\":{\"id\":\"gtraceid_HTx0RkGPRjELt7SN1LcU\",\"method\":\"GET\",\"path\":\"/v1/hosts?host_catalog_id=hcst_VFGukIXHL4\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:22.52541257Z\",\"status\":200,\"stop\":\"2022-11-30T17:17:22.539215021Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:22.539273229Z\"}"
  |   | 2022-11-30 22:47:41 | message="{\"id\":\"vDlCv17xb3\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.193128,\"request_info\":{\"id\":\"gtraceid_StjgezNeswCAwaLVFPKg\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_kve2TEymQ5%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:41.700485129Z\",\"status\":204,\"stop\":\"2022-11-30T17:17:41.700678258Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:41.700715853Z\"}"
  |   | 2022-11-30 22:47:41 | message="{\"id\":\"Z0ukNi7xi3\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":10.095222,\"request_info\":{\"id\":\"gtraceid_Jxrji5yQvq9efvuue128\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_kve2TEymQ5%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_mChZpUtfcN\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:41.982606683Z\",\"status\":200,\"stop\":\"2022-11-30T17:17:41.992701921Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:41.992751287Z\"}"
  |   | 2022-11-30 22:47:42 | message="{\"id\":\"cJSPkMbfhE\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.178491,\"request_info\":{\"id\":\"gtraceid_dZr1Ds4yvvbsGSJd0jDE\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:42.284845793Z\",\"status\":204,\"stop\":\"2022-11-30T17:17:42.285024286Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:42.285057399Z\"}"
  |   | 2022-11-30 22:47:42 | message="{\"id\":\"8r0UjTOcW1\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":12.605599,\"request_info\":{\"id\":\"gtraceid_h2m3ysbgkJEsKJSC1pdp\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_mChZpUtfcN\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:17:42.560059691Z\",\"status\":200,\"stop\":\"2022-11-30T17:17:42.572665265Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:17:42.572725615Z\"}"
  |   | 2022-11-30 22:48:03 | message="{\"id\":\"oWT9oWmSC6\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.206373,\"request_info\":{\"id\":\"gtraceid_eYcuWvPt5lTztupVlXzH\",\"method\":\"OPTIONS\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_Dlx769Wl9P%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:18:03.657749186Z\",\"status\":204,\"stop\":\"2022-11-30T17:18:03.657955576Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:18:03.657992179Z\"}"
  |   | 2022-11-30 22:48:03 | message="{\"id\":\"n97cWD28v6\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":24.593202,\"request_info\":{\"id\":\"gtraceid_VE1Z29SCQKOAeIfhHx70\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_Dlx769Wl9P%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_GQ0xCHud6e\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:18:03.674469908Z\",\"status\":200,\"stop\":\"2022-11-30T17:18:03.699063109Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:18:03.699115943Z\"}"
  |   | 2022-11-30 22:48:03 | message="{\"id\":\"ERFlqYfEfZ\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":0.161921,\"request_info\":{\"id\":\"gtraceid_on6re7a4ZEsnG0cGuR4W\",\"method\":\"OPTIONS\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:18:03.723227426Z\",\"status\":204,\"stop\":\"2022-11-30T17:18:03.723389352Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:18:03.723425291Z\"}"
  |   | 2022-11-30 22:48:03 | message="{\"id\":\"SMyR4WKtJ3\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":33.049961,\"request_info\":{\"id\":\"gtraceid_ImvFYab4VILKDPG44gdp\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_GQ0xCHud6e\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:18:03.754138803Z\",\"status\":200,\"stop\":\"2022-11-30T17:18:03.787188755Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:18:03.787263685Z\"}"
  |   | 2022-11-30 22:49:43 | message="{\"id\":\"qtxtkhIWiU\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":23.809616,\"request_info\":{\"id\":\"gtraceid_YDwZMqpi5zVLUr3IJuXn\",\"method\":\"GET\",\"path\":\"/v1/host-catalogs/hcst_IbA5yLJFxA\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:19:43.439336888Z\",\"status\":200,\"stop\":\"2022-11-30T17:19:43.463146504Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:19:43.463201643Z\"}"
  |   | 2022-11-30 22:49:44 | message="{\"id\":\"ouRCRslSbh\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":11.36696,\"request_info\":{\"id\":\"gtraceid_i9C1RitE1zHp9aYIImEm\",\"method\":\"GET\",\"path\":\"/v1/hosts?host_catalog_id=hcst_IbA5yLJFxA\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:19:44.803987644Z\",\"status\":200,\"stop\":\"2022-11-30T17:19:44.815354518Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:19:44.815415269Z\"}"
  |   | 2022-11-30 22:49:46 | message="{\"id\":\"PwV0z4ZyMP\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":27.466945,\"request_info\":{\"id\":\"gtraceid_V7PmoEaUC5mUPW08fKjv\",\"method\":\"GET\",\"path\":\"/v1/sessions?filter=(%22%2Fitem%2Fuser_id%22+%3D%3D+%22u_dd2RgmKlPr%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_MFrmUkeB2t\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:19:46.315203622Z\",\"status\":200,\"stop\":\"2022-11-30T17:19:46.342670537Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:19:46.34271381Z\"}"
  |   | 2022-11-30 22:49:46 | message="{\"id\":\"EMAjWwMZG3\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":44.294522,\"request_info\":{\"id\":\"gtraceid_vb1jzJ3VzcY2diYqMIQo\",\"method\":\"GET\",\"path\":\"/v1/targets?filter=(%22authorize-session%22+in+%22%2Fitem%2Fauthorized_actions%22)\\u0026recursive=true\\u0026scope_id=o_hCCGpdJlDc\",\"public_id\":\"at_MFrmUkeB2t\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:19:46.652937421Z\",\"status\":200,\"stop\":\"2022-11-30T17:19:46.69723189Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:19:46.697370468Z\"}"
  |   | 2022-11-30 22:49:47 | message="{\"id\":\"z4WS3wcF7w\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":9.399842,\"request_info\":{\"id\":\"gtraceid_MP8ehnzqTmTPJJjnc3E0\",\"method\":\"GET\",\"path\":\"/v1/hosts/hst_hrlhR0wzDA\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:19:47.503170285Z\",\"status\":200,\"stop\":\"2022-11-30T17:19:47.512570087Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:19:47.512624679Z\"}"
  |   | 2022-11-30 22:50:11 | message="{\"id\":\"RndTXRPOGh\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-vcmmv/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":27.296836,\"request_info\":{\"id\":\"gtraceid_bgOAYf8gko21qF4HR3z0\",\"method\":\"PATCH\",\"path\":\"/v1/hosts/hst_hrlhR0wzDA\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:20:11.728170483Z\",\"status\":200,\"stop\":\"2022-11-30T17:20:11.755467269Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:20:11.755524497Z\"}"
  |   | 2022-11-30 22:50:12 | message="{\"id\":\"l0XpdF0eKk\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":9.444668,\"request_info\":{\"id\":\"gtraceid_1B8v0n9DLmYglR1GgrCb\",\"method\":\"GET\",\"path\":\"/v1/hosts?host_catalog_id=hcst_IbA5yLJFxA\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:20:12.013435268Z\",\"status\":200,\"stop\":\"2022-11-30T17:20:12.022879936Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:20:12.022931168Z\"}"
  |   | 2022-11-30 22:50:12 | message="{\"id\":\"pvQ5OW3pIp\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":7.265387,\"request_info\":{\"id\":\"gtraceid_DT1m7m08AVfjijv9MjAn\",\"method\":\"GET\",\"path\":\"/v1/hosts/hst_hrlhR0wzDA\",\"public_id\":\"at_SYpdX0pM8x\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:20:12.274214233Z\",\"status\":200,\"stop\":\"2022-11-30T17:20:12.281479569Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:20:12.28153017Z\"}"
  |   | 2022-11-30 22:50:29 | message="{\"id\":\"XVYktiMWFt\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-v67xq/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":279.918281,\"request_info\":{\"id\":\"gtraceid_X1Q1BdrqwM3Jrn0y3EEv\",\"method\":\"POST\",\"path\":\"/v1/auth-methods/ampw_8WPopPNLON:authenticate\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:20:29.270623354Z\",\"status\":200,\"stop\":\"2022-11-30T17:20:29.550541588Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:20:29.550601683Z\"}"
  |   | 2022-11-30 22:50:51 | message="{\"id\":\"0z8Fo6RNGx\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":78.585382,\"request_info\":{\"id\":\"gtraceid_b5r89EVyRlBvMFItl0C4\",\"method\":\"POST\",\"path\":\"/v1/targets/o35-k8s-api:authorize-session\",\"public_id\":\"at_o3WexOW85R\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:20:51.021383049Z\",\"status\":200,\"stop\":\"2022-11-30T17:20:51.099968472Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:20:51.100008159Z\"}"
  |   | 2022-11-30 22:51:17 | message="{\"id\":\"4GUHdanUIO\",\"source\":\"https://hashicorp.com/boundary/boundary-controller-58d64/controller\",\"specversion\":\"1.0\",\"type\":\"observation\",\"data\":{\"latency-ms\":18.984846,\"request_info\":{\"id\":\"gtraceid_Qrn6sNQEOOKrK6G5N42N\",\"method\":\"POST\",\"path\":\"/v1/targets/o35-k8s-api:authorize-session\",\"public_id\":\"at_o3WexOW85R\",\"client_ip\":\"192.168.170.84\"},\"start\":\"2022-11-30T17:21:17.477506534Z\",\"status\":200,\"stop\":\"2022-11-30T17:21:17.496491333Z\",\"version\":\"v0.1\"},\"datacontentype\":\"application/cloudevents\",\"time\":\"2022-11-30T17:21:17.496544401Z\"}"

`
jefferai commented 1 year ago

SIGQUIT will stop the process -- it sounds like you had something autostart it again. The logs we need are not the observation logs but the ones that will go out to either stdout or stderr when SIGQUIT is sent -- it's a stack trace.

swapnil-raj commented 1 year ago

@jefferai , yes we are on k8s so i think that restarted it. There was just restarting log in STDOUT. Is there any other file which might have logs that you need?

irenarindos commented 1 year ago

This should be fixed by https://github.com/hashicorp/boundary/pull/3231

swapnil-raj commented 1 year ago

The issue is not resolved. Even after the upgrade :((