ConcurrentModificationException on WS message handling

andrewazores commented 4 years ago

I attempted to request a recording via the container-jfr-operator CR API and found this in the container-jfr logs:

[INFO] (10.128.0.1:56640): GET /clienturl 200 0ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:56766): GET /clienturl 200 0ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:56894): GET /clienturl 200 1ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:57020): GET /clienturl 200 1ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:57392): GET /clienturl 200 0ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:57856): GET /clienturl 200 0ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:58252): GET /clienturl 200 0ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:58378): GET /clienturl 200 0ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:58504): GET /clienturl 200 1ms
[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.1.99:35886): GET /clienturl 200 0ms
[INFO] Connected remote client 10.128.0.1:58582
[INFO] (10.128.0.1:58582): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:58582): CMD {"command":"connect","args":["172.30.181.199:9091"]}

[INFO] No active connection

[INFO] (10.128.0.1:58582): CMD {"command":"dump","args":["apimade","30","ALL"]}

[INFO] (10.128.0.1:56504): CMD {"command":"list-event-types","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58582): CMD {"command":"list","args":null}

[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58582): CMD {"command":"disconnect","args":null}

[INFO] (10.128.1.99:35886): GET /clienturl 200 0ms
[INFO] Disconnected remote client 10.128.0.1:58582
[INFO] Connected remote client 10.128.0.1:58644
[INFO] (10.128.0.1:58644): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:58644): CMD {"command":"disconnect","args":null}

[INFO] (10.128.0.1:58644): CMD {"command":"connect","args":["172.30.181.199:9091"]}

[INFO] No active connection

[INFO] (10.128.0.1:58644): CMD {"command":"dump","args":["apimade","30","ALL"]}

[INFO] (10.128.0.1:56504): CMD {"command":"list-event-types","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list-event-types","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] Disconnected remote client 10.128.0.1:58644
[INFO] (10.128.0.1:58684): GET /clienturl 200 0ms
[INFO] (10.128.1.99:35886): GET /clienturl 200 2ms
[INFO] Connected remote client 10.128.0.1:58690
[INFO] (10.128.0.1:58690): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58690): CMD {"command":"disconnect","args":null}

[INFO] Disconnected remote client 10.128.0.1:58690
[INFO] (10.128.1.99:35886): GET /clienturl 200 3ms
[INFO] Connected remote client 10.128.0.1:58722
[INFO] (10.128.0.1:58722): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:56504): CMD {"command":"ping","args":[]}
[INFO] (10.128.0.1:58722): CMD {"command":"connect","args":["172.30.181.199:9091"]}

[INFO] No active connection

[INFO] (10.128.0.1:58722): CMD {"command":"dump","args":["apimade","30","ALL"]}

[INFO] (10.128.0.1:56504): CMD {"command":"list-event-types","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58722): CMD {"command":"list","args":null}

[INFO] Disconnected remote client 10.128.0.1:58722
[INFO] (10.128.1.99:35886): GET /clienturl 200 0ms
[INFO] Connected remote client 10.128.0.1:58772
[INFO] (10.128.0.1:58772): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58772): CMD {"command":"disconnect","args":null}

[INFO] Disconnected remote client 10.128.0.1:58772
[INFO] (10.128.1.99:35886): GET /clienturl 200 1ms
[INFO] Connected remote client 10.128.0.1:58802
[INFO] (10.128.0.1:58802): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:58802): CMD {"command":"connect","args":["172.30.181.199:9091"]}

[INFO] No active connection

[INFO] (10.128.0.1:56504): CMD {"command":"list-event-types","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58802): CMD {"command":"dump","args":["apimade","30","ALL"]}

[INFO] Disconnected remote client 10.128.0.1:58802
[INFO] (10.128.1.99:35886): GET /clienturl 200 0ms
[INFO] Connected remote client 10.128.0.1:58848
[INFO] (10.128.0.1:58848): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58848): CMD {"command":"disconnect","args":null}

[INFO] Disconnected remote client 10.128.0.1:58848
[INFO] (10.128.1.99:35886): GET /clienturl 200 0ms
[INFO] Connected remote client 10.128.0.1:58872
[INFO] (10.128.0.1:58872): CMD {"command":"is-connected","args":null}

[INFO] (10.128.0.1:58872): CMD {"command":"connect","args":["172.30.181.199:9091"]}

[INFO] No active connection

[INFO] (10.128.0.1:56504): CMD {"command":"list-event-types","args":[]}
[INFO] (10.128.0.1:56504): CMD {"command":"list","args":[]}
[INFO] (10.128.0.1:58872): CMD {"command":"dump","args":["apimade","30","ALL"]}

[INFO] Disconnected remote client 10.128.0.1:58872
Exception in thread "main" java.util.ConcurrentModificationException
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1542)
    at com.redhat.rhjmc.containerjfr.tui.ws.MessagingServer.flush(MessagingServer.java:115)
    at com.redhat.rhjmc.containerjfr.tui.ws.WsCommandExecutor.flush(WsCommandExecutor.java:119)
    at com.redhat.rhjmc.containerjfr.tui.ws.WsCommandExecutor.run(WsCommandExecutor.java:85)
    at com.redhat.rhjmc.containerjfr.ContainerJfr.main(ContainerJfr.java:69)
[INFO] (10.128.1.99:35886): GET /clienturl 200 0ms

The requested recording was observed to be created in the -web UI, but container-jfr seemed to be broken after this exception was thrown. For example, analysis summaries for the recording could be retrieved, but targets could not be scanned (infinite spinner).

andrewazores commented 4 years ago

Here are the associated operator logs:

{"level":"info","ts":1580397383.4269035,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397383.5797656,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"false"}}
{"level":"info","ts":1580397383.5799122,"logger":"containerjfr_client","msg":"got is-connected response","resp":"false"}
{"level":"info","ts":1580397383.5800374,"logger":"containerjfr_client","msg":"sent command","json":{"command":"connect","args":["172.30.181.199:9091"]}}
{"level":"info","ts":1580397384.06504,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"connect","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397384.0651824,"logger":"containerjfr_client","msg":"got connect response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397384.0652645,"logger":"controller_flightrecorder","msg":"Syncing recording requests for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397384.0653644,"logger":"controller_flightrecorder","msg":"Creating new recording","name":"apimade","duration":"30s","eventOptions":["ALL"]}
{"level":"info","ts":1580397384.0655541,"logger":"containerjfr_client","msg":"sent command","json":{"command":"dump","args":["apimade","30","ALL"]}}
{"level":"info","ts":1580397385.1433392,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"dump","status":0,"payload":null}}
{"level":"info","ts":1580397385.1435227,"logger":"containerjfr_client","msg":"got dump response","resp":""}
{"level":"info","ts":1580397385.14363,"logger":"controller_flightrecorder","msg":"Listing recordings for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397385.1438181,"logger":"containerjfr_client","msg":"sent command","json":{"command":"list","args":null}}
{"level":"info","ts":1580397385.5457828,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"list-event-types","status":0,"payload":[{"id":0,"name":"Security Property Modification","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"TLS Handshake","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"X509 Validation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"X509 Certificate","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Flight Recording","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Recording Setting","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Error","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Exception Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Exception","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Socket Write","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Socket Read","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"File Write","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"File Read","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"File Force","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Start","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread End","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Sleep","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Park","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Monitor Blocked","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Monitor Wait","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Monitor Inflated","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Biased Lock Revocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Biased Lock Self Revocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Biased Lock Class Revocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Reserved Stack Activation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Load","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Define","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Unload","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Int Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Int Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Long Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Long Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Double Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Boolean Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"String Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Heap Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace GC Threshold","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Allocation Failure","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Out of Memory","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Chunk Free List Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Parallel Scavenge Heap Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Heap Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Parallel Old Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Young Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Old Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 MMU Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Evacuation Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Reference Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Object Count after GC","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Evacuation Statistics for Young","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Evacuation Memory Statistics for Old","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Basic IHOP Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Adaptive IHOP Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Promotion in new PLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Promotion outside PLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Promotion Failed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Evacuation Failed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Concurrent Mode Failure","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 1","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 2","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 3","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 4","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Concurrent","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Allocation Requiring GC","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Tenuring Distribution","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Heap Region Type Change","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compilation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compiler Phase","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compilation Failure","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Method Inlining","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Sweep Code Cache","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Cache Full","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Begin","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint State Synchronization","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Wait Blocked","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Cleanup","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Cleanup Task","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint End","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"VM Operation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"JVM Shutdown","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Allocation in new TLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Allocation outside TLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Old Object Sample","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Recording Reason","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Data Loss","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"JVM Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"OS Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Initial System Property","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Initial Environment Variable","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"System Process","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"CPU Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"CPU Time Stamp Counter","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"CPU Load","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread CPU Load","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread Context Switch Rate","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Network Utilization","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Loading Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Loader Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread Allocation Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Physical Memory","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Method Profiling Sample","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Method Profiling Sample Native","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread Dump","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Native Library","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Module Require","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Module Export","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compiler Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compiler Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Cache Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Cache Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Sweeper Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Sweeper Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Int Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Int Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Long Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Long Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Double Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Boolean Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"String Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Object Count","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Heap Region Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Survivor Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"TLAB Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Heap Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Young Generation Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"ZPage Allocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"ZGC Thread Phase","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Z Statistics Counter","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Z Statistics Sampler","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""}]}}
{"level":"info","ts":1580397385.5464346,"logger":"containerjfr_client","msg":"got list response","resp":[{"id":0,"name":"Security Property Modification","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"TLS Handshake","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"X509 Validation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"X509 Certificate","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Flight Recording","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Recording Setting","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Error","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Exception Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Exception","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Socket Write","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Socket Read","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"File Write","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"File Read","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"File Force","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Start","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread End","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Sleep","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Park","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Monitor Blocked","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Monitor Wait","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Monitor Inflated","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Biased Lock Revocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Biased Lock Self Revocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Biased Lock Class Revocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Reserved Stack Activation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Load","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Define","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Unload","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Int Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Int Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Long Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Long Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Double Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Boolean Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"String Flag Changed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Heap Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace GC Threshold","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Allocation Failure","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Out of Memory","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Metaspace Chunk Free List Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Parallel Scavenge Heap Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Heap Summary","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Parallel Old Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Young Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Old Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Garbage Collection","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 MMU Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Evacuation Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Reference Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Object Count after GC","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Evacuation Statistics for Young","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Evacuation Memory Statistics for Old","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Basic IHOP Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Adaptive IHOP Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Promotion in new PLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Promotion outside PLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Promotion Failed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Evacuation Failed","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Concurrent Mode Failure","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 1","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 2","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 3","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Pause Level 4","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Phase Concurrent","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Allocation Requiring GC","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Tenuring Distribution","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Heap Region Type Change","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compilation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compiler Phase","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compilation Failure","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Method Inlining","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Sweep Code Cache","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Cache Full","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Begin","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint State Synchronization","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Wait Blocked","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Cleanup","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint Cleanup Task","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Safepoint End","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"VM Operation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"JVM Shutdown","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Allocation in new TLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Allocation outside TLAB","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Old Object Sample","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Recording Reason","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Data Loss","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"JVM Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"OS Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Initial System Property","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Initial Environment Variable","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"System Process","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"CPU Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"CPU Time Stamp Counter","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"CPU Load","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread CPU Load","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread Context Switch Rate","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Network Utilization","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Java Thread Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Loading Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Class Loader Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread Allocation Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Physical Memory","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Method Profiling Sample","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Method Profiling Sample Native","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Thread Dump","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Native Library","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Module Require","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Module Export","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compiler Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Compiler Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Cache Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Cache Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Sweeper Statistics","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Code Sweeper Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Int Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Int Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Long Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Unsigned Long Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Double Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Boolean Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"String Flag","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Object Count","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"G1 Heap Region Information","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Survivor Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"TLAB Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"GC Heap Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Young Generation Configuration","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"ZPage Allocation","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"ZGC Thread Phase","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Z Statistics Counter","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""},{"id":0,"name":"Z Statistics Sampler","state":"","startTime":0,"duration":0,"continuous":false,"toDisk":false,"maxSize":0,"maxAge":0,"downloadUrl":"","reportUrl":""}]}
{"level":"info","ts":1580397385.5473762,"logger":"controller_flightrecorder","msg":"Updating FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr","Namespace":"default","Name":"containerjfr"}
{"level":"info","ts":1580397385.58308,"logger":"controller_flightrecorder","msg":"FlightRecorder successfully updated","Request.Namespace":"default","Request.Name":"containerjfr","Namespace":"default","Name":"containerjfr"}
{"level":"info","ts":1580397385.5832155,"logger":"containerjfr_client","msg":"sent command","json":{"command":"disconnect","args":null}}
{"level":"error","ts":1580397385.5833285,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"disconnect","args":null},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Disconnect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:112\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).disconnectClient\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:337\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:229\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397385.5834384,"logger":"controller_flightrecorder","msg":"failed to disconnect from target JVM","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).disconnectClient\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:339\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:229\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397385.5890143,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397385.5904746,"logger":"controller_service","msg":"Reconciling Service","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397385.596918,"logger":"controller_service","msg":"Skip reconcile: FlightRecorder already exists","Request.Namespace":"default","Request.Name":"containerjfr","Namespace":"default","Name":"containerjfr"}
{"level":"info","ts":1580397385.819401,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397386.117002,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397386.117046,"logger":"containerjfr_client","msg":"got is-connected response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397386.117056,"logger":"containerjfr_client","msg":"already connected, will disconnect first"}
{"level":"info","ts":1580397386.1171649,"logger":"containerjfr_client","msg":"sent command","json":{"command":"disconnect","args":null}}
{"level":"info","ts":1580397386.3233652,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"disconnect","status":0,"payload":null}}
{"level":"info","ts":1580397386.3234267,"logger":"containerjfr_client","msg":"got disconnect response","resp":""}
{"level":"info","ts":1580397386.3235528,"logger":"containerjfr_client","msg":"sent command","json":{"command":"connect","args":["172.30.181.199:9091"]}}
{"level":"info","ts":1580397386.5936754,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"connect","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397386.593782,"logger":"containerjfr_client","msg":"got connect response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397386.593796,"logger":"controller_flightrecorder","msg":"Syncing recording requests for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397386.59381,"logger":"controller_flightrecorder","msg":"Creating new recording","name":"apimade","duration":"30s","eventOptions":["ALL"]}
{"level":"info","ts":1580397386.5939608,"logger":"containerjfr_client","msg":"sent command","json":{"command":"dump","args":["apimade","30","ALL"]}}
{"level":"info","ts":1580397387.200465,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"dump","status":-1,"payload":"Recording with name \"apimade\" already exists"}}
{"level":"error","ts":1580397387.2005465,"logger":"containerjfr_client","msg":"command failed","request":{"command":"dump","args":["apimade","30","ALL"]},"error":"server failed to execute \"dump\": Recording with name \"apimade\" already exists (code -1)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:192\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).DumpRecording\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:136\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:139\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397387.2006216,"logger":"controller_flightrecorder","msg":"failed to create new recording","error":"server failed to execute \"dump\": Recording with name \"apimade\" already exists (code -1)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:141\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397387.2010257,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"server failed to execute \"dump\": Recording with name \"apimade\" already exists (code -1)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397388.2020977,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397388.2342432,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397388.4403822,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397388.440536,"logger":"containerjfr_client","msg":"got is-connected response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397388.44059,"logger":"containerjfr_client","msg":"already connected, will disconnect first"}
{"level":"info","ts":1580397388.4407194,"logger":"containerjfr_client","msg":"sent command","json":{"command":"disconnect","args":null}}
{"level":"error","ts":1580397388.7103586,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"disconnect","args":null},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Disconnect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:112\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Connect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:79\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397388.7105582,"logger":"controller_flightrecorder","msg":"failed to connect to target JVM","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:127\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397388.7108195,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397389.7112095,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397389.7493687,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397390.0453625,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"false"}}
{"level":"info","ts":1580397390.045493,"logger":"containerjfr_client","msg":"got is-connected response","resp":"false"}
{"level":"info","ts":1580397390.0456123,"logger":"containerjfr_client","msg":"sent command","json":{"command":"connect","args":["172.30.181.199:9091"]}}
{"level":"info","ts":1580397390.0473652,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"ping","status":0,"payload":null}}
{"level":"info","ts":1580397390.0474849,"logger":"containerjfr_client","msg":"got connect response","resp":""}
{"level":"info","ts":1580397390.0475352,"logger":"controller_flightrecorder","msg":"Syncing recording requests for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397390.0475771,"logger":"controller_flightrecorder","msg":"Creating new recording","name":"apimade","duration":"30s","eventOptions":["ALL"]}
{"level":"info","ts":1580397390.047706,"logger":"containerjfr_client","msg":"sent command","json":{"command":"dump","args":["apimade","30","ALL"]}}
{"level":"info","ts":1580397390.3124125,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"connect","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397390.3125741,"logger":"containerjfr_client","msg":"got dump response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397390.3126166,"logger":"controller_flightrecorder","msg":"Listing recordings for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397390.3127613,"logger":"containerjfr_client","msg":"sent command","json":{"command":"list","args":null}}
{"level":"info","ts":1580397390.6440794,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"dump","status":-1,"payload":"Recording with name \"apimade\" already exists"}}
{"level":"error","ts":1580397390.6442297,"logger":"containerjfr_client","msg":"command failed","request":{"command":"list","args":null},"error":"server failed to execute \"dump\": Recording with name \"apimade\" already exists (code -1)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:192\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).ListRecordings\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:124\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:150\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397390.6443605,"logger":"controller_flightrecorder","msg":"failed to list flight recordings","error":"server failed to execute \"dump\": Recording with name \"apimade\" already exists (code -1)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:152\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397390.6446018,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"server failed to execute \"dump\": Recording with name \"apimade\" already exists (code -1)","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397391.6449308,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397391.6675565,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397391.8736918,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397391.8738444,"logger":"containerjfr_client","msg":"got is-connected response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397391.8738961,"logger":"containerjfr_client","msg":"already connected, will disconnect first"}
{"level":"info","ts":1580397391.8740175,"logger":"containerjfr_client","msg":"sent command","json":{"command":"disconnect","args":null}}
{"level":"error","ts":1580397392.150301,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"disconnect","args":null},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Disconnect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:112\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Connect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:79\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397392.1505692,"logger":"controller_flightrecorder","msg":"failed to connect to target JVM","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:127\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397392.1508114,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397393.1512022,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397393.1780367,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397393.3651586,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"false"}}
{"level":"info","ts":1580397393.3653824,"logger":"containerjfr_client","msg":"got is-connected response","resp":"false"}
{"level":"info","ts":1580397393.3655598,"logger":"containerjfr_client","msg":"sent command","json":{"command":"connect","args":["172.30.181.199:9091"]}}
{"level":"info","ts":1580397393.6104245,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"connect","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397393.6105733,"logger":"containerjfr_client","msg":"got connect response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397393.610646,"logger":"controller_flightrecorder","msg":"Syncing recording requests for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397393.6107054,"logger":"controller_flightrecorder","msg":"Creating new recording","name":"apimade","duration":"30s","eventOptions":["ALL"]}
{"level":"info","ts":1580397393.6108546,"logger":"containerjfr_client","msg":"sent command","json":{"command":"dump","args":["apimade","30","ALL"]}}
{"level":"error","ts":1580397393.9772408,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"dump","args":["apimade","30","ALL"]},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).DumpRecording\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:136\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:139\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397393.978092,"logger":"controller_flightrecorder","msg":"failed to create new recording","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:141\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397393.9810128,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397394.98159,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397395.0099022,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397395.0843518,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397395.084515,"logger":"containerjfr_client","msg":"got is-connected response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397395.084555,"logger":"containerjfr_client","msg":"already connected, will disconnect first"}
{"level":"info","ts":1580397395.0846727,"logger":"containerjfr_client","msg":"sent command","json":{"command":"disconnect","args":null}}
{"level":"error","ts":1580397395.311363,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"disconnect","args":null},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Disconnect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:112\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).Connect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:79\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397395.3115947,"logger":"controller_flightrecorder","msg":"failed to connect to target JVM","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:127\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397395.3118715,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397396.3128097,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397396.3345609,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}
{"level":"info","ts":1580397396.4201334,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"is-connected","status":0,"payload":"false"}}
{"level":"info","ts":1580397396.4201827,"logger":"containerjfr_client","msg":"got is-connected response","resp":"false"}
{"level":"info","ts":1580397396.420393,"logger":"containerjfr_client","msg":"sent command","json":{"command":"connect","args":["172.30.181.199:9091"]}}
{"level":"info","ts":1580397396.655352,"logger":"containerjfr_client","msg":"got response","resp":{"commandName":"connect","status":0,"payload":"172.30.181.199:9091"}}
{"level":"info","ts":1580397396.6555955,"logger":"containerjfr_client","msg":"got connect response","resp":"172.30.181.199:9091"}
{"level":"info","ts":1580397396.6557114,"logger":"controller_flightrecorder","msg":"Syncing recording requests for service","service":"containerjfr","namespace":"default","host":"172.30.181.199","port":9091}
{"level":"info","ts":1580397396.655801,"logger":"controller_flightrecorder","msg":"Creating new recording","name":"apimade","duration":"30s","eventOptions":["ALL"]}
{"level":"info","ts":1580397396.6561186,"logger":"containerjfr_client","msg":"sent command","json":{"command":"dump","args":["apimade","30","ALL"]}}
{"level":"error","ts":1580397397.0298975,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"dump","args":["apimade","30","ALL"]},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(*ContainerJfrClient).DumpRecording\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:136\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:139\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397397.030092,"logger":"controller_flightrecorder","msg":"failed to create new recording","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(*ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:141\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1580397397.0305116,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"flightrecorder-controller","request":"default/containerjfr","error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1580397398.0309086,"logger":"controller_flightrecorder","msg":"Reconciling FlightRecorder","Request.Namespace":"default","Request.Name":"containerjfr"}
{"level":"info","ts":1580397398.0779355,"logger":"containerjfr_client","msg":"sent command","json":{"command":"is-connected","args":null}}

ebaron commented 4 years ago

{"level":"error","ts":1580397388.7103586,"logger":"containerjfr_client","msg":"could not read response","message":{"command":"disconnect","args":null},"error":"json: cannot unmarshal array into Go struct field responseMessageAlias.payload of type string","stacktrace":"github.com/go-logr/zapr.(zapLogger).Error\n\t/home/andrew/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(ContainerJfrClient).syncMessage\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:181\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(ContainerJfrClient).Disconnect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:112\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/client.(ContainerJfrClient).Connect\n\tcontainer-jfr-operator/pkg/client/containerjfr_client.go:79\ngithub.com/rh-jmc-team/container-jfr-operator/pkg/controller/flightrecorder.(ReconcileFlightRecorder).Reconcile\n\tcontainer-jfr-operator/pkg/controller/flightrecorder/flightrecorder_controller.go:125\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).reconcileHandler\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).processNextWorkItem\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(Controller).worker\n\t/home/andrew/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/andrew/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}

This may be unrelated, but it stood out to me. Is there any way a disconnect response would include an array payload?

andrewazores commented 4 years ago

No, that shouldn't happen:

https://github.com/rh-jmc-team/container-jfr/blob/master/src/main/java/com/redhat/rhjmc/containerjfr/commands/internal/DisconnectCommand.java#L47

Unless there was a very particular threading issue writing into the websocket pipe that resulted in interleaved output and yet a still valid JSON message. Which, given the threading issue present in this bug, might be theoretically possible, but the odds are super low.

ebaron commented 4 years ago

No, that shouldn't happen:

https://github.com/rh-jmc-team/container-jfr/blob/master/src/main/java/com/redhat/rhjmc/containerjfr/commands/internal/DisconnectCommand.java#L47

Unless there was a very particular threading issue writing into the websocket pipe that resulted in interleaved output and yet a still valid JSON message. Which, given the threading issue present in this bug, might be theoretically possible, but the odds are super low.

Perhaps what happened was the requests and responses of the operator's API and the web client becoming interleaved. I could be completely wrong, but it looks like when the WsCommandExecutor processes a command, it sends the reply to all connected clients. In this case, the operator might be getting a response for a command it didn't send.

andrewazores commented 4 years ago

No, that shouldn't happen: https://github.com/rh-jmc-team/container-jfr/blob/master/src/main/java/com/redhat/rhjmc/containerjfr/commands/internal/DisconnectCommand.java#L47 Unless there was a very particular threading issue writing into the websocket pipe that resulted in interleaved output and yet a still valid JSON message. Which, given the threading issue present in this bug, might be theoretically possible, but the odds are super low.

Perhaps what happened was the requests and responses of the operator's API and the web client becoming interleaved. I could be completely wrong, but it looks like when the WsCommandExecutor processes a command, it sends the reply to all connected clients. In this case, the operator might be getting a response for a command it didn't send.

Ah yea, that would be a much more plausible explanation. That's currently by design, because there isn't really much state within the container-jfr instance itself - all of the important state is in JFR in the target JVMs. So it initially made sense to essentially present the API as basically stateless (other than whether a target connection is active), and to ensure that all clients see changes in the target's state by multicasting command responses. But, this does mean that if multiple clients are actively trying to interact with container-jfr that things will be likely to break. Solving this by keeping sessions and more state in container-jfr might be very difficult since at the base level, it only really makes sense if targets can only be altered by one client at a time, so we would need multi-connection ability and then some kind of locking system per client connection.

Maybe a simpler approach would be to for clients put something like a UUID into command messages, which container-jfr can include in the corresponding responses, so clients can see if it's a response to a message they sent or a multicast they're just picking up? Then the clients can decide if it's something they need to act on or not, while still being kept in the loop for any target JVM state changes.

ebaron commented 4 years ago

That seems like a reasonable solution for now. Something like a client/message ID.

cryostatio / cryostat-legacy

ConcurrentModificationException on WS message handling #114