search

Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.86k stars 2.94k forks source link

fuse会无缘无故挂掉，然后无法启动 #10533

Closed we7team closed 3 years ago

we7team commented 4 years ago

Alluxio Version: alluxio/alluxio-fuse:2.1.0

Describe the bug

k8s上pod的状态： image image 可以看出来，master昨天挂了一次，fuse一直在重启，重启了N次都没启动成功。

fuse上的日志： failed to open log file "/var/log/pods/default_alluxio-fuse-r7jcs_5d43128f-1086-11ea-988d-1243c9721bb4/alluxio-fuse/189.log": open /var/log/pods/default_alluxio-fuse-r7jcs_5d43128f-1086-11ea-988d-1243c9721bb4/alluxio-fuse/189.log: no such file or directory

worker上的日志：

2019-11-26T20:52:11.655747383Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:11.655760574Z  ... 5 more
2019-11-26T20:52:11.655763384Z }
2019-11-26T20:52:12.703829153Z Nov 26, 2019 8:52:12 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:12.703859111Z WARNING: [io.grpc.internal.ManagedChannelImpl-2506] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:12.703863066Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:12.703866052Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:12.703868709Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:12.703871269Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:12.703873745Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:12.703876295Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:12.703878733Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:12.703881236Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:12.703883658Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:12.703886064Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:12.703888501Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:12.703891142Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:12.703893636Z  ... 5 more
2019-11-26T20:52:12.703896022Z }
2019-11-26T20:52:13.418022743Z Nov 26, 2019 8:52:13 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:13.418052336Z WARNING: [io.grpc.internal.ManagedChannelImpl-2508] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:13.418056567Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:13.418060119Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:13.418063218Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:13.418066174Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:13.418068981Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:13.418071895Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:13.418074676Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:13.418089445Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:13.418092179Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:13.418094607Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:13.418097068Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:13.418099697Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:13.418102176Z  ... 5 more
2019-11-26T20:52:13.41810465Z }
2019-11-26T20:52:13.514684823Z Nov 26, 2019 8:52:13 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:13.514708253Z WARNING: [io.grpc.internal.ManagedChannelImpl-2510] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:13.514717204Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:13.514720206Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:13.514722844Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:13.514725436Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:13.514727884Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:13.514730358Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:13.514732767Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:13.514735312Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:13.514737767Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:13.514740189Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:13.514742647Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:13.514745258Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:13.514747779Z  ... 5 more
2019-11-26T20:52:13.514750173Z }
2019-11-26T20:52:13.601605582Z Nov 26, 2019 8:52:13 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:13.601630019Z WARNING: [io.grpc.internal.ManagedChannelImpl-2512] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:13.601639189Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:13.601642102Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:13.601644821Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:13.601658041Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:13.601660847Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:13.601663344Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:13.601665785Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:13.601668276Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:13.601670698Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:13.601673271Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:13.601675813Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:13.601678346Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:13.601681049Z  ... 5 more
2019-11-26T20:52:13.601683522Z }
2019-11-26T20:52:14.177667372Z Nov 26, 2019 8:52:14 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:14.177690851Z WARNING: [io.grpc.internal.ManagedChannelImpl-2514] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:14.177694614Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:14.177697553Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:14.177700189Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:14.177702815Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:14.177705401Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:14.177707849Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:14.177710301Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:14.177712801Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:14.177715967Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:14.177718421Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:14.177720862Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:14.177723448Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:14.177726043Z  ... 5 more
2019-11-26T20:52:14.177728417Z }
2019-11-26T20:52:14.557626351Z Nov 26, 2019 8:52:14 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:14.557671426Z WARNING: [io.grpc.internal.ManagedChannelImpl-2516] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:14.557676026Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:14.557679418Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:14.55768236Z   at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:14.557685487Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:14.557688413Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:14.557691479Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:14.557694356Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:14.557697383Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:14.5577003Z    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:14.557703371Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:14.557706301Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:14.55770931Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:14.557712713Z  ... 5 more
2019-11-26T20:52:14.55771556Z }
2019-11-26T20:52:14.794872249Z Nov 26, 2019 8:52:14 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:14.794903553Z WARNING: [io.grpc.internal.ManagedChannelImpl-2518] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:14.794907519Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:14.794910676Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:14.794913306Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:14.794916Z     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:14.794918726Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:14.794921375Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:14.794923932Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:14.794926381Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:14.79492894Z   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:14.794931456Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:14.794946412Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:14.79494923Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:14.794951788Z  ... 5 more
2019-11-26T20:52:14.794954211Z }
2019-11-26T20:52:15.63169709Z Nov 26, 2019 8:52:15 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:15.631720325Z WARNING: [io.grpc.internal.ManagedChannelImpl-2518] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:15.63172393Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:15.631726862Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:15.631729916Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:15.631732467Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:15.631734921Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:15.631737382Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:15.631739815Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:15.63174236Z   at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:15.631744828Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:15.631747239Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:15.631749698Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:15.631752373Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:15.631754869Z  ... 5 more
2019-11-26T20:52:15.631757237Z }
2019-11-26T20:52:16.525782241Z Nov 26, 2019 8:52:16 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:16.525814126Z WARNING: [io.grpc.internal.ManagedChannelImpl-2520] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:16.525818225Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:16.525821409Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:16.525824241Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:16.525826995Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:16.525829699Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:16.525845588Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:16.525848589Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:16.52585112Z   at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:16.52585374Z   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:16.525856374Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:16.525858893Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:16.525861567Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:16.525864189Z  ... 5 more
2019-11-26T20:52:16.525866792Z }
2019-11-26T20:52:16.698748903Z Nov 26, 2019 8:52:16 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:16.698782333Z WARNING: [io.grpc.internal.ManagedChannelImpl-2522] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:16.698786164Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:16.698789153Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:16.698791812Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:16.698794474Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:16.698805826Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:16.698808425Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:16.698810865Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:16.698813486Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:16.69881602Z   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:16.698818737Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:16.698821436Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:16.698824225Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:16.698826743Z  ... 5 more
2019-11-26T20:52:16.698829103Z }
2019-11-26T20:52:16.706926861Z Nov 26, 2019 8:52:16 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:16.706941329Z WARNING: [io.grpc.internal.ManagedChannelImpl-2524] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:16.706955939Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:16.70695889Z   at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:16.706968085Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:16.706970756Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:16.706973242Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:16.70697573Z   at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:16.706978149Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:16.706980654Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:16.706983081Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:16.706985592Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:16.70698803Z   at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:16.706990595Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:16.706993132Z  ... 5 more
2019-11-26T20:52:16.706995605Z }
2019-11-26T20:52:17.37871599Z Nov 26, 2019 8:52:17 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:17.378746902Z WARNING: [io.grpc.internal.ManagedChannelImpl-2526] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:17.378751497Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:17.378755027Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:17.378758211Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:17.378761003Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:17.378763787Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:17.378766465Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:17.378769146Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:17.378771726Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:17.378774392Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:17.378776998Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:17.378779613Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:17.378782279Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:17.378796296Z  ... 5 more
2019-11-26T20:52:17.378799043Z }
2019-11-26T20:52:17.684741284Z Nov 26, 2019 8:52:17 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:17.684769262Z WARNING: [io.grpc.internal.ManagedChannelImpl-2528] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:17.684772963Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:17.684775993Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:17.684778578Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:17.684781089Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:17.684783576Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:17.684786172Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:17.684788623Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:17.684791058Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:17.684793512Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:17.684796063Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:17.684798509Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:17.684801079Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:17.684803691Z  ... 5 more
2019-11-26T20:52:17.684806084Z }
2019-11-26T20:52:17.944891571Z Nov 26, 2019 8:52:17 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:17.944914365Z WARNING: [io.grpc.internal.ManagedChannelImpl-2530] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:17.944924431Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:17.944927669Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:17.944930464Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:17.944933137Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:17.944935922Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:17.944938484Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:17.944941019Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:17.944958616Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:17.94496133Z   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:17.944963797Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:17.944966398Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:17.944968994Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:17.944971522Z  ... 5 more
2019-11-26T20:52:17.944973924Z }
2019-11-26T20:52:18.769747456Z Nov 26, 2019 8:52:18 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:18.769780699Z WARNING: [io.grpc.internal.ManagedChannelImpl-2530] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0: Name does not resolve
2019-11-26T20:52:18.769784865Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:18.769787938Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:18.76979065Z   at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:18.769793269Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:18.769795772Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:18.769798223Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:18.769800814Z Caused by: java.net.UnknownHostException: alluxio-master-0: Name does not resolve
2019-11-26T20:52:18.769803288Z  at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
2019-11-26T20:52:18.769805725Z  at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
2019-11-26T20:52:18.769808834Z  at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
2019-11-26T20:52:18.769811315Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
2019-11-26T20:52:18.769813765Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:18.76981618Z   at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:18.769818716Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:18.76982128Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:18.769823838Z  ... 5 more
2019-11-26T20:52:18.769826288Z }
2019-11-26T20:52:19.55392307Z Nov 26, 2019 8:52:19 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:19.55394604Z WARNING: [io.grpc.internal.ManagedChannelImpl-2532] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:19.553962179Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:19.553965294Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:19.553967927Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:19.55397057Z   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:19.553973056Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:19.553975508Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:19.553977967Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:19.553980462Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:19.553983268Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:19.553985712Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:19.553988146Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:19.553990787Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:19.553993301Z  ... 5 more
2019-11-26T20:52:19.553995681Z }
2019-11-26T20:52:19.731880217Z Nov 26, 2019 8:52:19 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:19.731911028Z WARNING: [io.grpc.internal.ManagedChannelImpl-2534] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:19.731914995Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:19.731918009Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:19.731920689Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:19.731923232Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:19.731925761Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:19.731928251Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:19.731931152Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:19.731933577Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:19.731936058Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:19.731938494Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:19.731941101Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:19.731954751Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:19.731957467Z  ... 5 more
2019-11-26T20:52:19.731959922Z }
2019-11-26T20:52:19.76872127Z Nov 26, 2019 8:52:19 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:19.768753913Z WARNING: [io.grpc.internal.ManagedChannelImpl-2536] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:19.768759135Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:19.768763237Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:19.768767052Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:19.768770472Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:19.768773842Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:19.768777196Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:19.768780559Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:19.768783896Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:19.768787225Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:19.768790523Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:19.768793838Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:19.7687973Z    at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:19.768800682Z  ... 5 more
2019-11-26T20:52:19.768803933Z }
2019-11-26T20:52:20.515298044Z Nov 26, 2019 8:52:20 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:20.51532867Z WARNING: [io.grpc.internal.ManagedChannelImpl-2538] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:20.515332578Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:20.515335933Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:20.515338651Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:20.515341175Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:20.51534363Z   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:20.515346146Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:20.515359361Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:20.515362074Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:20.515364532Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:20.515367026Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:20.515369476Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:20.515372096Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:20.515374639Z  ... 5 more
2019-11-26T20:52:20.515384611Z }
2019-11-26T20:52:20.71229432Z Nov 26, 2019 8:52:20 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:20.712323145Z WARNING: [io.grpc.internal.ManagedChannelImpl-2540] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:20.712327177Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:20.712330286Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:20.712332936Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:20.712335381Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:20.71233799Z   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:20.712340486Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:20.712343021Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:20.712345435Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:20.712347885Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:20.712350393Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:20.712353526Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:20.712356112Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:20.712358689Z  ... 5 more
2019-11-26T20:52:20.712361047Z }
2019-11-26T20:52:21.114814049Z Nov 26, 2019 8:52:21 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:21.114843697Z WARNING: [io.grpc.internal.ManagedChannelImpl-2542] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:21.114847577Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:21.11485127Z   at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:21.114864696Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:21.114867378Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:21.114869975Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:21.114872454Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:21.114874909Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:21.114877438Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:21.114879884Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:21.114882396Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:21.114884853Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:21.114887518Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:21.11489006Z   ... 5 more
2019-11-26T20:52:21.114892544Z }
2019-11-26T20:52:22.173298646Z Nov 26, 2019 8:52:22 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:22.173324146Z WARNING: [io.grpc.internal.ManagedChannelImpl-2542] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:22.17332857Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:22.17333244Z   at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:22.173335571Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:22.173338679Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:22.17334197Z   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:22.17334504Z   at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:22.173347961Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:22.173350973Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:22.173353888Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:22.173356831Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:22.173359853Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:22.173362885Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:22.173365819Z  ... 5 more
2019-11-26T20:52:22.173368726Z }
2019-11-26T20:52:22.670732101Z Nov 26, 2019 8:52:22 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:22.670770257Z WARNING: [io.grpc.internal.ManagedChannelImpl-2544] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:22.670774973Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:22.670778255Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:22.670781241Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:22.670784012Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:22.670789174Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:22.670792202Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:22.670795027Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:22.670797691Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:22.670800618Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:22.670803344Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:22.670806059Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:22.670808967Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:22.670811727Z  ... 5 more
2019-11-26T20:52:22.670814406Z }
2019-11-26T20:52:22.882991236Z Nov 26, 2019 8:52:22 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:22.883022807Z WARNING: [io.grpc.internal.ManagedChannelImpl-2546] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:22.883026698Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:22.883029813Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:22.883032379Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:22.883034843Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:22.883037393Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:22.883039884Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:22.883042331Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:22.88304474Z   at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:22.883047372Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:22.883061425Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:22.883064111Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:22.883066717Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:22.883069289Z  ... 5 more
2019-11-26T20:52:22.883071974Z }
2019-11-26T20:52:23.040834553Z Nov 26, 2019 8:52:23 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:23.040865782Z WARNING: [io.grpc.internal.ManagedChannelImpl-2548] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:23.04087082Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:23.04087464Z   at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:23.040877913Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:23.040881076Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:23.040884437Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:23.040887624Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:23.040890684Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:23.040893699Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:23.040896822Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:23.040899858Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:23.040902902Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:23.040906007Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:23.040909126Z  ... 5 more
2019-11-26T20:52:23.040912113Z }
2019-11-26T20:52:23.557919482Z Nov 26, 2019 8:52:23 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:23.557943545Z WARNING: [io.grpc.internal.ManagedChannelImpl-2550] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:23.557947638Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:23.557950841Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:23.557953678Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:23.557956335Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:23.557971835Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:23.557974681Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:23.557977255Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:23.557979832Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:23.557982496Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:23.557985074Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:23.557987668Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:23.557990324Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:23.557993028Z  ... 5 more
2019-11-26T20:52:23.557995576Z }
2019-11-26T20:52:23.772818829Z Nov 26, 2019 8:52:23 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:23.772849668Z WARNING: [io.grpc.internal.ManagedChannelImpl-2552] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:23.772853438Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:23.772856573Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:23.772859264Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:23.772862142Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:23.772864585Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:23.772867112Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:23.77286953Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:23.772872128Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:23.77287471Z   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:23.772877155Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:23.7728796Z    at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:23.772882185Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:23.772884744Z  ... 5 more
2019-11-26T20:52:23.772888707Z }
2019-11-26T20:52:24.198851732Z Nov 26, 2019 8:52:24 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:24.198877009Z WARNING: [io.grpc.internal.ManagedChannelImpl-2554] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:24.19889654Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:24.198899836Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:24.19890278Z   at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:24.198905508Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:24.198908222Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:24.198910981Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:24.198913681Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:24.198916153Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:24.198918799Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:24.198921579Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:24.198924167Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:24.198926881Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:24.198929498Z  ... 5 more
2019-11-26T20:52:24.19893198Z }
2019-11-26T20:52:25.085131388Z Nov 26, 2019 8:52:25 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:25.085155919Z WARNING: [io.grpc.internal.ManagedChannelImpl-2554] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:25.085160369Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:25.08516384Z   at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:25.0851672Z    at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:25.085170821Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:25.08517389Z   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:25.085176962Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:25.085179754Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:25.085182935Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:25.085185639Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:25.085188353Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:25.085190974Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:25.085219769Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:25.085222885Z  ... 5 more
2019-11-26T20:52:25.085225394Z }
2019-11-26T20:52:25.853743873Z Nov 26, 2019 8:52:25 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:25.853774962Z WARNING: [io.grpc.internal.ManagedChannelImpl-2556] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:25.853779265Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:25.853782605Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:25.853785647Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:25.853788762Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:25.853791757Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:25.853794606Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:25.853797386Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:25.853800201Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:25.853802936Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:25.853805722Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:25.853808531Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:25.85381148Z   at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:25.85381436Z   ... 5 more
2019-11-26T20:52:25.853817173Z }
2019-11-26T20:52:26.12403576Z Nov 26, 2019 8:52:26 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:26.124068153Z WARNING: [io.grpc.internal.ManagedChannelImpl-2558] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:26.124073039Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:26.124076798Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:26.124080578Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:26.124083942Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:26.124086863Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:26.124089584Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:26.124092224Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:26.124112957Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:26.12411568Z   at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:26.124118277Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:26.124120866Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:26.124123619Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:26.12412622Z   ... 5 more
2019-11-26T20:52:26.124128773Z }
2019-11-26T20:52:26.31098046Z Nov 26, 2019 8:52:26 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:26.311009202Z WARNING: [io.grpc.internal.ManagedChannelImpl-2560] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:26.311013913Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:26.311017342Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:26.311020556Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:26.311023546Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:26.311026489Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:26.311029447Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:26.311032471Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:26.311035378Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:26.311038299Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:26.311041179Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:26.311044211Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:26.311047301Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:26.311050281Z  ... 5 more
2019-11-26T20:52:26.31105313Z }
2019-11-26T20:52:26.676933791Z Nov 26, 2019 8:52:26 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:26.676957934Z WARNING: [io.grpc.internal.ManagedChannelImpl-2562] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:26.676962459Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:26.676973228Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:26.676988568Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:26.676991753Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:26.676994538Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:26.676997617Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:26.677000607Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:26.677003706Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:26.677006685Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:26.677009666Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:26.6770127Z    at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:26.677015848Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:26.677018877Z  ... 5 more
2019-11-26T20:52:26.677021755Z }
2019-11-26T20:52:27.071800094Z Nov 26, 2019 8:52:27 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:27.07183006Z WARNING: [io.grpc.internal.ManagedChannelImpl-2564] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:27.071834144Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:27.071837259Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:27.071840358Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:27.071843055Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:27.071845785Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:27.071848511Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:27.071850995Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:27.071853566Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:27.071856025Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:27.071858544Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:27.071861071Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:27.071863768Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:27.071866326Z  ... 5 more
2019-11-26T20:52:27.071868739Z }
2019-11-26T20:52:27.306882527Z Nov 26, 2019 8:52:27 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:27.306927139Z WARNING: [io.grpc.internal.ManagedChannelImpl-2566] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:27.306932123Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:27.306935738Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:27.306938988Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:27.306942183Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:27.30694529Z   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:27.306948418Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:27.306951464Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:27.306954546Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:27.306957641Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:27.306960602Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:27.30696369Z   at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:27.306966592Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:27.306969674Z  ... 5 more
2019-11-26T20:52:27.306972639Z }
2019-11-26T20:52:28.384410036Z Nov 26, 2019 8:52:28 PM io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl onError
2019-11-26T20:52:28.384439671Z WARNING: [io.grpc.internal.ManagedChannelImpl-2566] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host alluxio-master-0, cause=java.lang.RuntimeException: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:28.384444627Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:399)
2019-11-26T20:52:28.384449709Z  at io.grpc.internal.DnsNameResolver$Resolve.resolveInternal(DnsNameResolver.java:269)
2019-11-26T20:52:28.384453534Z  at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:225)
2019-11-26T20:52:28.384456932Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T20:52:28.384460437Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T20:52:28.384467386Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:52:28.384473159Z Caused by: java.net.UnknownHostException: alluxio-master-0
2019-11-26T20:52:28.384476564Z  at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
2019-11-26T20:52:28.384479991Z  at java.net.InetAddress.getAllByName(InetAddress.java:1193)
2019-11-26T20:52:28.384497386Z  at java.net.InetAddress.getAllByName(InetAddress.java:1127)
2019-11-26T20:52:28.384500712Z  at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:624)
2019-11-26T20:52:28.384503918Z  at io.grpc.internal.DnsNameResolver.resolveAll(DnsNameResolver.java:367)
2019-11-26T20:52:28.384507164Z  ... 5 more
2019-11-26T20:52:28.38451008Z }
2019-11-26T20:52:36.240181969Z 2019-11-26 20:52:36,240 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 38202 ms. Longer than the interval 1000
2019-11-26T20:52:36.752449503Z 2019-11-26 20:52:36,752 WARN  SleepingTimer - Worker Block Sync last execution took 38174 ms. Longer than the interval 1000
2019-11-26T20:52:36.898637946Z 2019-11-26 20:52:36,898 WARN  SleepingTimer - Worker Pin List Sync last execution took 38259 ms. Longer than the interval 1000
2019-11-26T20:58:07.065944764Z Nov 26, 2019 8:58:07 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-26T20:58:07.065980483Z WARNING: Stream Error
2019-11-26T20:58:07.065984302Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 63
2019-11-26T20:58:07.065987708Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-26T20:58:07.065991214Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-26T20:58:07.065994355Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-26T20:58:07.065997258Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-26T20:58:07.066000241Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-26T20:58:07.066003089Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-26T20:58:07.066005994Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-26T20:58:07.066008784Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-26T20:58:07.066011701Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-26T20:58:07.066014526Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-26T20:58:07.066017373Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-26T20:58:07.066020188Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-26T20:58:07.066023115Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-26T20:58:07.066025907Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-26T20:58:07.066039797Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:07.066042565Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:07.06604519Z   at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-26T20:58:07.066048768Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-26T20:58:07.066051296Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:07.066053787Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:07.066056378Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-26T20:58:07.066058853Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-26T20:58:07.06606136Z   at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-26T20:58:07.066063875Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-26T20:58:07.066066419Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-26T20:58:07.066068845Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-26T20:58:07.066071371Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:58:07.066073817Z 
2019-11-26T20:58:13.905990464Z Nov 26, 2019 8:58:13 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-26T20:58:13.906015802Z WARNING: Stream Error
2019-11-26T20:58:13.906019008Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 91
2019-11-26T20:58:13.906021865Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-26T20:58:13.906024841Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-26T20:58:13.906027801Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-26T20:58:13.906030428Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-26T20:58:13.906033027Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-26T20:58:13.906035546Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-26T20:58:13.906038146Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-26T20:58:13.906050646Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-26T20:58:13.90605336Z   at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-26T20:58:13.906056124Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-26T20:58:13.906058699Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-26T20:58:13.906061234Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-26T20:58:13.906063782Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-26T20:58:13.906066358Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-26T20:58:13.906068887Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:13.906071453Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:13.906074071Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-26T20:58:13.906077284Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-26T20:58:13.906079833Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:13.906082348Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:13.906084963Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-26T20:58:13.906087474Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-26T20:58:13.90609001Z   at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-26T20:58:13.906092585Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-26T20:58:13.90609517Z   at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-26T20:58:13.906097635Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-26T20:58:13.90610022Z   at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:58:13.906102722Z 
2019-11-26T20:58:19.773289886Z Nov 26, 2019 8:58:19 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-26T20:58:19.773321768Z WARNING: Stream Error
2019-11-26T20:58:19.773325266Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 129
2019-11-26T20:58:19.773328282Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-26T20:58:19.773341697Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-26T20:58:19.77334463Z   at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-26T20:58:19.77334735Z   at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-26T20:58:19.773349942Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-26T20:58:19.773352507Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-26T20:58:19.773355129Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-26T20:58:19.773357602Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-26T20:58:19.773360165Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-26T20:58:19.773362758Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-26T20:58:19.77336527Z   at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-26T20:58:19.773367831Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-26T20:58:19.773370347Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-26T20:58:19.773372824Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-26T20:58:19.773375323Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:19.773378088Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:19.773380668Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-26T20:58:19.773384015Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-26T20:58:19.773386552Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:19.773389133Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:19.77339165Z   at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-26T20:58:19.773394223Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-26T20:58:19.773396795Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-26T20:58:19.773402495Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-26T20:58:19.773405104Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-26T20:58:19.773407531Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-26T20:58:19.773410084Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:58:19.77341251Z 
2019-11-26T20:58:21.907698671Z Nov 26, 2019 8:58:21 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-26T20:58:21.907723182Z WARNING: Stream Error
2019-11-26T20:58:21.907727061Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 143
2019-11-26T20:58:21.907730667Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-26T20:58:21.907734121Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-26T20:58:21.90773746Z   at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-26T20:58:21.907740724Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-26T20:58:21.907743837Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-26T20:58:21.907746945Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-26T20:58:21.907750174Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-26T20:58:21.907753235Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-26T20:58:21.907756611Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-26T20:58:21.907759738Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-26T20:58:21.907762819Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-26T20:58:21.907765871Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-26T20:58:21.907768928Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-26T20:58:21.907772008Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-26T20:58:21.907775064Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:21.907778175Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:21.907781153Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-26T20:58:21.907794324Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-26T20:58:21.907797289Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T20:58:21.90780028Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T20:58:21.907803134Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-26T20:58:21.907806053Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-26T20:58:21.907808993Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-26T20:58:21.907812078Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-26T20:58:21.907814997Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-26T20:58:21.907817918Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-26T20:58:21.907820858Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T20:58:21.90782378Z 
2019-11-26T21:46:58.842729737Z 2019-11-26 21:46:58,841 WARN  SleepingTimer - Worker Pin List Sync last execution took 4676 ms. Longer than the interval 1000
2019-11-26T21:46:58.842758133Z 2019-11-26 21:46:58,842 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 4327 ms. Longer than the interval 1000
2019-11-26T21:46:58.842761943Z 2019-11-26 21:46:58,842 WARN  SleepingTimer - Worker Block Sync last execution took 4804 ms. Longer than the interval 1000
2019-11-26T22:21:29.789319022Z 2019-11-26 22:21:29,788 WARN  SleepingTimer - Worker Pin List Sync last execution took 7777 ms. Longer than the interval 1000
2019-11-26T22:21:29.789348871Z 2019-11-26 22:21:29,788 WARN  SleepingTimer - Worker Block Sync last execution took 7763 ms. Longer than the interval 1000
2019-11-26T22:21:29.789352947Z 2019-11-26 22:21:29,789 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 7764 ms. Longer than the interval 1000
2019-11-26T22:24:41.354843018Z 2019-11-26 22:24:41,353 WARN  SaslStreamServerDriver - Error received for channel: 0522d67f-9cfc-4f9f-8de6-68317d32b85c. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-26T22:24:41.35542755Z 2019-11-26 22:24:41,354 WARN  SaslStreamServerDriver - Error received for channel: b0e069d5-a222-4475-9e83-a54bc91ddff1. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-26T22:25:16.800565843Z 2019-11-26 22:25:16,798 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 34991 ms. Longer than the interval 1000
2019-11-26T22:25:16.839092258Z 2019-11-26 22:25:16,838 WARN  SleepingTimer - Worker Block Sync last execution took 35033 ms. Longer than the interval 1000
2019-11-26T22:25:16.925128827Z 2019-11-26 22:25:16,924 WARN  SleepingTimer - Worker Pin List Sync last execution took 35118 ms. Longer than the interval 1000
2019-11-26T23:30:44.0374018Z Nov 26, 2019 11:30:44 PM io.grpc.netty.NettyServerTransport notifyTerminated
2019-11-26T23:30:44.037433051Z INFO: Transport failed
2019-11-26T23:30:44.037436791Z io.netty.handler.codec.http2.Http2Exception: HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 0300002f2ae00000000000436f6f6b69653a206d73747368
2019-11-26T23:30:44.037440294Z  at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
2019-11-26T23:30:44.037443523Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:318)
2019-11-26T23:30:44.037499564Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:251)
2019-11-26T23:30:44.037504117Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-26T23:30:44.03750713Z   at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-26T23:30:44.037510182Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-26T23:30:44.037513177Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-26T23:30:44.037516095Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T23:30:44.037519444Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T23:30:44.037522336Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-26T23:30:44.037525263Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-26T23:30:44.037528087Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-26T23:30:44.037531033Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-26T23:30:44.03753388Z   at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-26T23:30:44.037536808Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-26T23:30:44.037539684Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-26T23:30:44.037542605Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-26T23:30:44.037546123Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-26T23:30:44.037549053Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T23:30:44.037551855Z 
2019-11-26T23:49:19.818016484Z 2019-11-26 23:49:19,817 WARN  SleepingTimer - Worker Pin List Sync last execution took 3455 ms. Longer than the interval 1000
2019-11-26T23:49:19.821328653Z 2019-11-26 23:49:19,821 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 3600 ms. Longer than the interval 1000
2019-11-26T23:49:19.821353689Z 2019-11-26 23:49:19,821 WARN  SleepingTimer - Worker Block Sync last execution took 3506 ms. Longer than the interval 1000
2019-11-27T00:12:21.34026837Z 2019-11-27 00:12:21,339 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@152d7e08[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x01\xBbW\xF8b\x07\x00>>>ic/js/main.bba0b4...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T00:12:21.340474534Z 2019-11-27 00:12:21,340 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@6962bb83{r=0,c=false,a=IDLE,uri=}
2019-11-27T00:12:21.83157605Z 2019-11-27 00:12:21,831 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@152d7e08[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\xBbW\xF8b\x07\x00ic/js/main....\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T00:12:21.831599885Z 2019-11-27 00:12:21,831 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@67a558df{r=0,c=false,a=IDLE,uri=}
2019-11-27T00:34:35.925802538Z 2019-11-27 00:34:35,925 WARN  SleepingTimer - Worker Pin List Sync last execution took 3858 ms. Longer than the interval 1000
2019-11-27T00:34:35.926180884Z 2019-11-27 00:34:35,926 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 3862 ms. Longer than the interval 1000
2019-11-27T00:34:35.926924871Z 2019-11-27 00:34:35,926 WARN  SleepingTimer - Worker Block Sync last execution took 3874 ms. Longer than the interval 1000
2019-11-27T04:32:23.46724417Z 2019-11-27 04:32:23,467 WARN  HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@6d8c619d[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>3.125.114.144\r\nPr...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T04:32:23.467308852Z 2019-11-27 04:32:23,467 WARN  HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@69b3f8f4{r=0,c=false,a=IDLE,uri=}
2019-11-27T05:42:20.570450396Z 2019-11-27 05:42:20,570 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@152d7e08[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x01\xBbW\xF8b\x07\x00>>>ic/js/main.bba0b4...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T05:42:20.570472564Z 2019-11-27 05:42:20,570 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@6496e16e{r=0,c=false,a=IDLE,uri=}
2019-11-27T05:42:21.014916095Z 2019-11-27 05:42:21,014 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@152d7e08[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\xBbW\xF8b\x07\x00ic/js/main....\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T05:42:21.015014986Z 2019-11-27 05:42:21,014 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@60a97924{r=0,c=false,a=IDLE,uri=}
2019-11-27T07:50:32.200383651Z 2019-11-27 07:50:32,198 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 5109 ms. Longer than the interval 1000
2019-11-27T07:50:32.200426531Z 2019-11-27 07:50:32,198 WARN  SleepingTimer - Worker Block Sync last execution took 4941 ms. Longer than the interval 1000
2019-11-27T07:50:32.200430767Z 2019-11-27 07:50:32,199 WARN  SleepingTimer - Worker Pin List Sync last execution took 5032 ms. Longer than the interval 1000
2019-11-27T10:21:31.415958422Z 2019-11-27 10:21:31,415 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x01\xBbW\xF8b\x08\x00>>>ic/js/2.6d8a380c....\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T10:21:31.415986572Z 2019-11-27 10:21:31,415 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@12a83fee{r=0,c=false,a=IDLE,uri=}
2019-11-27T10:21:32.839371107Z 2019-11-27 10:21:32,839 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\xBbW\xF8b\x08\x00ic/js/2.6d8...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T10:21:32.839465135Z 2019-11-27 10:21:32,839 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@57aab3df{r=0,c=false,a=IDLE,uri=}
2019-11-27T10:59:55.430773616Z 2019-11-27 10:59:55,430 WARN  SleepingTimer - Worker Pin List Sync last execution took 3211 ms. Longer than the interval 1000
2019-11-27T10:59:55.430897773Z 2019-11-27 10:59:55,430 WARN  SleepingTimer - Worker FileSystemMaster Sync last execution took 3174 ms. Longer than the interval 1000
2019-11-27T10:59:55.43101791Z 2019-11-27 10:59:55,430 WARN  SleepingTimer - Worker Block Sync last execution took 3155 ms. Longer than the interval 1000
2019-11-27T16:23:25.329732852Z Nov 27, 2019 4:23:25 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-27T16:23:25.329761043Z WARNING: Stream Error
2019-11-27T16:23:25.329764818Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 151
2019-11-27T16:23:25.329767944Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-27T16:23:25.329771162Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-27T16:23:25.329774375Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-27T16:23:25.329777342Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-27T16:23:25.329780195Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-27T16:23:25.329783194Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-27T16:23:25.329786143Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-27T16:23:25.329788958Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-27T16:23:25.329803208Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-27T16:23:25.329806048Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-27T16:23:25.329808589Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T16:23:25.32981107Z   at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T16:23:25.329813698Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T16:23:25.329816271Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T16:23:25.329818737Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:25.32982128Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:25.329823763Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T16:23:25.329827033Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T16:23:25.329829539Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:25.329832073Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:25.329834579Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T16:23:25.329837144Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T16:23:25.329839676Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-27T16:23:25.329842191Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T16:23:25.329844654Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T16:23:25.329847185Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T16:23:25.329849705Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T16:23:25.329852133Z 
2019-11-27T16:23:33.59108926Z Nov 27, 2019 4:23:33 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-27T16:23:33.591121495Z WARNING: Stream Error
2019-11-27T16:23:33.591125227Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 159
2019-11-27T16:23:33.59112831Z   at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-27T16:23:33.591131388Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-27T16:23:33.591145122Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-27T16:23:33.591148294Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-27T16:23:33.591150956Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-27T16:23:33.591153652Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-27T16:23:33.591156331Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-27T16:23:33.591158919Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-27T16:23:33.591161538Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-27T16:23:33.591164125Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-27T16:23:33.591166656Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T16:23:33.591169208Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T16:23:33.591171881Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T16:23:33.591174418Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T16:23:33.591176937Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:33.591179535Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:33.591182127Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T16:23:33.591185355Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T16:23:33.59118794Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:33.591190514Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:33.591193007Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T16:23:33.591205685Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T16:23:33.591209251Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-27T16:23:33.591211834Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T16:23:33.591217854Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T16:23:33.591220462Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T16:23:33.591222989Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T16:23:33.591225426Z 
2019-11-27T16:23:35.143672059Z Nov 27, 2019 4:23:35 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-27T16:23:35.143704421Z WARNING: Stream Error
2019-11-27T16:23:35.143708603Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 163
2019-11-27T16:23:35.14371211Z   at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-27T16:23:35.14371595Z   at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-27T16:23:35.143719337Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-27T16:23:35.143722474Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-27T16:23:35.143725561Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-27T16:23:35.143728598Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-27T16:23:35.143733335Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-27T16:23:35.143736962Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-27T16:23:35.143740181Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-27T16:23:35.143743372Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-27T16:23:35.14374682Z   at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T16:23:35.143749925Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T16:23:35.143753117Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T16:23:35.143756098Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T16:23:35.143759075Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:35.14376216Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:35.143765182Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T16:23:35.143768893Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T16:23:35.14378389Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:35.143787132Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:35.143790231Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T16:23:35.143796552Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T16:23:35.143799625Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-27T16:23:35.143802672Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T16:23:35.143805625Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T16:23:35.143808642Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T16:23:35.143811752Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T16:23:35.143814763Z 
2019-11-27T16:23:43.247990751Z Nov 27, 2019 4:23:43 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-27T16:23:43.248018598Z WARNING: Stream Error
2019-11-27T16:23:43.248021883Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 175
2019-11-27T16:23:43.248024653Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-27T16:23:43.248027849Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-27T16:23:43.248030569Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-27T16:23:43.248033107Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-27T16:23:43.248035699Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-27T16:23:43.248038347Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-27T16:23:43.248040829Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-27T16:23:43.248043282Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-27T16:23:43.248045761Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-27T16:23:43.24804833Z   at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-27T16:23:43.248050954Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T16:23:43.248070178Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T16:23:43.248073117Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T16:23:43.248075596Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T16:23:43.248078218Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:43.248080689Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:43.24808329Z   at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T16:23:43.248086448Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T16:23:43.248088968Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:43.248091443Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:43.248094054Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T16:23:43.248096493Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T16:23:43.248098998Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-27T16:23:43.248101545Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T16:23:43.248104095Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T16:23:43.248106498Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T16:23:43.248109014Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T16:23:43.248111474Z 
2019-11-27T16:23:58.444166487Z Nov 27, 2019 4:23:58 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-27T16:23:58.444194571Z WARNING: Stream Error
2019-11-27T16:23:58.444213418Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 185
2019-11-27T16:23:58.444216862Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-27T16:23:58.444220288Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-27T16:23:58.444223433Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-27T16:23:58.444226572Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-27T16:23:58.444229457Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-27T16:23:58.444243599Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-27T16:23:58.444246343Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-27T16:23:58.444248945Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-27T16:23:58.444251453Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-27T16:23:58.444253969Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-27T16:23:58.444256468Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T16:23:58.444259055Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T16:23:58.444261574Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T16:23:58.444264036Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T16:23:58.444266533Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:58.444269163Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:58.444271742Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T16:23:58.444275233Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T16:23:58.444277743Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:23:58.444280254Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:23:58.444282769Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T16:23:58.444285381Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T16:23:58.444287932Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-27T16:23:58.444290479Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T16:23:58.444292937Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T16:23:58.444295676Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T16:23:58.444298219Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T16:23:58.444300754Z 
2019-11-27T16:24:03.121191603Z Nov 27, 2019 4:24:03 PM io.grpc.netty.NettyServerHandler onStreamError
2019-11-27T16:24:03.121243776Z WARNING: Stream Error
2019-11-27T16:24:03.121248517Z io.netty.handler.codec.http2.Http2Exception$StreamException: Received DATA frame for an unknown stream 189
2019-11-27T16:24:03.121252299Z  at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:129)
2019-11-27T16:24:03.121255861Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.shouldIgnoreHeadersOrDataFrame(DefaultHttp2ConnectionDecoder.java:531)
2019-11-27T16:24:03.121259313Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:183)
2019-11-27T16:24:03.121262736Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
2019-11-27T16:24:03.121266101Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:421)
2019-11-27T16:24:03.121269297Z  at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:251)
2019-11-27T16:24:03.12127261Z   at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
2019-11-27T16:24:03.121276235Z  at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
2019-11-27T16:24:03.121279079Z  at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
2019-11-27T16:24:03.12128168Z   at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
2019-11-27T16:24:03.121284197Z  at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T16:24:03.121286712Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T16:24:03.121289308Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T16:24:03.121291761Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T16:24:03.12129424Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:24:03.121297062Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:24:03.121299592Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T16:24:03.121302827Z  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T16:24:03.121305325Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T16:24:03.121307926Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T16:24:03.121310423Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T16:24:03.121316331Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T16:24:03.121319014Z  at io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:138)
2019-11-27T16:24:03.121321683Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T16:24:03.121324156Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T16:24:03.121326567Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T16:24:03.121329101Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T16:24:03.121331551Z 
2019-11-27T18:47:52.154699323Z Nov 27, 2019 6:47:52 PM io.grpc.netty.NettyServerTransport notifyTerminated
2019-11-27T18:47:52.154726024Z INFO: Transport failed
2019-11-27T18:47:52.154730145Z io.netty.handler.codec.http2.Http2Exception: HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 0300002f2ae00000000000436f6f6b69653a206d73747368
2019-11-27T18:47:52.154733524Z  at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
2019-11-27T18:47:52.154737055Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:318)
2019-11-27T18:47:52.154740229Z  at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:251)
2019-11-27T18:47:52.15474339Z   at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
2019-11-27T18:47:52.154746527Z  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
2019-11-27T18:47:52.154749602Z  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
2019-11-27T18:47:52.154752755Z  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
2019-11-27T18:47:52.154755795Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T18:47:52.154758876Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T18:47:52.154761904Z  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
2019-11-27T18:47:52.15476495Z   at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
2019-11-27T18:47:52.154767987Z  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
2019-11-27T18:47:52.15477109Z   at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
2019-11-27T18:47:52.154774113Z  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
2019-11-27T18:47:52.154786708Z  at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2019-11-27T18:47:52.154789592Z  at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
2019-11-27T18:47:52.154792214Z  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
2019-11-27T18:47:52.154795397Z  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
2019-11-27T18:47:52.154798308Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T18:47:52.154800908Z 
2019-11-27T19:31:35.238592684Z 2019-11-27 19:31:35,238 WARN  SaslStreamServerDriver - Error received for channel: 26da8e11-2162-414d-96a8-e1ec384452f4. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T19:31:35.238712307Z 2019-11-27 19:31:35,238 WARN  SaslStreamServerDriver - Error received for channel: 4a55dd5c-348b-4e99-bd95-94225e9d17b3. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T20:18:05.99073719Z 2019-11-27 20:18:05,990 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x01\xBbW\xF8b\x07\x00>>>ic/js/2.6d8a380c....\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T20:18:05.990806562Z 2019-11-27 20:18:05,990 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@2a030f76{r=0,c=false,a=IDLE,uri=}
2019-11-27T20:18:06.428294721Z 2019-11-27 20:18:06,428 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\xBbW\xF8b\x07\x00ic/js/2.6d8...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-27T20:18:06.428375575Z 2019-11-27 20:18:06,428 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@2e3c8eee{r=0,c=false,a=IDLE,uri=}
2019-11-28T00:08:34.851718829Z 2019-11-28 00:08:34,851 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@152d7e08[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x00\x19\xBc}I\x1d\x00>>>//122.51.98.31:30...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T00:08:34.851751757Z 2019-11-28 00:08:34,851 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@44a5f8bb{r=0,c=false,a=IDLE,uri=}
2019-11-28T00:08:35.268751881Z 2019-11-28 00:08:35,268 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@152d7e08[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\x19\xBc}I\x1d\x00//122.51.98...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T00:08:35.268782302Z 2019-11-28 00:08:35,268 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@629eb2ff{r=0,c=false,a=IDLE,uri=}
2019-11-28T00:31:49.374988012Z 2019-11-28 00:31:49,374 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x01\xBbW\xF8b\x07\x00>>>//122.51.98.31:30...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T00:31:49.375072129Z 2019-11-28 00:31:49,374 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@cce7c53{r=0,c=false,a=IDLE,uri=}
2019-11-28T00:31:49.781580801Z 2019-11-28 00:31:49,781 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\xBbW\xF8b\x07\x00//122.51.98...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T00:31:49.781652435Z 2019-11-28 00:31:49,781 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@8f1f912{r=0,c=false,a=IDLE,uri=}
2019-11-28T03:38:51.126361589Z 2019-11-28 03:38:51,126 WARN  HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>: text/html,appli...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T03:38:51.126431959Z 2019-11-28 03:38:51,126 WARN  HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@5599d27d{r=0,c=false,a=IDLE,uri=}
2019-11-28T05:45:53.65289301Z 2019-11-28 05:45:53,652 WARN  HttpParser - Illegal character 0x4 in state=START for buffer HeapByteBuffer@6d8c619d[p=1,l=9,c=16384,r=8]={\x04<<<\x01\x01\xBbW\xF8b\x07\x00>>>//122.51.98.31:30...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T05:45:53.652966258Z 2019-11-28 05:45:53,652 WARN  HttpParser - badMessage: 400 Illegal character 0x4 for HttpChannelOverHttp@263e33f1{r=0,c=false,a=IDLE,uri=}
2019-11-28T05:45:56.541864393Z 2019-11-28 05:45:56,541 WARN  HttpParser - Illegal character 0x5 in state=START for buffer HeapByteBuffer@6d8c619d[p=1,l=3,c=16384,r=2]={\x05<<<\x01\x00>>>\xBbW\xF8b\x07\x00//122.51.98...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T05:45:56.54193515Z 2019-11-28 05:45:56,541 WARN  HttpParser - badMessage: 400 Illegal character 0x5 for HttpChannelOverHttp@2201a3ee{r=0,c=false,a=IDLE,uri=}
2019-11-28T06:09:12.732058131Z 2019-11-28 06:09:12,731 WARN  HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@1697b9e4[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>ip.net:443\r\nProxy...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
2019-11-28T06:09:12.732147566Z 2019-11-28 06:09:12,731 WARN  HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@1f2a5f37{r=0,c=false,a=IDLE,uri=}

master上的日志（从k8s里抓的，也不知道全不全）：

2019-11-26T22:24:43.141178127Z 2019-11-26 22:24:43,140 INFO  FileSystemMasterFactory - Creating alluxio.master.file.FileSystemMaster 
2019-11-26T22:24:43.141299977Z 2019-11-26 22:24:43,140 INFO  MetaMasterFactory - Creating alluxio.master.meta.MetaMaster 
2019-11-26T22:24:43.141720266Z 2019-11-26 22:24:43,140 INFO  MetricsMasterFactory - Creating alluxio.master.metrics.MetricsMaster 
2019-11-26T22:24:43.142486247Z 2019-11-26 22:24:43,140 INFO  TableMasterFactory - Creating alluxio.master.table.TableMaster 
2019-11-26T22:24:43.142498376Z 2019-11-26 22:24:43,140 INFO  BlockMasterFactory - Creating alluxio.master.block.BlockMaster 
2019-11-26T22:24:43.159148297Z 2019-11-26 22:24:43,158 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.203866951Z 2019-11-26 22:24:43,203 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.222120872Z 2019-11-26 22:24:43,221 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-26T22:24:43.232298188Z 2019-11-26 22:24:43,231 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-26T22:24:43.321058665Z 2019-11-26 22:24:43,320 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.332314079Z 2019-11-26 22:24:43,331 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-26T22:24:43.333741615Z 2019-11-26 22:24:43,333 INFO  NettyUtils - EPOLL_MODE is available
2019-11-26T22:24:43.417890122Z 2019-11-26 22:24:43,416 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.42551901Z 2019-11-26 22:24:43,425 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=alluxio-master-0, rack=null)
2019-11-26T22:24:43.426333232Z 2019-11-26 22:24:43,426 INFO  UnderDatabaseRegistry - Loading udb jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.427320312Z 2019-11-26 22:24:43,427 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-26T22:24:43.437966122Z 2019-11-26 22:24:43,437 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.438454433Z 2019-11-26 22:24:43,438 INFO  UnderDatabaseRegistry - Registered UDBs: hive
2019-11-26T22:24:43.440002326Z 2019-11-26 22:24:43,439 INFO  LayoutRegistry - Registered Table Layouts: hive
2019-11-26T22:24:43.504473193Z 2019-11-26 22:24:43,504 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-26T22:24:43.514314244Z 2019-11-26 22:24:43,514 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-26T22:24:43.52183487Z 2019-11-26 22:24:43,521 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-26T22:24:43.954069371Z 2019-11-26 22:24:43,953 INFO  ProcessUtils - Starting Alluxio master @alluxio-master-0:19998.
2019-11-26T22:24:43.956427004Z 2019-11-26 22:24:43,956 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpoint thread started.
2019-11-26T22:24:43.956549864Z 2019-11-26 22:24:43,956 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpoint thread started.
2019-11-26T22:24:43.956670322Z 2019-11-26 22:24:43,956 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpoint thread started.
2019-11-26T22:24:43.956784366Z 2019-11-26 22:24:43,956 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpoint thread started.
2019-11-26T22:24:43.956847895Z 2019-11-26 22:24:43,956 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpoint thread started.
2019-11-26T22:24:43.996208055Z 2019-11-26 22:24:43,995 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpointer shutdown has been initiated.
2019-11-26T22:24:43.996364806Z 2019-11-26 22:24:43,996 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpointer shutdown has been initiated.
2019-11-26T22:24:43.996493218Z 2019-11-26 22:24:43,996 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpointer shutdown has been initiated.
2019-11-26T22:24:43.996709277Z 2019-11-26 22:24:43,996 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpointer shutdown has been initiated.
2019-11-26T22:24:43.996785526Z 2019-11-26 22:24:43,996 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpointer shutdown has been initiated.
2019-11-26T22:24:44.000631897Z 2019-11-26 22:24:44,000 INFO  UfsJournalReader - Reading journal file /journal/MetaMaster/v1/logs/0x0-0x1.
2019-11-26T22:24:44.002022749Z 2019-11-26 22:24:44,001 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x0-0x7.
2019-11-26T22:24:44.007092912Z 2019-11-26 22:24:44,006 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x0-0xa.
2019-11-26T22:24:44.020607003Z 2019-11-26 22:24:44,019 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x7-0x1295d.
2019-11-26T22:24:44.146287983Z 2019-11-26 22:24:44,146 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xa-0xed56.
2019-11-26T22:24:44.52427703Z 2019-11-26 22:24:44,523 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x1295d-0x1295e.
2019-11-26T22:24:44.524298535Z 2019-11-26 22:24:44,523 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x1295e-0x137f9.
2019-11-26T22:24:44.595709413Z 2019-11-26 22:24:44,595 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x137f9-0x13804.
2019-11-26T22:24:44.596005543Z 2019-11-26 22:24:44,595 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x13804-0x13809.
2019-11-26T22:24:44.596214827Z 2019-11-26 22:24:44,596 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x13809-0x1380d.
2019-11-26T22:24:44.596452365Z 2019-11-26 22:24:44,596 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x1380d-0x13811.
2019-11-26T22:24:44.596633342Z 2019-11-26 22:24:44,596 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x13811-0x13817.
2019-11-26T22:24:44.596841043Z 2019-11-26 22:24:44,596 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x13817-0x1381a.
2019-11-26T22:24:44.596991377Z 2019-11-26 22:24:44,596 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x1381a-0x1381b.
2019-11-26T22:24:44.597217111Z 2019-11-26 22:24:44,597 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x1381b-0x1381d.
2019-11-26T22:24:44.597422305Z 2019-11-26 22:24:44,597 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x1381d-0x46e09.
2019-11-26T22:24:45.616666239Z 2019-11-26 22:24:45,616 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x46e09-0x625c5.
2019-11-26T22:24:45.722992407Z 2019-11-26 22:24:45,722 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xed56-0x1d608.
2019-11-26T22:24:46.030095731Z 2019-11-26 22:24:46,029 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x625c5-0x71093.
2019-11-26T22:24:47.224899849Z 2019-11-26 22:24:47,224 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x71093-0x75e5f.
2019-11-26T22:24:47.23504887Z 2019-11-26 22:24:47,234 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x1d608-0x2c132.
2019-11-26T22:24:47.306308486Z 2019-11-26 22:24:47,306 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x75e5f-0x77041.
2019-11-26T22:24:47.313056083Z 2019-11-26 22:24:47,312 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x77041-0x77be4.
2019-11-26T22:24:47.317556375Z 2019-11-26 22:24:47,317 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x77be4-0x78353.
2019-11-26T22:24:47.321132495Z 2019-11-26 22:24:47,320 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x78353-0x78c01.
2019-11-26T22:24:47.325119911Z 2019-11-26 22:24:47,324 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x78c01-0x799ef.
2019-11-26T22:24:47.331403263Z 2019-11-26 22:24:47,331 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x799ef-0x7a347.
2019-11-26T22:24:47.395940425Z 2019-11-26 22:24:47,395 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x7a347-0x7a870.
2019-11-26T22:24:47.398779294Z 2019-11-26 22:24:47,398 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x7a870-0x7a873.
2019-11-26T22:24:47.399974026Z 2019-11-26 22:24:47,399 INFO  UfsJournalReader - Reading journal file /journal/BlockMaster/v1/logs/0x7a873-0x7a878.
2019-11-26T22:24:49.522934989Z 2019-11-26 22:24:49,522 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2019-11-26T22:24:49.523129611Z 2019-11-26 22:24:49,523 INFO  UfsJournalCheckpointThread - MetaMaster: Journal shutdown complete
2019-11-26T22:24:49.636378454Z 2019-11-26 22:24:49,636 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x2c132-0x39051.
2019-11-26T22:24:49.953031442Z 2019-11-26 22:24:49,952 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x39051-0x40678.
2019-11-26T22:24:49.998156616Z 2019-11-26 22:24:49,997 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2019-11-26T22:24:49.998314058Z 2019-11-26 22:24:49,998 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal shutdown complete
2019-11-26T22:24:49.99881562Z 2019-11-26 22:24:49,998 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2019-11-26T22:24:49.999892857Z 2019-11-26 22:24:49,998 INFO  UfsJournalCheckpointThread - TableMaster: Journal shutdown complete
2019-11-26T22:24:50.027295616Z 2019-11-26 22:24:50,027 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x40678-0x40794.
2019-11-26T22:24:50.027947222Z 2019-11-26 22:24:50,027 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x40794-0x407b2.
2019-11-26T22:24:50.028292941Z 2019-11-26 22:24:50,028 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x407b2-0x407c7.
2019-11-26T22:24:50.030014608Z 2019-11-26 22:24:50,028 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x407c7-0x407dd.
2019-11-26T22:24:50.0300269Z 2019-11-26 22:24:50,028 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x407dd-0x40803.
2019-11-26T22:24:50.030030329Z 2019-11-26 22:24:50,028 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x40803-0x40813.
2019-11-26T22:24:50.03003328Z 2019-11-26 22:24:50,029 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x40813-0x40823.
2019-11-26T22:24:50.030036056Z 2019-11-26 22:24:50,029 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x40823-0x40838.
2019-11-26T22:24:50.030038694Z 2019-11-26 22:24:50,029 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x40838-0x4f5cc.
2019-11-26T22:24:50.425255299Z 2019-11-26 22:24:50,425 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x4f5cc-0x5de06.
2019-11-26T22:24:50.749762829Z 2019-11-26 22:24:50,749 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x5de06-0x6b97d.
2019-11-26T22:24:51.030090057Z 2019-11-26 22:24:51,029 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x6b97d-0x78d94.
2019-11-26T22:24:51.200878434Z 2019-11-26 22:24:51,200 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x78d94-0x8617c.
2019-11-26T22:24:53.496700016Z 2019-11-26 22:24:53,496 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2019-11-26T22:24:53.496718805Z 2019-11-26 22:24:53,496 INFO  UfsJournalCheckpointThread - BlockMaster: Journal shutdown complete
2019-11-26T22:24:53.567439941Z 2019-11-26 22:24:53,567 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x8617c-0x9352b.
2019-11-26T22:24:53.739274961Z 2019-11-26 22:24:53,739 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x9352b-0xa07be.
2019-11-26T22:24:53.838162677Z 2019-11-26 22:24:53,837 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xa07be-0xad97e.
2019-11-26T22:24:54.05788176Z 2019-11-26 22:24:54,057 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xad97e-0xbabe4.
2019-11-26T22:24:54.234335839Z 2019-11-26 22:24:54,234 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xbabe4-0xc7e37.
2019-11-26T22:24:54.320180722Z 2019-11-26 22:24:54,320 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xc7e37-0xd4ea3.
2019-11-26T22:24:56.552008293Z 2019-11-26 22:24:56,551 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xd4ea3-0xdabfd.
2019-11-26T22:24:56.602107572Z 2019-11-26 22:24:56,601 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xdabfd-0xe7c65.
2019-11-26T22:24:56.764104504Z 2019-11-26 22:24:56,763 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xe7c65-0xf4b70.
2019-11-26T22:24:56.879281052Z 2019-11-26 22:24:56,879 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0xf4b70-0x101a52.
2019-11-26T22:24:57.187175657Z 2019-11-26 22:24:57,186 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x101a52-0x110714.
2019-11-26T22:24:57.416740158Z 2019-11-26 22:24:57,416 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x110714-0x11fbc2.
2019-11-26T22:24:57.527648808Z 2019-11-26 22:24:57,527 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x11fbc2-0x12d11a.
2019-11-26T22:24:57.874452008Z 2019-11-26 22:24:57,874 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x12d11a-0x13bd42.
2019-11-26T22:24:57.967361847Z 2019-11-26 22:24:57,967 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x13bd42-0x14a9a2.
2019-11-26T22:24:58.189174076Z 2019-11-26 22:24:58,188 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x14a9a2-0x1592dc.
2019-11-26T22:24:58.540090517Z 2019-11-26 22:24:58,539 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x1592dc-0x167cf7.
2019-11-26T22:24:58.659852519Z 2019-11-26 22:24:58,659 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x167cf7-0x16b29e.
2019-11-26T22:24:58.686688383Z 2019-11-26 22:24:58,686 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x16b29e-0x16d584.
2019-11-26T22:25:02.441318885Z 2019-11-26 22:25:02,441 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x16d584-0x16eb90.
2019-11-26T22:25:02.451823783Z 2019-11-26 22:25:02,451 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x16eb90-0x170500.
2019-11-26T22:25:02.496504772Z 2019-11-26 22:25:02,496 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x170500-0x172edd.
2019-11-26T22:25:02.515238245Z 2019-11-26 22:25:02,514 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x172edd-0x174abc.
2019-11-26T22:25:02.527854504Z 2019-11-26 22:25:02,527 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x174abc-0x175a13.
2019-11-26T22:25:02.534916153Z 2019-11-26 22:25:02,534 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x175a13-0x175a18.
2019-11-26T22:25:02.535144909Z 2019-11-26 22:25:02,535 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x175a18-0x175a31.
2019-11-26T22:25:02.536913151Z 2019-11-26 22:25:02,536 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x175a31-0x175a32.
2019-11-26T22:25:02.537107603Z 2019-11-26 22:25:02,537 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x175a32-0x175a4a.
2019-11-26T22:25:07.546209759Z 2019-11-26 22:25:07,546 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2019-11-26T22:25:07.54632028Z 2019-11-26 22:25:07,546 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal shutdown complete
2019-11-26T22:25:07.549907319Z 2019-11-26 22:25:07,549 INFO  UfsJournalReader - Reading journal file /journal/FileSystemMaster/v1/logs/0x175a4a-0x7fffffffffffffff.
2019-11-26T22:25:07.587443211Z 2019-11-26 22:25:07,587 INFO  AbstractMaster - MetricsMaster: Starting primary master.
2019-11-26T22:25:07.58765217Z 2019-11-26 22:25:07,587 INFO  AbstractMaster - BlockMaster: Starting primary master.
2019-11-26T22:25:07.588623814Z 2019-11-26 22:25:07,588 INFO  AbstractMaster - FileSystemMaster: Starting primary master.
2019-11-26T22:25:07.588639561Z 2019-11-26 22:25:07,588 INFO  DefaultFileSystemMaster - Starting fs master as primary
2019-11-26T22:25:07.775847001Z 2019-11-26 22:25:07,775 INFO  AbstractMaster - MetaMaster: Starting primary master.
2019-11-26T22:25:07.784868361Z 2019-11-26 22:25:07,784 INFO  DefaultMetaMaster - Detected existing cluster ID a6462635-e5f8-48b0-afa4-5d7db43df4c7
2019-11-26T22:25:07.78494299Z 2019-11-26 22:25:07,784 INFO  AbstractMaster - TableMaster: Starting primary master.
2019-11-26T22:25:07.786229362Z 2019-11-26 22:25:07,786 INFO  AlluxioMasterProcess - All masters started
2019-11-26T22:25:07.787364066Z 2019-11-26 22:25:07,787 INFO  MetricsSystem - Starting sinks with config: {}.
2019-11-26T22:25:07.787499977Z 2019-11-26 22:25:07,787 INFO  AlluxioMasterProcess - Alluxio master web server version 2.1.0 starting. webAddress=/0.0.0.0:19999
2019-11-26T22:25:07.796150432Z 2019-11-26 22:25:07,796 INFO  log - Logging initialized @25216ms
2019-11-26T22:25:08.226770337Z 2019-11-26 22:25:08,226 INFO  WebServer - Alluxio Master Web service starting @ /0.0.0.0:19999
2019-11-26T22:25:08.228698811Z 2019-11-26 22:25:08,228 INFO  Server - jetty-9.2.z-SNAPSHOT
2019-11-26T22:25:08.245816071Z 2019-11-26 22:25:08,245 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@4a94ee4{/metrics/prometheus,null,AVAILABLE}
2019-11-26T22:25:08.245940651Z 2019-11-26 22:25:08,245 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@6379eb{/metrics/json,null,AVAILABLE}
2019-11-26T22:25:14.938343222Z 2019-11-26 22:25:14,937 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@3e5499cc{/,file:/opt/alluxio-2.1.0/webui/master/build/,AVAILABLE}
2019-11-26T22:25:14.945702907Z 2019-11-26 22:25:14,945 INFO  ServerConnector - Started ServerConnector@338494fa{HTTP/1.1}{0.0.0.0:19999}
2019-11-26T22:25:14.945734855Z 2019-11-26 22:25:14,945 INFO  Server - Started @32365ms
2019-11-26T22:25:14.946174467Z 2019-11-26 22:25:14,946 INFO  WebServer - Alluxio Master Web service started @ /0.0.0.0:19999
2019-11-26T22:25:14.94628345Z 2019-11-26 22:25:14,946 INFO  AlluxioMasterProcess - Alluxio master version 2.1.0 started. bindAddress=/0.0.0.0:19998, connectAddress=alluxio-master-0:19998, webAddress=/0.0.0.0:19999
2019-11-26T22:25:14.946333222Z 2019-11-26 22:25:14,946 INFO  AlluxioMasterProcess - Starting Alluxio master gRPC server on address /0.0.0.0:19998
2019-11-26T22:25:15.027568684Z 2019-11-26 22:25:15,027 INFO  MasterProcess - registered service METRICS_MASTER_CLIENT_SERVICE
2019-11-26T22:25:15.069488315Z 2019-11-26 22:25:15,069 INFO  MasterProcess - registered service BLOCK_MASTER_CLIENT_SERVICE
2019-11-26T22:25:15.069583861Z 2019-11-26 22:25:15,069 INFO  MasterProcess - registered service BLOCK_MASTER_WORKER_SERVICE
2019-11-26T22:25:15.150032716Z 2019-11-26 22:25:15,149 INFO  MasterProcess - registered service FILE_SYSTEM_MASTER_JOB_SERVICE
2019-11-26T22:25:15.150099864Z 2019-11-26 22:25:15,150 INFO  MasterProcess - registered service FILE_SYSTEM_MASTER_WORKER_SERVICE
2019-11-26T22:25:15.150127743Z 2019-11-26 22:25:15,150 INFO  MasterProcess - registered service FILE_SYSTEM_MASTER_CLIENT_SERVICE
2019-11-26T22:25:15.182635862Z 2019-11-26 22:25:15,182 INFO  MasterProcess - registered service META_MASTER_CLIENT_SERVICE
2019-11-26T22:25:15.182658645Z 2019-11-26 22:25:15,182 INFO  MasterProcess - registered service META_MASTER_CONFIG_SERVICE
2019-11-26T22:25:15.182671642Z 2019-11-26 22:25:15,182 INFO  MasterProcess - registered service META_MASTER_MASTER_SERVICE
2019-11-26T22:25:15.212642098Z 2019-11-26 22:25:15,212 INFO  MasterProcess - registered service TABLE_MASTER_CLIENT_SERVICE
2019-11-26T22:25:15.313919283Z 2019-11-26 22:25:15,313 INFO  DefaultSafeModeManager - Rpc server started, waiting 5000ms for workers to register
2019-11-26T22:25:15.313941457Z 2019-11-26 22:25:15,313 INFO  AlluxioMasterProcess - Started Alluxio master gRPC server on address alluxio-master-0:19998
2019-11-26T22:25:16.810255453Z 2019-11-26 22:25:16,810 WARN  DefaultBlockMaster - Could not find worker id: 6252324878552688017 for heartbeat.
2019-11-26T22:25:16.822557505Z 2019-11-26 22:25:16,822 INFO  DefaultBlockMaster - getWorkerId(): WorkerNetAddress: WorkerNetAddress{host=10.0.68.212, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=/opt/domain/b2a34bc8-e589-44a2-bc36-d5660d6bffc5, tieredIdentity=TieredIdentity(node=10.0.68.212, rack=null)} id: 2849935057917294514
2019-11-26T22:25:16.837508234Z 2019-11-26 22:25:16,837 INFO  DefaultBlockMaster - registerWorker(): MasterWorkerInfo{id=2849935057917294514, workerAddress=WorkerNetAddress{host=10.0.68.212, rpcPort=29999, dataPort=29999, webPort=30000, domainSocketPath=/opt/domain/b2a34bc8-e589-44a2-bc36-d5660d6bffc5, tieredIdentity=TieredIdentity(node=10.0.68.212, rack=null)}, capacityBytes=23622320128, usedBytes=8536708, lastUpdatedTimeMs=1574807116837, blocks=[6061406814208, 8623505801216], lostStorage={}}
2019-11-26T23:25:07.786235168Z 2019-11-26 23:25:07,785 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T00:25:07.786013628Z 2019-11-27 00:25:07,785 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T00:56:18.677567398Z 2019-11-27 00:56:18,675 WARN  SaslStreamServerDriver - Error received for channel: b39f5659-b172-4bfd-a1e7-6872f35f2b15. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.677604008Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: 43567074-35b1-4c37-8acf-394d5abd79ca. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.677611343Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: 984a7f1f-b6bc-4e4d-bcaf-a63b8ef778b7. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.677677713Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: 933c8228-eeff-46a8-892f-e31adb529500. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.677858453Z 2019-11-27 00:56:18,675 WARN  SaslStreamServerDriver - Error received for channel: 4e52116f-d108-45fc-b94e-89c6a781d2af. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.677865675Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: 18c18f2a-2226-4f4e-ba89-285fe4ea48a4. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.677995157Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: 7cf7ee1f-34f2-4834-8b84-98ab0200accc. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.67805712Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: c7854a7e-bc56-451b-8548-05c6f816e359. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.678092866Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: 0d81f62b-fafa-4de8-8f97-3bc01b1467cb. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.678332409Z 2019-11-27 00:56:18,677 WARN  SaslStreamServerDriver - Error received for channel: fdfacb79-c486-423e-aa87-6972fc228d2a. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T00:56:18.678341123Z 2019-11-27 00:56:18,676 WARN  SaslStreamServerDriver - Error received for channel: 4a32842c-fb4d-4f04-a0a8-efe159ba1765. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T01:25:07.786347138Z 2019-11-27 01:25:07,786 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T02:25:07.786563863Z 2019-11-27 02:25:07,786 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T03:25:07.786814479Z 2019-11-27 03:25:07,786 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T04:25:07.787074154Z 2019-11-27 04:25:07,786 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T05:25:07.78732029Z 2019-11-27 05:25:07,787 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T06:25:07.787590737Z 2019-11-27 06:25:07,787 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T07:25:07.787809978Z 2019-11-27 07:25:07,787 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T07:49:59.652636131Z 2019-11-27 07:49:59,652 INFO  ServerConfigurationChecker - All server-side configurations are consistent.
2019-11-27T07:50:20.775261851Z 2019-11-27 07:50:20,773 ERROR COSUnderFileSystem - Failed to create object: /
2019-11-27T07:50:20.775291807Z com.qcloud.cos.exception.CosServiceException: Your previous request to create the named bucket succeeded and you already own it. (Status Code: 409; Error Code: BucketAlreadyOwnedByYou; Request ID: NWRkZTJhYmNfNWYxZDFkMDlfMjI2NzhfMjEwMTc2Mw==); Trace ID: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0NzJmZTI0ZmJhYTZmZjYyNmU5ZGNlZDI5YjkyODkwYjNhN2MyMTYxN2NhOGE5YjMyYjE2MzI4NzljYTllNDc4N2Q=
2019-11-27T07:50:20.775296449Z  at com.qcloud.cos.http.DefaultCosHttpClient.handlerErrorMessage(DefaultCosHttpClient.java:293)
2019-11-27T07:50:20.775300374Z  at com.qcloud.cos.http.DefaultCosHttpClient.exeute(DefaultCosHttpClient.java:385)
2019-11-27T07:50:20.775307139Z  at com.qcloud.cos.COSClient.invoke(COSClient.java:463)
2019-11-27T07:50:20.775310341Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:695)
2019-11-27T07:50:20.775313483Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:750)
2019-11-27T07:50:20.77531699Z   at alluxio.underfs.cos.COSUnderFileSystem.createEmptyObject(COSUnderFileSystem.java:142)
2019-11-27T07:50:20.775320439Z  at alluxio.underfs.ObjectUnderFileSystem.mkdirsInternal(ObjectUnderFileSystem.java:1080)
2019-11-27T07:50:20.775323762Z  at alluxio.underfs.ObjectUnderFileSystem.getObjectListingChunkForPath(ObjectUnderFileSystem.java:946)
2019-11-27T07:50:20.775326762Z  at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:970)
2019-11-27T07:50:20.775329612Z  at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:605)
2019-11-27T07:50:20.775332396Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:584)
2019-11-27T07:50:20.7753354Z    at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:581)
2019-11-27T07:50:20.77533829Z   at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:949)
2019-11-27T07:50:20.775341085Z  at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:581)
2019-11-27T07:50:20.7753441Z    at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:374)
2019-11-27T07:50:20.775347013Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:371)
2019-11-27T07:50:20.775349864Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.block(ManagedBlockingUfsForwarder.java:596)
2019-11-27T07:50:20.775353424Z  at alluxio.concurrent.jsr.ForkJoinPool.managedBlock(ForkJoinPool.java:1013)
2019-11-27T07:50:20.775365386Z  at alluxio.concurrent.ForkJoinPoolHelper.safeManagedBlock(ForkJoinPoolHelper.java:41)
2019-11-27T07:50:20.775368072Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.get(ManagedBlockingUfsForwarder.java:582)
2019-11-27T07:50:20.775370653Z  at alluxio.concurrent.ManagedBlockingUfsForwarder.listStatus(ManagedBlockingUfsForwarder.java:376)
2019-11-27T07:50:20.775373159Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataInternal(DefaultFileSystemMaster.java:2401)
2019-11-27T07:50:20.77537564Z   at alluxio.master.file.DefaultFileSystemMaster.loadMetadataIfNotExist(DefaultFileSystemMaster.java:2650)
2019-11-27T07:50:20.775378188Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:864)
2019-11-27T07:50:20.775380696Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:881)
2019-11-27T07:50:20.775383139Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.lambda$getWebUIBrowse$3(AlluxioMasterRestServiceHandler.java:477)
2019-11-27T07:50:20.775385688Z  at alluxio.RestUtils.call(RestUtils.java:62)
2019-11-27T07:50:20.775388108Z  at alluxio.RestUtils.call(RestUtils.java:79)
2019-11-27T07:50:20.775390529Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.getWebUIBrowse(AlluxioMasterRestServiceHandler.java:357)
2019-11-27T07:50:20.775393033Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-11-27T07:50:20.77539551Z   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-11-27T07:50:20.775398006Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-11-27T07:50:20.775400562Z  at java.lang.reflect.Method.invoke(Method.java:498)
2019-11-27T07:50:20.775403054Z  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2019-11-27T07:50:20.775405626Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2019-11-27T07:50:20.775408176Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2019-11-27T07:50:20.77541084Z   at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
2019-11-27T07:50:20.775413731Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2019-11-27T07:50:20.775416361Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2019-11-27T07:50:20.775418835Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2019-11-27T07:50:20.77542143Z   at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2019-11-27T07:50:20.775427192Z  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
2019-11-27T07:50:20.775429819Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2019-11-27T07:50:20.775432286Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2019-11-27T07:50:20.775434787Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2019-11-27T07:50:20.775437535Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2019-11-27T07:50:20.775439959Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2019-11-27T07:50:20.775442361Z  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2019-11-27T07:50:20.77544498Z   at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
2019-11-27T07:50:20.775447443Z  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
2019-11-27T07:50:20.775449937Z  at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
2019-11-27T07:50:20.775452417Z  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
2019-11-27T07:50:20.775454863Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
2019-11-27T07:50:20.775457309Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
2019-11-27T07:50:20.775459765Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
2019-11-27T07:50:20.775462213Z  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
2019-11-27T07:50:20.77546463Z   at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
2019-11-27T07:50:20.775467079Z  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
2019-11-27T07:50:20.775469526Z  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
2019-11-27T07:50:20.775471943Z  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
2019-11-27T07:50:20.775474381Z  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2019-11-27T07:50:20.775476823Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:50:20.77547925Z   at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:50:20.775482031Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:50:20.775484478Z  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
2019-11-27T07:50:20.775486927Z  at org.eclipse.jetty.server.Server.handle(Server.java:499)
2019-11-27T07:50:20.775489358Z  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
2019-11-27T07:50:20.77549178Z   at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
2019-11-27T07:50:20.775496967Z  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
2019-11-27T07:50:20.775499504Z  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
2019-11-27T07:50:20.775502006Z  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
2019-11-27T07:50:20.775505157Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T07:50:20.799717111Z 2019-11-27 07:50:20,799 INFO  UfsJournalLogWriter - Marking /journal/FileSystemMaster/v1/logs/0x175a4a-0x7fffffffffffffff as complete with log entries within [1530442, 1530461).
2019-11-27T07:50:20.799805937Z 2019-11-27 07:50:20,799 INFO  UfsJournalLogWriter - Completing log /journal/FileSystemMaster/v1/logs/0x175a4a-0x7fffffffffffffff with next sequence number 1530461
2019-11-27T07:50:20.80096961Z 2019-11-27 07:50:20,800 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-27T07:50:20.802708683Z 2019-11-27 07:50:20,802 INFO  UfsJournalLogWriter - Created current log file: UfsJournalFile{location=/journal/FileSystemMaster/v1/logs/0x175a5d-0x7fffffffffffffff, start=1530461, end=9223372036854775807, isCheckpoint=false}
2019-11-27T07:50:20.810584852Z 2019-11-27 07:50:20,810 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-27T07:51:15.041592276Z 2019-11-27 07:51:15,041 INFO  ExtensionFactoryRegistry - Loading core jars from /opt/alluxio-2.1.0/lib
2019-11-27T07:51:15.049927523Z 2019-11-27 07:51:15,049 INFO  ExtensionFactoryRegistry - Loading extension jars from /opt/alluxio-2.1.0/extensions
2019-11-27T07:58:08.185241177Z 2019-11-27 07:58:08,184 ERROR COSUnderFileSystem - Failed to create object: /
2019-11-27T07:58:08.185268432Z com.qcloud.cos.exception.CosServiceException: Your previous request to create the named bucket succeeded and you already own it. (Status Code: 409; Error Code: BucketAlreadyOwnedByYou; Request ID: NWRkZTJjOTBfMzA5ZDA4MDlfNzAwXzIxYzQyNmI=); Trace ID: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0NzJmZTI0ZmJhYTZmZjYyNmU5ZGNlZDI5YjkyODkwYjNhN2MyMTYxN2NhOGE5YjMyYjE2MzI4NzljYTllNDc4N2Q=
2019-11-27T07:58:08.185272975Z  at com.qcloud.cos.http.DefaultCosHttpClient.handlerErrorMessage(DefaultCosHttpClient.java:293)
2019-11-27T07:58:08.185276478Z  at com.qcloud.cos.http.DefaultCosHttpClient.exeute(DefaultCosHttpClient.java:385)
2019-11-27T07:58:08.185279721Z  at com.qcloud.cos.COSClient.invoke(COSClient.java:463)
2019-11-27T07:58:08.185282644Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:695)
2019-11-27T07:58:08.185285405Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:750)
2019-11-27T07:58:08.185288167Z  at alluxio.underfs.cos.COSUnderFileSystem.createEmptyObject(COSUnderFileSystem.java:142)
2019-11-27T07:58:08.185291015Z  at alluxio.underfs.ObjectUnderFileSystem.mkdirsInternal(ObjectUnderFileSystem.java:1080)
2019-11-27T07:58:08.185294102Z  at alluxio.underfs.ObjectUnderFileSystem.getObjectListingChunkForPath(ObjectUnderFileSystem.java:946)
2019-11-27T07:58:08.185296918Z  at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:970)
2019-11-27T07:58:08.185308874Z  at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:605)
2019-11-27T07:58:08.185311568Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:584)
2019-11-27T07:58:08.185314194Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:08.185316662Z  at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:949)
2019-11-27T07:58:08.185319127Z  at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:08.185321652Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:374)
2019-11-27T07:58:08.185324292Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:371)
2019-11-27T07:58:08.185326781Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.block(ManagedBlockingUfsForwarder.java:596)
2019-11-27T07:58:08.185330175Z  at alluxio.concurrent.jsr.ForkJoinPool.managedBlock(ForkJoinPool.java:1013)
2019-11-27T07:58:08.185332683Z  at alluxio.concurrent.ForkJoinPoolHelper.safeManagedBlock(ForkJoinPoolHelper.java:41)
2019-11-27T07:58:08.185335245Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.get(ManagedBlockingUfsForwarder.java:582)
2019-11-27T07:58:08.185337752Z  at alluxio.concurrent.ManagedBlockingUfsForwarder.listStatus(ManagedBlockingUfsForwarder.java:376)
2019-11-27T07:58:08.185340317Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataInternal(DefaultFileSystemMaster.java:2401)
2019-11-27T07:58:08.185343007Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataIfNotExist(DefaultFileSystemMaster.java:2650)
2019-11-27T07:58:08.185345623Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:864)
2019-11-27T07:58:08.185348128Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:881)
2019-11-27T07:58:08.185350582Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.lambda$getWebUIBrowse$3(AlluxioMasterRestServiceHandler.java:477)
2019-11-27T07:58:08.185353212Z  at alluxio.RestUtils.call(RestUtils.java:62)
2019-11-27T07:58:08.185355615Z  at alluxio.RestUtils.call(RestUtils.java:79)
2019-11-27T07:58:08.185358028Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.getWebUIBrowse(AlluxioMasterRestServiceHandler.java:357)
2019-11-27T07:58:08.18536057Z   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-11-27T07:58:08.185363163Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-11-27T07:58:08.185365646Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-11-27T07:58:08.185368143Z  at java.lang.reflect.Method.invoke(Method.java:498)
2019-11-27T07:58:08.185370603Z  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2019-11-27T07:58:08.185376748Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2019-11-27T07:58:08.185379413Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2019-11-27T07:58:08.185381992Z  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
2019-11-27T07:58:08.185384903Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2019-11-27T07:58:08.185387523Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2019-11-27T07:58:08.185390185Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2019-11-27T07:58:08.185392681Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2019-11-27T07:58:08.185395278Z  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
2019-11-27T07:58:08.18539775Z   at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2019-11-27T07:58:08.185400204Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2019-11-27T07:58:08.185402665Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2019-11-27T07:58:08.185405158Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2019-11-27T07:58:08.18540777Z   at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2019-11-27T07:58:08.18541029Z   at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2019-11-27T07:58:08.185412804Z  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
2019-11-27T07:58:08.185415442Z  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
2019-11-27T07:58:08.18541805Z   at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
2019-11-27T07:58:08.185420507Z  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
2019-11-27T07:58:08.185423015Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
2019-11-27T07:58:08.185425458Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
2019-11-27T07:58:08.185427896Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
2019-11-27T07:58:08.185430363Z  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
2019-11-27T07:58:08.185432819Z  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
2019-11-27T07:58:08.185435384Z  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
2019-11-27T07:58:08.185440868Z  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
2019-11-27T07:58:08.185443385Z  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
2019-11-27T07:58:08.185445825Z  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2019-11-27T07:58:08.185448285Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:08.185450777Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:08.185453537Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:08.185456Z     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
2019-11-27T07:58:08.185458569Z  at org.eclipse.jetty.server.Server.handle(Server.java:499)
2019-11-27T07:58:08.185460998Z  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
2019-11-27T07:58:08.185463514Z  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
2019-11-27T07:58:08.185465959Z  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
2019-11-27T07:58:08.18546839Z   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
2019-11-27T07:58:08.18547088Z   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
2019-11-27T07:58:08.185473365Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T07:58:23.006374555Z 2019-11-27 07:58:23,005 ERROR COSUnderFileSystem - Failed to create object: /
2019-11-27T07:58:23.006400432Z com.qcloud.cos.exception.CosServiceException: Your previous request to create the named bucket succeeded and you already own it. (Status Code: 409; Error Code: BucketAlreadyOwnedByYou; Request ID: NWRkZTJjOWZfMmE5ZDA4MDlfNDAwM18yMWYxNWNi); Trace ID: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0NzJmZTI0ZmJhYTZmZjYyNmU5ZGNlZDI5YjkyODkwYjNhN2MyMTYxN2NhOGE5YjMyYjE2MzI4NzljYTllNDc4N2Q=
2019-11-27T07:58:23.006405436Z  at com.qcloud.cos.http.DefaultCosHttpClient.handlerErrorMessage(DefaultCosHttpClient.java:293)
2019-11-27T07:58:23.006409209Z  at com.qcloud.cos.http.DefaultCosHttpClient.exeute(DefaultCosHttpClient.java:385)
2019-11-27T07:58:23.006412428Z  at com.qcloud.cos.COSClient.invoke(COSClient.java:463)
2019-11-27T07:58:23.006415555Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:695)
2019-11-27T07:58:23.006418464Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:750)
2019-11-27T07:58:23.006421518Z  at alluxio.underfs.cos.COSUnderFileSystem.createEmptyObject(COSUnderFileSystem.java:142)
2019-11-27T07:58:23.006424487Z  at alluxio.underfs.ObjectUnderFileSystem.mkdirsInternal(ObjectUnderFileSystem.java:1080)
2019-11-27T07:58:23.006427558Z  at alluxio.underfs.ObjectUnderFileSystem.getObjectListingChunkForPath(ObjectUnderFileSystem.java:946)
2019-11-27T07:58:23.006430572Z  at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:970)
2019-11-27T07:58:23.006443008Z  at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:605)
2019-11-27T07:58:23.006446225Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:584)
2019-11-27T07:58:23.006449201Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:23.00645221Z   at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:949)
2019-11-27T07:58:23.006455181Z  at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:23.006458183Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:374)
2019-11-27T07:58:23.006461519Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:371)
2019-11-27T07:58:23.006464528Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.block(ManagedBlockingUfsForwarder.java:596)
2019-11-27T07:58:23.006468498Z  at alluxio.concurrent.jsr.ForkJoinPool.managedBlock(ForkJoinPool.java:1013)
2019-11-27T07:58:23.006471448Z  at alluxio.concurrent.ForkJoinPoolHelper.safeManagedBlock(ForkJoinPoolHelper.java:41)
2019-11-27T07:58:23.006474497Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.get(ManagedBlockingUfsForwarder.java:582)
2019-11-27T07:58:23.006477469Z  at alluxio.concurrent.ManagedBlockingUfsForwarder.listStatus(ManagedBlockingUfsForwarder.java:376)
2019-11-27T07:58:23.00648049Z   at alluxio.master.file.DefaultFileSystemMaster.loadMetadataInternal(DefaultFileSystemMaster.java:2401)
2019-11-27T07:58:23.006483446Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataIfNotExist(DefaultFileSystemMaster.java:2650)
2019-11-27T07:58:23.006486417Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:864)
2019-11-27T07:58:23.006489445Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:881)
2019-11-27T07:58:23.006492359Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.lambda$getWebUIBrowse$3(AlluxioMasterRestServiceHandler.java:477)
2019-11-27T07:58:23.006495474Z  at alluxio.RestUtils.call(RestUtils.java:62)
2019-11-27T07:58:23.006498359Z  at alluxio.RestUtils.call(RestUtils.java:79)
2019-11-27T07:58:23.006501194Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.getWebUIBrowse(AlluxioMasterRestServiceHandler.java:357)
2019-11-27T07:58:23.006504195Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-11-27T07:58:23.006507324Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-11-27T07:58:23.006510396Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-11-27T07:58:23.006513396Z  at java.lang.reflect.Method.invoke(Method.java:498)
2019-11-27T07:58:23.006517598Z  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2019-11-27T07:58:23.006523937Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2019-11-27T07:58:23.00652712Z   at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2019-11-27T07:58:23.00653038Z   at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
2019-11-27T07:58:23.006533891Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2019-11-27T07:58:23.00653702Z   at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2019-11-27T07:58:23.006540056Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2019-11-27T07:58:23.006543119Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2019-11-27T07:58:23.006546095Z  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
2019-11-27T07:58:23.006549113Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2019-11-27T07:58:23.006552275Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2019-11-27T07:58:23.006555193Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2019-11-27T07:58:23.006558092Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2019-11-27T07:58:23.006560993Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2019-11-27T07:58:23.006572791Z  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2019-11-27T07:58:23.006575939Z  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
2019-11-27T07:58:23.006578918Z  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
2019-11-27T07:58:23.006581951Z  at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
2019-11-27T07:58:23.006584915Z  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
2019-11-27T07:58:23.00658789Z   at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
2019-11-27T07:58:23.006590841Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
2019-11-27T07:58:23.006593851Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
2019-11-27T07:58:23.006596939Z  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
2019-11-27T07:58:23.006599833Z  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
2019-11-27T07:58:23.006602743Z  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
2019-11-27T07:58:23.006608827Z  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
2019-11-27T07:58:23.006611962Z  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
2019-11-27T07:58:23.006615229Z  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2019-11-27T07:58:23.006618192Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:23.006621129Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:23.006624505Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:23.006627463Z  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
2019-11-27T07:58:23.006630482Z  at org.eclipse.jetty.server.Server.handle(Server.java:499)
2019-11-27T07:58:23.006633846Z  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
2019-11-27T07:58:23.006636778Z  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
2019-11-27T07:58:23.006639946Z  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
2019-11-27T07:58:23.006642867Z  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
2019-11-27T07:58:23.006645923Z  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
2019-11-27T07:58:23.006648971Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T07:58:25.341498026Z 2019-11-27 07:58:25,340 ERROR COSUnderFileSystem - Failed to create object: /
2019-11-27T07:58:25.34153335Z com.qcloud.cos.exception.CosServiceException: Your previous request to create the named bucket succeeded and you already own it. (Status Code: 409; Error Code: BucketAlreadyOwnedByYou; Request ID: NWRkZTJjYTFfMmE5ZDA4MDlfNDAwMl8yMGQ0ZTgx); Trace ID: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0NzJmZTI0ZmJhYTZmZjYyNmU5ZGNlZDI5YjkyODkwYjNhN2MyMTYxN2NhOGE5YjMyYjE2MzI4NzljYTllNDc4N2Q=
2019-11-27T07:58:25.341538611Z  at com.qcloud.cos.http.DefaultCosHttpClient.handlerErrorMessage(DefaultCosHttpClient.java:293)
2019-11-27T07:58:25.341542459Z  at com.qcloud.cos.http.DefaultCosHttpClient.exeute(DefaultCosHttpClient.java:385)
2019-11-27T07:58:25.3415456Z    at com.qcloud.cos.COSClient.invoke(COSClient.java:463)
2019-11-27T07:58:25.341548748Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:695)
2019-11-27T07:58:25.341551725Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:750)
2019-11-27T07:58:25.341554633Z  at alluxio.underfs.cos.COSUnderFileSystem.createEmptyObject(COSUnderFileSystem.java:142)
2019-11-27T07:58:25.341557588Z  at alluxio.underfs.ObjectUnderFileSystem.mkdirsInternal(ObjectUnderFileSystem.java:1080)
2019-11-27T07:58:25.341560558Z  at alluxio.underfs.ObjectUnderFileSystem.getObjectListingChunkForPath(ObjectUnderFileSystem.java:946)
2019-11-27T07:58:25.341563627Z  at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:970)
2019-11-27T07:58:25.341575344Z  at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:605)
2019-11-27T07:58:25.341578534Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:584)
2019-11-27T07:58:25.34158154Z   at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:25.341584638Z  at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:949)
2019-11-27T07:58:25.341587562Z  at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:25.341590539Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:374)
2019-11-27T07:58:25.341593543Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:371)
2019-11-27T07:58:25.341596604Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.block(ManagedBlockingUfsForwarder.java:596)
2019-11-27T07:58:25.341600509Z  at alluxio.concurrent.jsr.ForkJoinPool.managedBlock(ForkJoinPool.java:1013)
2019-11-27T07:58:25.341603669Z  at alluxio.concurrent.ForkJoinPoolHelper.safeManagedBlock(ForkJoinPoolHelper.java:41)
2019-11-27T07:58:25.341606624Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.get(ManagedBlockingUfsForwarder.java:582)
2019-11-27T07:58:25.341609625Z  at alluxio.concurrent.ManagedBlockingUfsForwarder.listStatus(ManagedBlockingUfsForwarder.java:376)
2019-11-27T07:58:25.341612704Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataInternal(DefaultFileSystemMaster.java:2401)
2019-11-27T07:58:25.341615744Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataIfNotExist(DefaultFileSystemMaster.java:2650)
2019-11-27T07:58:25.341618708Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:864)
2019-11-27T07:58:25.341621646Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:881)
2019-11-27T07:58:25.341624683Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.lambda$getWebUIBrowse$3(AlluxioMasterRestServiceHandler.java:477)
2019-11-27T07:58:25.34162771Z   at alluxio.RestUtils.call(RestUtils.java:62)
2019-11-27T07:58:25.34163065Z   at alluxio.RestUtils.call(RestUtils.java:79)
2019-11-27T07:58:25.341633533Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.getWebUIBrowse(AlluxioMasterRestServiceHandler.java:357)
2019-11-27T07:58:25.341636549Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-11-27T07:58:25.341639479Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-11-27T07:58:25.341642444Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-11-27T07:58:25.34164542Z   at java.lang.reflect.Method.invoke(Method.java:498)
2019-11-27T07:58:25.341648365Z  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2019-11-27T07:58:25.341654725Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2019-11-27T07:58:25.341657871Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2019-11-27T07:58:25.341661014Z  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
2019-11-27T07:58:25.341664432Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2019-11-27T07:58:25.341667523Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2019-11-27T07:58:25.341670656Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2019-11-27T07:58:25.341673648Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2019-11-27T07:58:25.341676602Z  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
2019-11-27T07:58:25.341679551Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2019-11-27T07:58:25.341682468Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2019-11-27T07:58:25.34168537Z   at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2019-11-27T07:58:25.341688301Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2019-11-27T07:58:25.341691224Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2019-11-27T07:58:25.341694095Z  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2019-11-27T07:58:25.341697074Z  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
2019-11-27T07:58:25.341700034Z  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
2019-11-27T07:58:25.341703004Z  at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
2019-11-27T07:58:25.341705936Z  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
2019-11-27T07:58:25.341708935Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
2019-11-27T07:58:25.341711977Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
2019-11-27T07:58:25.341714892Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
2019-11-27T07:58:25.341717802Z  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
2019-11-27T07:58:25.341720716Z  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
2019-11-27T07:58:25.341723618Z  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
2019-11-27T07:58:25.341729923Z  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
2019-11-27T07:58:25.341732958Z  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
2019-11-27T07:58:25.341735926Z  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2019-11-27T07:58:25.341738844Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:25.341741802Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:25.341745082Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:25.341747999Z  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
2019-11-27T07:58:25.341751215Z  at org.eclipse.jetty.server.Server.handle(Server.java:499)
2019-11-27T07:58:25.34175411Z   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
2019-11-27T07:58:25.341757109Z  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
2019-11-27T07:58:25.341760036Z  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
2019-11-27T07:58:25.341762935Z  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
2019-11-27T07:58:25.341765843Z  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
2019-11-27T07:58:25.341768907Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T07:58:28.352498022Z 2019-11-27 07:58:28,351 ERROR COSUnderFileSystem - Failed to create object: /
2019-11-27T07:58:28.352526039Z com.qcloud.cos.exception.CosServiceException: Your previous request to create the named bucket succeeded and you already own it. (Status Code: 409; Error Code: BucketAlreadyOwnedByYou; Request ID: NWRkZTJjYTRfMmE5ZDA4MDlfM2ZlYl8yMTliM2Q5); Trace ID: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0NzJmZTI0ZmJhYTZmZjYyNmU5ZGNlZDI5YjkyODkwYjNhN2MyMTYxN2NhOGE5YjMyYjE2MzI4NzljYTllNDc4N2Q=
2019-11-27T07:58:28.352530064Z  at com.qcloud.cos.http.DefaultCosHttpClient.handlerErrorMessage(DefaultCosHttpClient.java:293)
2019-11-27T07:58:28.352532952Z  at com.qcloud.cos.http.DefaultCosHttpClient.exeute(DefaultCosHttpClient.java:385)
2019-11-27T07:58:28.35253566Z   at com.qcloud.cos.COSClient.invoke(COSClient.java:463)
2019-11-27T07:58:28.352538202Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:695)
2019-11-27T07:58:28.352540643Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:750)
2019-11-27T07:58:28.352543106Z  at alluxio.underfs.cos.COSUnderFileSystem.createEmptyObject(COSUnderFileSystem.java:142)
2019-11-27T07:58:28.352545689Z  at alluxio.underfs.ObjectUnderFileSystem.mkdirsInternal(ObjectUnderFileSystem.java:1080)
2019-11-27T07:58:28.352548254Z  at alluxio.underfs.ObjectUnderFileSystem.getObjectListingChunkForPath(ObjectUnderFileSystem.java:946)
2019-11-27T07:58:28.352551523Z  at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:970)
2019-11-27T07:58:28.35256342Z   at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:605)
2019-11-27T07:58:28.352566078Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:584)
2019-11-27T07:58:28.35256868Z   at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:28.352571149Z  at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:949)
2019-11-27T07:58:28.352573697Z  at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:581)
2019-11-27T07:58:28.35257621Z   at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:374)
2019-11-27T07:58:28.352578806Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:371)
2019-11-27T07:58:28.352581269Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.block(ManagedBlockingUfsForwarder.java:596)
2019-11-27T07:58:28.352584496Z  at alluxio.concurrent.jsr.ForkJoinPool.managedBlock(ForkJoinPool.java:1013)
2019-11-27T07:58:28.352586956Z  at alluxio.concurrent.ForkJoinPoolHelper.safeManagedBlock(ForkJoinPoolHelper.java:41)
2019-11-27T07:58:28.352589492Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.get(ManagedBlockingUfsForwarder.java:582)
2019-11-27T07:58:28.352592018Z  at alluxio.concurrent.ManagedBlockingUfsForwarder.listStatus(ManagedBlockingUfsForwarder.java:376)
2019-11-27T07:58:28.352594472Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataInternal(DefaultFileSystemMaster.java:2401)
2019-11-27T07:58:28.352596978Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataIfNotExist(DefaultFileSystemMaster.java:2650)
2019-11-27T07:58:28.352599561Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:864)
2019-11-27T07:58:28.352602042Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:881)
2019-11-27T07:58:28.352604465Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.lambda$getWebUIBrowse$3(AlluxioMasterRestServiceHandler.java:477)
2019-11-27T07:58:28.352606977Z  at alluxio.RestUtils.call(RestUtils.java:62)
2019-11-27T07:58:28.352609459Z  at alluxio.RestUtils.call(RestUtils.java:79)
2019-11-27T07:58:28.352611816Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.getWebUIBrowse(AlluxioMasterRestServiceHandler.java:357)
2019-11-27T07:58:28.352614327Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-11-27T07:58:28.352616777Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-11-27T07:58:28.352619243Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-11-27T07:58:28.352621692Z  at java.lang.reflect.Method.invoke(Method.java:498)
2019-11-27T07:58:28.352624129Z  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2019-11-27T07:58:28.352630085Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2019-11-27T07:58:28.352632757Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2019-11-27T07:58:28.352635338Z  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
2019-11-27T07:58:28.352638207Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2019-11-27T07:58:28.352640799Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2019-11-27T07:58:28.352643283Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2019-11-27T07:58:28.352645751Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2019-11-27T07:58:28.352648217Z  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
2019-11-27T07:58:28.352650659Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2019-11-27T07:58:28.352653107Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2019-11-27T07:58:28.352655525Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2019-11-27T07:58:28.352657982Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2019-11-27T07:58:28.3526605Z    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2019-11-27T07:58:28.352662972Z  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2019-11-27T07:58:28.352665426Z  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
2019-11-27T07:58:28.352671728Z  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
2019-11-27T07:58:28.352674267Z  at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
2019-11-27T07:58:28.352676712Z  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
2019-11-27T07:58:28.352679152Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
2019-11-27T07:58:28.352681614Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
2019-11-27T07:58:28.352684065Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
2019-11-27T07:58:28.352686523Z  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
2019-11-27T07:58:28.352688946Z  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
2019-11-27T07:58:28.352691398Z  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
2019-11-27T07:58:28.35269726Z   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
2019-11-27T07:58:28.352699836Z  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
2019-11-27T07:58:28.352702309Z  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2019-11-27T07:58:28.352704759Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:28.352707194Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:28.352709964Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T07:58:28.352712412Z  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
2019-11-27T07:58:28.352714879Z  at org.eclipse.jetty.server.Server.handle(Server.java:499)
2019-11-27T07:58:28.352717285Z  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
2019-11-27T07:58:28.352719725Z  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
2019-11-27T07:58:28.352722154Z  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
2019-11-27T07:58:28.352724586Z  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
2019-11-27T07:58:28.352727241Z  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
2019-11-27T07:58:28.352729719Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T08:25:07.790425703Z 2019-11-27 08:25:07,790 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T08:25:07.790459481Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T09:25:07.787818381Z 2019-11-27 09:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T09:25:07.787843494Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T09:47:48.466711383Z 2019-11-27 09:47:48,465 ERROR COSUnderFileSystem - Failed to create object: /
2019-11-27T09:47:48.466740659Z com.qcloud.cos.exception.CosServiceException: Your previous request to create the named bucket succeeded and you already own it. (Status Code: 409; Error Code: BucketAlreadyOwnedByYou; Request ID: NWRkZTQ2NDRfMzRhNzAzMDlfNmZhZV8yMThhZmMw); Trace ID: OGVmYzZiMmQzYjA2OWNhODk0NTRkMTBiOWVmMDAxODczNTBmNjMwZmQ0MTZkMjg0NjlkNTYyNmY4ZTRkZTk0NzJmZTI0ZmJhYTZmZjYyNmU5ZGNlZDI5YjkyODkwYjNhN2MyMTYxN2NhOGE5YjMyYjE2MzI4NzljYTllNDc4N2Q=
2019-11-27T09:47:48.466755354Z  at com.qcloud.cos.http.DefaultCosHttpClient.handlerErrorMessage(DefaultCosHttpClient.java:293)
2019-11-27T09:47:48.466758491Z  at com.qcloud.cos.http.DefaultCosHttpClient.exeute(DefaultCosHttpClient.java:385)
2019-11-27T09:47:48.466761148Z  at com.qcloud.cos.COSClient.invoke(COSClient.java:463)
2019-11-27T09:47:48.466763719Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:695)
2019-11-27T09:47:48.466766117Z  at com.qcloud.cos.COSClient.putObject(COSClient.java:750)
2019-11-27T09:47:48.466768527Z  at alluxio.underfs.cos.COSUnderFileSystem.createEmptyObject(COSUnderFileSystem.java:142)
2019-11-27T09:47:48.466770999Z  at alluxio.underfs.ObjectUnderFileSystem.mkdirsInternal(ObjectUnderFileSystem.java:1080)
2019-11-27T09:47:48.46677358Z   at alluxio.underfs.ObjectUnderFileSystem.getObjectListingChunkForPath(ObjectUnderFileSystem.java:946)
2019-11-27T09:47:48.466776194Z  at alluxio.underfs.ObjectUnderFileSystem.listInternal(ObjectUnderFileSystem.java:970)
2019-11-27T09:47:48.466778628Z  at alluxio.underfs.ObjectUnderFileSystem.listStatus(ObjectUnderFileSystem.java:605)
2019-11-27T09:47:48.466781049Z  at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:584)
2019-11-27T09:47:48.4667838Z    at alluxio.underfs.UnderFileSystemWithLogging$32.call(UnderFileSystemWithLogging.java:581)
2019-11-27T09:47:48.466786287Z  at alluxio.underfs.UnderFileSystemWithLogging.call(UnderFileSystemWithLogging.java:949)
2019-11-27T09:47:48.466788718Z  at alluxio.underfs.UnderFileSystemWithLogging.listStatus(UnderFileSystemWithLogging.java:581)
2019-11-27T09:47:48.466791235Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:374)
2019-11-27T09:47:48.466793838Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$27.execute(ManagedBlockingUfsForwarder.java:371)
2019-11-27T09:47:48.466796298Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.block(ManagedBlockingUfsForwarder.java:596)
2019-11-27T09:47:48.466799421Z  at alluxio.concurrent.jsr.ForkJoinPool.managedBlock(ForkJoinPool.java:1013)
2019-11-27T09:47:48.466801896Z  at alluxio.concurrent.ForkJoinPoolHelper.safeManagedBlock(ForkJoinPoolHelper.java:41)
2019-11-27T09:47:48.466804455Z  at alluxio.concurrent.ManagedBlockingUfsForwarder$ManagedBlockingUfsMethod.get(ManagedBlockingUfsForwarder.java:582)
2019-11-27T09:47:48.466806966Z  at alluxio.concurrent.ManagedBlockingUfsForwarder.listStatus(ManagedBlockingUfsForwarder.java:376)
2019-11-27T09:47:48.466809427Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataInternal(DefaultFileSystemMaster.java:2401)
2019-11-27T09:47:48.466811989Z  at alluxio.master.file.DefaultFileSystemMaster.loadMetadataIfNotExist(DefaultFileSystemMaster.java:2650)
2019-11-27T09:47:48.46681446Z   at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:864)
2019-11-27T09:47:48.466820185Z  at alluxio.master.file.DefaultFileSystemMaster.listStatus(DefaultFileSystemMaster.java:881)
2019-11-27T09:47:48.466822763Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.lambda$getWebUIBrowse$3(AlluxioMasterRestServiceHandler.java:477)
2019-11-27T09:47:48.466825327Z  at alluxio.RestUtils.call(RestUtils.java:62)
2019-11-27T09:47:48.466827736Z  at alluxio.RestUtils.call(RestUtils.java:79)
2019-11-27T09:47:48.466830092Z  at alluxio.master.meta.AlluxioMasterRestServiceHandler.getWebUIBrowse(AlluxioMasterRestServiceHandler.java:357)
2019-11-27T09:47:48.466832618Z  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2019-11-27T09:47:48.466835174Z  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2019-11-27T09:47:48.466837644Z  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2019-11-27T09:47:48.466840304Z  at java.lang.reflect.Method.invoke(Method.java:498)
2019-11-27T09:47:48.466842841Z  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
2019-11-27T09:47:48.466845394Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
2019-11-27T09:47:48.466848052Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
2019-11-27T09:47:48.466850818Z  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
2019-11-27T09:47:48.466853723Z  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
2019-11-27T09:47:48.466856481Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
2019-11-27T09:47:48.466859037Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
2019-11-27T09:47:48.466861667Z  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
2019-11-27T09:47:48.466864186Z  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:317)
2019-11-27T09:47:48.466866677Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
2019-11-27T09:47:48.466869093Z  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
2019-11-27T09:47:48.466871577Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
2019-11-27T09:47:48.46687401Z   at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
2019-11-27T09:47:48.466876422Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
2019-11-27T09:47:48.466878834Z  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
2019-11-27T09:47:48.466885258Z  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:298)
2019-11-27T09:47:48.466887875Z  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
2019-11-27T09:47:48.46689037Z   at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:471)
2019-11-27T09:47:48.466892799Z  at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:425)
2019-11-27T09:47:48.466895265Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:383)
2019-11-27T09:47:48.466897706Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:336)
2019-11-27T09:47:48.466900142Z  at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:223)
2019-11-27T09:47:48.466902603Z  at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
2019-11-27T09:47:48.466905026Z  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
2019-11-27T09:47:48.466907659Z  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
2019-11-27T09:47:48.466910107Z  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
2019-11-27T09:47:48.466912538Z  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
2019-11-27T09:47:48.466914993Z  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
2019-11-27T09:47:48.466917415Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T09:47:48.466919847Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T09:47:48.466922596Z  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
2019-11-27T09:47:48.46692505Z   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
2019-11-27T09:47:48.466927528Z  at org.eclipse.jetty.server.Server.handle(Server.java:499)
2019-11-27T09:47:48.466929935Z  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
2019-11-27T09:47:48.466932355Z  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
2019-11-27T09:47:48.46693723Z   at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
2019-11-27T09:47:48.466939821Z  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
2019-11-27T09:47:48.466942393Z  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
2019-11-27T09:47:48.466944849Z  at java.lang.Thread.run(Thread.java:748)
2019-11-27T09:48:00.330537835Z 2019-11-27 09:48:00,329 INFO  UfsJournalLogWriter - Created current log file: UfsJournalFile{location=/journal/BlockMaster/v1/logs/0x7a878-0x7fffffffffffffff, start=501880, end=9223372036854775807, isCheckpoint=false}
2019-11-27T10:25:07.788309083Z 2019-11-27 10:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T10:25:07.788343396Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T11:25:07.787743855Z 2019-11-27 11:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T11:25:07.787774044Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T12:25:07.788254961Z 2019-11-27 12:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T12:25:07.788274876Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T13:25:07.787641969Z 2019-11-27 13:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T13:25:07.787675777Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T14:25:07.788029996Z 2019-11-27 14:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T14:25:07.788050805Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T15:25:07.788409192Z 2019-11-27 15:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T15:25:07.788428183Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T16:25:07.787827714Z 2019-11-27 16:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T16:25:07.787853618Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T17:25:07.788173336Z 2019-11-27 17:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T17:25:07.788211606Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T18:25:07.787553745Z 2019-11-27 18:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T18:25:07.787573257Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T19:25:07.787942508Z 2019-11-27 19:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T19:25:07.787967046Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T19:31:35.238638613Z 2019-11-27 19:31:35,237 WARN  SaslStreamServerDriver - Error received for channel: aa908e7a-f809-4601-80c2-26c82b663286. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T19:31:35.239127283Z 2019-11-27 19:31:35,237 WARN  SaslStreamServerDriver - Error received for channel: 081c6cf2-916d-4c5e-aa3f-0db2cefa5645. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T19:31:35.239227066Z 2019-11-27 19:31:35,237 WARN  SaslStreamServerDriver - Error received for channel: 9c859bbe-777e-467f-a828-4fc1be2fec4a. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T19:31:35.23932468Z 2019-11-27 19:31:35,237 WARN  SaslStreamServerDriver - Error received for channel: 0c26242f-13d8-49a9-b60d-826a724c9515. Error: io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2019-11-27T20:25:07.788283855Z 2019-11-27 20:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T20:25:07.788309166Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T21:25:07.787618597Z 2019-11-27 21:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T21:25:07.787650191Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T22:25:07.787923941Z 2019-11-27 22:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T22:25:07.787948133Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-27T23:25:07.788286945Z 2019-11-27 23:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-27T23:25:07.788313958Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T00:25:07.787565242Z 2019-11-28 00:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T00:25:07.787590164Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T01:25:07.787953694Z 2019-11-28 01:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T01:25:07.787999188Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T02:25:07.788274907Z 2019-11-28 02:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T02:25:07.78830222Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T03:25:07.787594704Z 2019-11-28 03:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T03:25:07.787614902Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T04:25:07.787937813Z 2019-11-28 04:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T04:25:07.787958267Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T05:25:07.788284818Z 2019-11-28 05:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T05:25:07.788305204Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T06:25:07.787595998Z 2019-11-28 06:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T06:25:07.787617017Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T07:25:07.78791197Z 2019-11-28 07:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T07:25:07.787937773Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T08:25:07.788312187Z 2019-11-28 08:25:07,788 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T08:25:07.788343243Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T09:25:07.787661822Z 2019-11-28 09:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T09:25:07.787679957Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]
2019-11-28T10:25:07.787995002Z 2019-11-28 10:25:07,787 WARN  ServerConfigurationChecker - Inconsistent configuration detected. Only a limited set of inconsistent configuration will be shown here. For details, please visit Alluxio web UI or run fsadmin doctor CLI.
2019-11-28T10:25:07.788021899Z Warnings: [InconsistentProperty{key=alluxio.work.dir, values=/opt/alluxio-2.1.0 (10.0.68.212:29999), ${alluxio.home} (alluxio-master-0:19998)}, InconsistentProperty{key=alluxio.site.conf.dir, values=${alluxio.conf.dir}/,/home/alluxio/.alluxio/,/etc/alluxio/ (alluxio-master-0:19998), /opt/alluxio-2.1.0/conf/,/home/alluxio/.alluxio/,/etc/alluxio/ (10.0.68.212:29999)}]

通过排查，我们发现fuse一直无法启动，是因为宿主机器上的挂载点还存在，一直没有被umount image 这就导致fuse一直挂载不上/alluxio-fuse，导致pod一直启动失败，我认为这一块可以加个init容器，启动前检测挂载点，如果存在先umount。

不过问题的根本还是在于master无缘无故挂掉，导致fuse也突然挂掉，无法执行prestop里的命令。昨天已经给matser调成了2G，也是会挂，现在也没有上线，只是我自己测，大半夜也没有大量使用请求，不知道是因为什么。 image

we7team commented 4 years ago

使用mountPropagation: Bidirectional会造成不少问题，非正常重启导致宿主机器上的挂载点还存在。然后fuse服务就无法挂载上去了。另外就是如果其他pod使用了宿主上的这个共享目录，一旦fuse重启了，其他pod上的挂载点就全部出错了。 image image

我感觉想从根本上解决，还得让k8s支持alluxio的fuse协议进行挂载，就像支持nfs直接挂载一样。

we7team commented 4 years ago

或者alluxio多支持一种协议，nfs，这样的话就完美与k8s兼容了，k8s的存储类型支持nfs协议，也是比较通用的解决方案。之前提过一个相关的issue：https://github.com/Alluxio/alluxio/issues/8746

we7team commented 4 years ago

参考：https://github.com/dCache/nfs4j

saberrey commented 4 years ago

使用mountPropagation: Bidirectional会造成不少问题，非正常重启导致宿主机器上的挂载点还存在。然后fuse服务就无法挂载上去了。另外就是如果其他pod使用了宿主上的这个共享目录，一旦fuse重启了，其他pod上的挂载点就全部出错了。 image image

我感觉想从根本上解决，还得让k8s支持alluxio的fuse协议进行挂载，就像支持nfs直接挂载一样。

我与遇到了这个问题，老哥可以具体解释一下为什么使用Bidirectional会造成fuse的这种问题吗

yuzhu commented 3 years ago

please try with latest version of Alluxio, as we have made significant changes to use JNI-fuse