search

smart-edge-open / edgeapps

Applications that can be onboarded to an Intel® Smart Edge Open edge node.
Apache License 2.0
51 stars 73 forks source link

Issue in kafka-service and live-service when deployed CDN-Transcode-Sample #26

Closed pushpraj527 closed 3 years ago

pushpraj527 commented 3 years ago

cdn-service.log kafka-service.log live-services.log redis-service.log zookeeper-service.log

We tried to deploy CDN-Transcode-Sample, we are seeing issue with kafka-service and live-service. Kafka-service and live-service pod going in a "CrashLoopBackOff" state.

Following is the cluster information and log snippet. Let me know what other information required for debugging.

[root@openmaster CDN-Transcode-Sample]# kubectl get pods NAME READY STATUS RESTARTS AGE cdn-service-5665d5459-frjt8 1/1 Running 0 92m kafka-service-858d7c4d89-7m55l 0/1 CrashLoopBackOff 21 92m live-service-0-7b76467668-8bdhm 0/1 CrashLoopBackOff 22 92m redis-service-5d4bddb55-nmdws 1/1 Running 0 92m vod-service-8495d66cc5-52pdj 1/1 Running 0 92m vod-service-8495d66cc5-z5nxh 1/1 Running 0 92m zookeeper-service-65bc7f7bbc-9qqqm 1/1 Running 0 92m

[root@openmaster CDN-Transcode-Sample]# kubectl logs cdn-service-5665d5459-frjt8 Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='ANSI_X3.4-1968'> BrokenPipeError: [Errno 32] Broken pipe

[root@openmaster CDN-Transcode-Sample]# kubectl logs kafka-service-858d7c4d89-7m55l [2020-09-18 16:46:52,328] INFO Client environment:os.version=3.10.0-957.12.2.el7.x86_64 (org.apache.zookeeper.ZooKeeper[0/1890] [2020-09-18 16:46:52,328] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,329] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,329] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,329] INFO Client environment:os.memory.free=976MB (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,329] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,329] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,334] INFO Initiating client connection, connectString=zookeeper-service:2181 sessionTimeout=18000 watcher= kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@7722c3c3 (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:46:52,345] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket) [2020-09-18 16:46:52,363] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn) [2020-09-18 16:46:52,369] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient) [2020-09-18 16:46:52,397] INFO Opening socket connection to server zookeeper-service/10.110.225.58:2181. Will not attempt to au thenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) waiting for kafka to be ready waiting for kafka to be ready [2020-09-18 16:47:10,371] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient) [2020-09-18 16:47:10,381] WARN Client session timed out, have not heard from server in 18017ms for sessionid 0x0 (org.apache.zo okeeper.ClientCnxn) [2020-09-18 16:47:10,492] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper) [2020-09-18 16:47:10,548] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn) [2020-09-18 16:47:10,551] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient) [2020-09-18 16:47:10,559] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262) at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:119) at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1865) at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:419) at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:444) at kafka.server.KafkaServer.startup(KafkaServer.scala:222) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:82) at kafka.Kafka.main(Kafka.scala) [2020-09-18 16:47:10,565] INFO shutting down (kafka.server.KafkaServer) [2020-09-18 16:47:10,577] INFO shut down completed (kafka.server.KafkaServer) [2020-09-18 16:47:10,578] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable) [2020-09-18 16:47:10,580] INFO shutting down (kafka.server.KafkaServer)

[root@openmaster CDN-Transcode-Sample]# kubectl logs live-service-0-7b76467668-8bdhm Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/var/www/archive/bbb_sunflower_1080p_30fps_normal.mp4': Metadata: major_brand : isom minor_version : 1 compatible_brands: isomavc1 creation_time : 2013-12-16T17:44:39.000000Z title : Big Buck Bunny, Sunflower version artist : Blender Foundation 2008, Janus Bager Kristensen 2013 comment : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net genre : Animation composer : Sacha Goedegebure Duration: 00:10:34.53, start: 0.000000, bitrate: 3481 kb/s Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 2998 kb/s, 30 fps, 30 tbr, 30k tbn, 60 tbc (default) Metadata: creation_time : 2013-12-16T17:44:39.000000Z handler_name : GPAC ISO Video Handler Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default) Metadata: creation_time : 2013-12-16T17:44:42.000000Z handler_name : GPAC ISO Audio Handler Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default) Metadata: creation_time : 2013-12-16T17:44:42.000000Z handler_name : GPAC ISO Audio Handler Side data: audio service type: main [tcp @ 0x55809c7e60c0] Connection to tcp://cdn-service:1935 failed: Connection timed out [rtmp @ 0x55809c73c880] Cannot open connection tcp://cdn-service:1935 rtmp://cdn-service/dash/media_0_0: Connection timed out

[root@openmaster CDN-Transcode-Sample]# kubectl logs zookeeper-service-65bc7f7bbc-9qqqm 2020-09-18 15:15:11,543 [myid:1] - INFO [main:ZKDatabase@132] - zookeeper.snapshotSizeFactor = 0.33 2020-09-18 15:15:11,543 [myid:1] - INFO [main:ZKDatabase@152] - zookeeper.commitLogCount=500 2020-09-18 15:15:11,552 [myid:1] - INFO [main:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED 2020-09-18 15:15:11,553 [myid:1] - INFO [main:FileTxnSnapLog@470] - Snapshotting: 0x0 to /data/version-2/snapshot.0 2020-09-18 15:15:11,558 [myid:1] - INFO [main:ZKDatabase@289] - Snapshot loaded in 14 ms, highest zxid is 0x0, digest is 1371985504 2020-09-18 15:15:11,558 [myid:1] - INFO [main:FileTxnSnapLog@470] - Snapshotting: 0x0 to /data/version-2/snapshot.0 2020-09-18 15:15:11,559 [myid:1] - INFO [main:ZooKeeperServer@519] - Snapshot taken in 0 ms 2020-09-18 15:15:11,576 [myid:1] - INFO [main:RequestThrottler@74] - zookeeper.request_throttler.shutdownTimeout = 10000 2020-09-18 15:15:11,611 [myid:1] - INFO [main:ContainerManager@83] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 2020-09-18 15:15:11,612 [myid:1] - INFO [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.

pushpraj527 commented 3 years ago

Issue resolved, some networking issue at my end.