livepeer / go-livepeer

Official Go implementation of the Livepeer protocol
http://livepeer.org
MIT License
542 stars 169 forks source link

P3 - ZeroSegment errors still occuring on 0.5.32 #2502

Open stronk-dev opened 2 years ago

stronk-dev commented 2 years ago

single IDR frame as input causing ZeroSegments error Why? We see a few ZeroSegments errors, but we usually recover so not high priority right now.

If you need a live feed of these errors happening, go to https://grafana.stronk.tech/d/71b6OZ0Gz/orchestrator-overview and open the error logs panel.

Leiden (0.5.32-674a9048)


  |   | E0714 08:20:02.616793 2181420 orchestrator.go:582] manifestID=b5ba4c86-e440-4fa4-b92a-3275e0d0f518 seqNo=1 orchSessionID=c6bbb8d0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:19:16.051579 2181420 segment_rpc.go:231] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:19:16.051513 2181420 orchestrator.go:582] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:19:15.953313 2181420 segment_rpc.go:231] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:19:15.953234 2181420 orchestrator.go:582] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:19:15.805626 2181420 segment_rpc.go:231] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=1 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:19:15.805527 2181420 orchestrator.go:582] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=1 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:18:00.687761 2181420 segment_rpc.go:231] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=fa6477e0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:18:00.687705 2181420 orchestrator.go:582] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=fa6477e0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:48.575826 2181420 segment_rpc.go:231] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=2 orchSessionID=8e6edc0b clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:17:48.575764 2181420 orchestrator.go:582] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=2 orchSessionID=8e6edc0b sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=212.102.38.92 Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:48.477612 2181420 segment_rpc.go:231] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=8e6edc0b clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:17:48.477514 2181420 orchestrator.go:582] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=8e6edc0b clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:37.192034 2181420 segment_rpc.go:231] manifestID=dfa79952-e8b7-4e72-b0b9-65076b82f150 seqNo=2 orchSessionID=18012ff1 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:17:37.191966 2181420 orchestrator.go:582] manifestID=dfa79952-e8b7-4e72-b0b9-65076b82f150 seqNo=2 orchSessionID=18012ff1 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:34.324573 2181420 segment_rpc.go:231] manifestID=dfa79952-e8b7-4e72-b0b9-65076b82f150 seqNo=1 orchSessionID=9812682d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"

Chicago (0.5.32)


E0714 08:20:02.616878 2181420 segment_rpc.go:231] manifestID=b5ba4c86-e440-4fa4-b92a-3275e0d0f518 seqNo=1 orchSessionID=c6bbb8d0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
--
  |   | E0714 08:20:02.616793 2181420 orchestrator.go:582] manifestID=b5ba4c86-e440-4fa4-b92a-3275e0d0f518 seqNo=1 orchSessionID=c6bbb8d0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:19:16.051579 2181420 segment_rpc.go:231] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:19:16.051513 2181420 orchestrator.go:582] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:19:15.953313 2181420 segment_rpc.go:231] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:19:15.953234 2181420 orchestrator.go:582] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=2 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:19:15.805626 2181420 segment_rpc.go:231] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=1 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:19:15.805527 2181420 orchestrator.go:582] manifestID=29541aaa-d9c3-4d61-95bf-8381827a36c7 seqNo=1 orchSessionID=eeb6852e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:18:00.687761 2181420 segment_rpc.go:231] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=fa6477e0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:18:00.687705 2181420 orchestrator.go:582] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=fa6477e0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:48.575826 2181420 segment_rpc.go:231] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=2 orchSessionID=8e6edc0b clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:17:48.575764 2181420 orchestrator.go:582] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=2 orchSessionID=8e6edc0b sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=212.102.38.92 Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:48.477612 2181420 segment_rpc.go:231] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=8e6edc0b clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:17:48.477514 2181420 orchestrator.go:582] manifestID=a7b8a8e8-17eb-4773-9e10-5530a94f85a9 seqNo=1 orchSessionID=8e6edc0b clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:37.192034 2181420 segment_rpc.go:231] manifestID=dfa79952-e8b7-4e72-b0b9-65076b82f150 seqNo=2 orchSessionID=18012ff1 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
  |   | E0714 08:17:37.191966 2181420 orchestrator.go:582] manifestID=dfa79952-e8b7-4e72-b0b9-65076b82f150 seqNo=2 orchSessionID=18012ff1 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
  |   | E0714 08:17:34.324573 2181420 segment_rpc.go:231] manifestID=dfa79952-e8b7-4e72-b0b9-65076b82f150 seqNo=1 orchSessionID=9812682d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
Franck-UltimaRatio commented 2 years ago

got tons of them too on 0.5.32, linux T & O split.

Titan-Node commented 2 years ago

Getting a lot too Here are my logs (v6)

I0714 17:34:44.775972 3583963 segment_rpc.go:100] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Received segment dur=1.016s
I0714 17:34:44.776049 3583963 census.go:1136] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentEmerged... duration=1.016
I0714 17:34:44.776342 3583963 orchestrator.go:170] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Receiving ticket sessionID=44543ca5 faceValue=0.05215739712 ETH winProb=0.0000191727 ev=1000000000000.00
I0714 17:34:44.889175 3583963 segment_rpc.go:140] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Downloaded segment dur=112.455488ms
I0714 17:34:44.889239 3583963 census.go:1198] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentDownloaded... dur=112.455488ms
I0714 17:34:44.891255 3583963 orchestrator.go:475] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Starting to transcode segment
I0714 17:34:44.891312 3583963 orchestrator.go:463] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Creating new segment chan
I0714 17:34:44.891338 3583963 orchestrator.go:608] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=212.102.38.92 Starting transcode segment loop for manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 sessionID=44543ca5
I0714 17:34:44.891362 3583963 orchestrator.go:484] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Submitted segment to transcode loop 
I0714 17:34:44.923120 3583963 player.go:105] LPMS got HTTP request @ /stream/44543ca5/2.tempfile
I0714 17:34:45.383235 3583963 ot_rpc.go:414] Downloaded results from remote transcoder=<ip hidden> taskId=63445 dur=24.812µs
I0714 17:34:45.383319 3583963 census.go:1198] Logging SegmentDownloaded... dur=24.812µs
I0714 17:34:45.383397 3583963 orchestrator.go:762] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Successfully received results from remote transcoder=<ip hidden> segments=2 taskId=63445 fname=https://161.35.223.50:8935/stream/44543ca5/2.tempfile dur=491.384908ms
I0714 17:34:45.383421 3583963 orchestrator.go:567] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Transcoding of segment took=491.45058ms
I0714 17:34:45.383440 3583963 census.go:1303] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentTranscode nonce=0 seqNo=2 dur=491.45058ms trusted=true verified=true
E0714 17:34:45.383480 3583963 orchestrator.go:578] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=212.102.38.92 Cannot find transcoded segment for bytes=0
E0714 17:34:45.383714 3583963 segment_rpc.go:234] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
I0714 17:35:45.384420 3583963 orchestrator.go:643] manifestID=703910d3-e4dc-4a47-bf43-d27f6ce238c6 seqNo=2 orchSessionID=44543ca5 clientIP=212.102.38.92 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Segment loop timed out; closing

A lot in FRA and SIN

Titan-Node commented 2 years ago

It's almost as if the broadcaster is successfully sending the segments and the T transcodes them but when it is sends it back to the O it cannot find the profiles.

stronk-dev commented 2 years ago

Issue seems to persist on 0.5.33

Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.006048 3294220 segment_rpc.go:95] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Received segment dur=1.895s
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.006065 3294220 census.go:1118] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentEmerged... duration=1.895
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.133948 3294220 orchestrator.go:171] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Receiving ticket sessionID=c8c9d56d faceValue=0.015 ETH winProb=0.0000666667 ev=1000000000000.00
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.144986 3294220 segment_rpc.go:135] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Downloaded segment dur=10.816559ms
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.144999 3294220 census.go:1180] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Logging SegmentDownloaded... dur=10.816559ms
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146139 3294220 orchestrator.go:476] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Starting to transcode segment
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146149 3294220 orchestrator.go:464] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Creating new segment chan
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146159 3294220 orchestrator.go:612] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Starting transcode segment loop for manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 sessionID=c8c9d56d
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146173 3294220 orchestrator.go:485] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Submitted segment to transcode loop
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146406 3294220 lb.go:85] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E LB: Creating transcode session for job=c8c9d56d
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146430 3294220 lb.go:130] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E LB: Created transcode session for key=c8c9d56d_0
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.146436 3294220 lb.go:209] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E LB: Transcode submitted for key=c8c9d56d_0
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.533933 3294220 orchestrator.go:571] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Transcoding of segment took=387.519338ms
Jul 20 10:20:06 koios livepeer[3294220]: I0720 10:20:06.533961 3294220 census.go:1285] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentTranscode nonce=0 seqNo=1 dur=387.519338ms trusted=true verified=true
Jul 20 10:20:06 koios livepeer[3294220]: E0720 10:20:06.533993 3294220 orchestrator.go:582] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
Jul 20 10:20:06 koios livepeer[3294220]: E0720 10:20:06.534156 3294220 segment_rpc.go:231] manifestID=f7ef73b3-c3b6-4e5d-9cd3-cae532346e22 seqNo=1 orchSessionID=c8c9d56d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.080520 3294220 segment_rpc.go:95] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Received segment dur=3.748s
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.080551 3294220 census.go:1118] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentEmerged... duration=3.748
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.080697 3294220 orchestrator.go:171] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Receiving ticket sessionID=fdc3f243 faceValue=0.015 ETH winProb=0.0000666667 ev=1000000000000.00
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.118187 3294220 segment_rpc.go:135] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Downloaded segment dur=37.344371ms
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.118200 3294220 census.go:1180] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentDownloaded... dur=37.344371ms
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119208 3294220 orchestrator.go:476] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Starting to transcode segment
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119219 3294220 orchestrator.go:464] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Creating new segment chan
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119229 3294220 orchestrator.go:612] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Starting transcode segment loop for manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c sessionID=fdc3f243
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119244 3294220 orchestrator.go:485] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Submitted segment to transcode loop
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119462 3294220 lb.go:85] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E LB: Creating transcode session for job=fdc3f243
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119512 3294220 lb.go:130] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 LB: Created transcode session for key=fdc3f243_0
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.119519 3294220 lb.go:209] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E LB: Transcode submitted for key=fdc3f243_0
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.257607 3294220 orchestrator.go:571] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Transcoding of segment took=138.133967ms
Jul 20 10:22:42 koios livepeer[3294220]: I0720 10:22:42.257626 3294220 census.go:1285] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Logging SegmentTranscode nonce=0 seqNo=2 dur=138.133967ms trusted=true verified=true
Jul 20 10:22:42 koios livepeer[3294220]: E0720 10:22:42.257641 3294220 orchestrator.go:582] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0
Jul 20 10:22:42 koios livepeer[3294220]: E0720 10:22:42.257749 3294220 segment_rpc.go:231] manifestID=0bb5f33d-6d27-4880-a3bc-de8933f3971c seqNo=2 orchSessionID=fdc3f243 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"
thomshutt commented 2 years ago

We've been investigating under https://github.com/livepeer/lpms/issues/311, but will try to accelerate and get this sorted

stronk-dev commented 2 years ago

For the past while the issue has been occuring on EU only. Some of the streams affected:

  |   | E0727 21:32:06.111025 3500031 segment_rpc.go:231] manifestID=a08f3555-b5a7-4760-bf3d-597a241bbdb1 seqNo=802 orchSessionID=143bdf2e clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"   |   | E0727 21:32:06.110961 3500031 orchestrator.go:582] manifestID=a08f3555-b5a7-4760-bf3d-597a241bbdb1 seqNo=802 orchSessionID=143bdf2e sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Cannot find transcoded segment for bytes=0   |   | E0727 21:03:42.233882 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=971 orchSessionID=76fc7f4d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"   |   | E0727 21:03:42.233610 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=971 orchSessionID=76fc7f4d clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0   |   | E0727 21:01:57.821806 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=950 orchSessionID=e3697fcd clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"   |   | E0727 21:01:57.821411 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=950 orchSessionID=e3697fcd sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Cannot find transcoded segment for bytes=0   |   | E0727 21:00:55.587207 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=938 orchSessionID=5b8733b7 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"   |   | E0727 21:00:55.586828 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=938 orchSessionID=5b8733b7 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0   |   | E0727 20:59:22.964761 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=920 orchSessionID=ab55a166 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments"   |   | E0727 20:59:22.964516 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=920 orchSessionID=ab55a166 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0

E0727 20:57:43.023543 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=901 orchSessionID=86b9081c clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:57:43.023319 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=901 orchSessionID=86b9081c clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:56:08.016250 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=882 orchSessionID=9ad6852f sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Could not transcode err="ZeroSegments" E0727 20:56:08.016001 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=882 orchSessionID=9ad6852f clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:54:43.938407 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=865 orchSessionID=d8e87fcf clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:54:43.938025 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=865 orchSessionID=d8e87fcf clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:53:44.366015 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=854 orchSessionID=fd072d8f clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:53:44.365708 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=854 orchSessionID=fd072d8f clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:51:59.969965 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=835 orchSessionID=48a2be4a clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:51:59.969690 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=835 orchSessionID=48a2be4a clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:49:09.895883 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=804 orchSessionID=dc56d400 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:49:09.895592 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=804 orchSessionID=dc56d400 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:48:53.514643 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=801 orchSessionID=5c5a799a clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:48:53.514265 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=801 orchSessionID=5c5a799a clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:46:57.601369 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=779 orchSessionID=13a7215a clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:46:57.601049 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=779 orchSessionID=13a7215a sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Cannot find transcoded segment for bytes=0 E0727 20:45:53.679319 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=767 orchSessionID=6b4a2ad6 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:45:53.679014 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=767 orchSessionID=6b4a2ad6 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:43:54.258121 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=746 orchSessionID=c2fccfae clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:43:54.257823 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=746 orchSessionID=c2fccfae clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:42:33.977767 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=731 orchSessionID=a6d2bf64 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:42:33.977314 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=731 orchSessionID=a6d2bf64 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:39:57.692282 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=701 orchSessionID=93e3e814 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:39:57.691981 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=701 orchSessionID=93e3e814 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:39:30.693251 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=696 orchSessionID=95dfafd8 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:39:30.692903 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=696 orchSessionID=95dfafd8 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:38:04.491805 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=678 orchSessionID=e7cfed15 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:38:04.491516 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=678 orchSessionID=e7cfed15 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:32:49.079328 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=617 orchSessionID=005f9df0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:32:49.079085 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=617 orchSessionID=005f9df0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:31:28.649787 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=602 orchSessionID=cca8d878 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:31:28.649484 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=602 orchSessionID=cca8d878 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:30:01.200041 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=584 orchSessionID=6e0591f4 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:30:01.199731 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=584 orchSessionID=6e0591f4 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:28:26.387682 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=566 orchSessionID=86b8bdf2 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:28:26.387309 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=566 orchSessionID=86b8bdf2 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:25:46.795391 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=535 orchSessionID=2d3c9c14 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:25:46.794949 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=535 orchSessionID=2d3c9c14 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:25:10.826348 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=528 orchSessionID=0bf9e7c0 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Could not transcode err="ZeroSegments" E0727 20:25:10.826052 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=528 orchSessionID=0bf9e7c0 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:23:15.938231 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=505 orchSessionID=6896c9ad clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:23:15.937943 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=505 orchSessionID=6896c9ad clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:21:32.430775 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=485 orchSessionID=8d22db60 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:21:32.430335 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=485 orchSessionID=8d22db60 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:20:23.250506 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=472 orchSessionID=08b067ae clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:20:23.250251 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=472 orchSessionID=08b067ae clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 20:18:48.410339 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=454 orchSessionID=d44b74eb clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 20:18:48.410071 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=454 orchSessionID=d44b74eb clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:36:00.671376 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=15 orchSessionID=e738ec7f clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:36:00.670835 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=15 orchSessionID=e738ec7f clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:34:57.666616 3500031 segment_rpc.go:231] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=1 orchSessionID=51115847 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:34:57.666345 3500031 orchestrator.go:582] manifestID=4014bd1f-be48-46c4-9388-e63eaeae6347 seqNo=1 orchSessionID=51115847 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:33:03.052100 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=943 orchSessionID=1d6c7703 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=84.17.50.98 Could not transcode err="ZeroSegments" E0727 19:33:03.052030 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=943 orchSessionID=1d6c7703 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:32:26.140849 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=904 orchSessionID=2a26aef5 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:32:26.140785 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=904 orchSessionID=2a26aef5 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:32:00.256946 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=888 orchSessionID=a0ba62ba clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:32:00.256832 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=888 orchSessionID=a0ba62ba clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:27:16.821032 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=640 orchSessionID=577bfde9 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:27:16.820916 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=640 orchSessionID=577bfde9 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=84.17.50.98 Cannot find transcoded segment for bytes=0 E0727 19:26:29.088563 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=599 orchSessionID=7be4c5fc clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:26:29.088398 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=599 orchSessionID=7be4c5fc clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:26:28.948557 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=598 orchSessionID=7be4c5fc clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:26:28.948465 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=598 orchSessionID=7be4c5fc clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:25:03.434600 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=525 orchSessionID=ec3bfa81 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:25:03.434492 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=525 orchSessionID=ec3bfa81 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=84.17.50.98 Cannot find transcoded segment for bytes=0 E0727 19:16:01.269569 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=38 orchSessionID=b3a2badd clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:16:01.269391 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=38 orchSessionID=b3a2badd clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:16:01.079642 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=37 orchSessionID=b3a2badd clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:16:01.079423 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=37 orchSessionID=b3a2badd clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:15:44.313417 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=18 orchSessionID=1219a774 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Could not transcode err="ZeroSegments" E0727 19:15:44.313267 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=18 orchSessionID=1219a774 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:15:44.119256 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=17 orchSessionID=1219a774 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:15:44.118929 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=17 orchSessionID=1219a774 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:15:43.925183 3500031 segment_rpc.go:231] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=22 orchSessionID=1219a774 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Could not transcode err="ZeroSegments" E0727 19:15:43.925017 3500031 orchestrator.go:582] manifestID=70fd2279-8bb9-4d84-99be-68e3a97508f3 seqNo=22 orchSessionID=1219a774 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:14:37.171987 3500031 segment_rpc.go:231] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=346 orchSessionID=f28cc508 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Could not transcode err="ZeroSegments" E0727 19:14:37.171876 3500031 orchestrator.go:582] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=346 orchSessionID=f28cc508 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:12:39.811683 3500031 segment_rpc.go:231] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=248 orchSessionID=45ae9a26 clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:12:39.811446 3500031 orchestrator.go:582] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=248 orchSessionID=45ae9a26 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E clientIP=195.181.174.186 Cannot find transcoded segment for bytes=0 E0727 19:12:18.262891 3500031 segment_rpc.go:231] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=214 orchSessionID=12244f65 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:12:18.262798 3500031 orchestrator.go:582] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=214 orchSessionID=12244f65 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 19:12:18.142908 3500031 segment_rpc.go:231] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=212 orchSessionID=12244f65 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 19:12:18.142685 3500031 orchestrator.go:582] manifestID=be711b40-8025-4343-9845-d45972a1eea7 seqNo=212 orchSessionID=12244f65 clientIP=84.17.50.98 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0 E0727 18:34:35.382982 3500031 segment_rpc.go:231] manifestID=f6d7e45c-2e5f-4661-9901-3d1f0aaddad0 seqNo=1 orchSessionID=aef22b1b clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="ZeroSegments" E0727 18:34:35.382889 3500031 orchestrator.go:582] manifestID=f6d7e45c-2e5f-4661-9901-3d1f0aaddad0 seqNo=1 orchSessionID=aef22b1b clientIP=195.181.174.186 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Cannot find transcoded segment for bytes=0