search

livepeer / go-livepeer

Official Go implementation of the Livepeer protocol
http://livepeer.org
MIT License
539 stars 169 forks source link

T Crashes "End of stream receive cycle because of err="EOF", waiting for running transcode jobs to complete" #2702

Closed Titan-Node closed 1 year ago

Titan-Node commented 1 year ago

Describe the bug A regular T crashes after a few days of being online. This didn't happen before, not sure whats causing it. Been transcoding for over a year with this machine.

I1219 13:00:48.901495    9080 ot_rpc.go:128] End of stream receive cycle because of err="EOF", waiting for running transcode jobs to complete

Version 0.5.35

leszko commented 1 year ago

Thanks for reporting @Titan-Node

Do you maybe have any steps to reproduce? Or it just happens from time to time without any specific reason?

Titan-Node commented 1 year ago

@leszko trying to reproduce but it happens so infrequently. Also I should mention it doesn't crash but instead ends up in a state where the program stays alive but does not do any more transcoding and disconnects from the O.

eliteprox commented 1 year ago

Yesterday I observed a recurrence of related issue https://github.com/livepeer/go-livepeer/issues/2605 which I thought was solved in the 0.5.37 release. Three transcoders connected to an O, one is intentionally disconnected and the O loses all streams, no transcoders available. Transcoders map went empty. I will try to reproduce the issue in a test B/O/T configuration as soon as possible.

Titan-Node commented 1 year ago

I believe this issue was fixed in this PR. https://github.com/livepeer/go-livepeer/pull/2747 Haven't seen this issue again since the update. Thanks @eliteprox